Security Monitoring: Proven Methods for Incident Detection on Enterprise Networks [1 ed.] 0596518161, 9780596518165

Citation preview

Se cu r it y M on it or in g, 1 st Edit ion by Chris Fry; Mart in Nyst rom Publisher: O'Re illy M e dia , I n c. Pub Dat e: Fe br u a r y 2 4 , 2 0 0 9 Print I SBN- 13: 9 7 8 - 0 - 5 9 6 - 5 1 8 1 6 - 5 Pages: 2 5 6

Ove r vie w How well does your ent erprise st and up against t oday's sophist icat ed securit y t hreat s? Wit h t his book, securit y expert s from Cisco Syst em s dem onst rat e how you can det ect dam aging securit y incident s on your global net work first by discovering which asset s you need t o m onit or closely, t hen by helping you develop t arget ed st rat egies and pragm at ic t echniques t o ident ify securit y incident s. Securit y Monit oring offers six st eps t o im prove net work m onit oring, based on t he aut hors' years of experience conduct ing incident response t o keep Cisco's global net work secure. These st eps will guide you t hrough t he following:

Develop Policies: define t he rules, regulat ions, and crit eria against which t o m onit or Know Your Net work: build knowledge of your infrast ruct ure wit h net work t elem et ry Select Your Target s: define t he subset of infrast ruct ure where you'll focus m onit oring Choose Event Sources: ident ify t he event t ypes needed t o discover policy violat ions Feed and Tune: collect dat a and generat e alert s, t uning syst em s using cont ext Maint ain Dependable Event Sources: prevent crit ical gaps in your event collect ion and m onit oring To help you underst and t his fram ework, Securit y Monit oring illust rat es it s recom m endat ions using a fict ional m obile t elephony provider. Each chapt er's approach and t echniques are overlaid against t his fict ional exam ple wit h diagram s and det ailed exam ples. These recom m endat ions will help you select and deploy t he best t echniques for m onit oring your own ent erprise net work.

Copyr igh t Copyright © 2009, Chris Fry and Mart in Nyst rom . All right s reserved. Print ed in t he Unit ed St at es of Am erica. Published by O'Reilly Media, I nc., 1005 Gravenst ein Highway Nort h, Sebast opol, CA 95472. O'Reilly books m ay be purchased for educat ional, business, or sales prom ot ional use. Online edit ions are also available for m ost t it les ( ht t p: / / safari.oreilly.com ) . For m ore inform at ion, cont act our corporat e/ inst it ut ional sales depart m ent : ( 800) 998- 9938 or corporat [email protected] . Edit or: Mike Loukides Product ion Edit or: Sum it a Mukherj i Edit or: Audrey Doyle Nut shell Handbook, t he Nut shell Handbook logo, and t he O'Reilly logo are regist ered t radem arks of O'Reilly Media, I nc. Securit y Monit oring, t he im age of a m an using a t elescope, and relat ed t rade dress are t radem arks of O'Reilly Media, I nc. Many of t he designat ions uses by m anufact urers and sellers t o dist inguish t heir product s are claim ed as t radem arks. Where t hose designat ions appear in t his book, and O'Reilly Media, I nc. was aware of a t radem ark claim , t he designat ions have been print ed in caps or init ial caps. While every precaut ion has been t aken in t he preparat ion of t his book, t he publisher and aut hors assum e no responsibilit y for errors or om issions, or for dam ages result ing from t he use of t he inform at ion cont ained herein.

Prefa ce Our securit y t eam found a new way t o m ake m oney. I n 2006, aft er perfect ing our ent erprise m alware m onit oring, we began t o deploy t ools for m onit oring Cisco's infrast ruct ure m ore deeply. I n doing so, we found our t eam posit ioned t o m onit or applicat ions in new ways. Weary of ignoring t he risk present ed by new vent ures, we offered a solut ion: fund st aff t o m onit or t arget ed risk areas, and handle t he infrast ruct ure ourselves. The solut ion paid off—our m onit oring t eam has grown, and we've developed new t echniques for finding and addressing t he necessary risks of a growing ent erprise. I n 2007, we shared t his experience wit h our Forum for I ncident Response and Securit y Team s ( FI RST) buddies at t he annual conference. Som e say we chose t hat conference because it was being held in Seville, Spain, but we were j ust doing our part for t he securit y com m unit y. We want ed a crowd, so we t it led our present at ion " I nside t he Perim et er: 6 St eps t o I m prove Your Securit y Monit oring." We received enough encouragem ent t o repeat t he present at ion at t he annual Cisco Net workers conference lat er t hat year, where we expanded t he t alk t o t wo hours and packed t he house wit h an ent husiast ic audience. Feedback was posit ive, and we were asked t o repeat it in Brisbane, Aust ralia; Orlando, Florida; and Barcelona, Spain over t he next several m ont hs. I n t he m eant im e, we felt we had enough ideas t o fill a book, and t he edit ors at O'Reilly agreed. Our audiences t old us t hey liked t he present at ions because t hey craved honest experience from securit y pract it ioners. We share t he challenges you face; we're on t he hook for securit y, and have t o priorit ize resources t o m ake it happen. We like reading aut hent ic books—t he ones t hat don't t ry t o sell us gear or consult ing services—and we've endeavored t o writ e t his book wit h t hat angle. This book aim s t o share our experience, successes, and failures t o im prove securit y m onit oring wit h t arget ed t echniques.

P.1 . W h a t Th is Book I s N ot This book is not an int roduct ion t o net work, server, or dat abase adm inist rat ion. I t 's not an int roduct ion t o securit y t ools or t echniques, eit her. We assum e t hat you have a foundat ional underst anding of t hese areas and seek t o build on t hem via specialized applicat ion of t hem . I f we lose you along t he way, put a bookm ark where you left off, and reference t he following excellent books:

The Tao of Net work Securit y Monit oring, by Richard Bej t lich ( Addison- Wesley Professional) Essent ial Syst em Adm inist rat ion, by Æleen Frisch ( O'Reilly) Count er Hack Reloaded, by Ed Skoudis and Tom List on ( Prent ice Hall PTR) Com put er Viruses and Malware, by John Aycock ( Springer) Writ ing Secure Code, by Michael Howard and David LeBlanc ( Microsoft Press)

P.2 . W h a t Th is Book I s Hopefully, you've already read books on securit y. This one aim s t o t ake you deeper int o your net work, guiding you t o carve out t he m ore sensit ive, im port ant part s of t he net work for focused m onit oring. We haven't coined a t erm for t his, but if we did, it would be t arget ed m onit oring or policy- based m onit oring or t arget ed realit y- based policy m onit oring for det ect ing ext rusions. Here is a short sum m ary of t he chapt ers in t his book and what you'll find inside:

Chapt er 1 Provides rat ionale for m onit oring and challenges, and int roduces our m onit oring philosophy Follow ing Chapt er 1 are t he six core chapt ers of t he book, each successively building on t opics discussed in previous chapt ers:

Chapt er 2 Defines rules, regulat ions, and crit eria t o m onit or

Chapt er 3 Builds knowledge of your infrast ruct ure wit h net work t elem et ry

Chapt er 4 Defines t he subset of infrast ruct ure t o m onit or

Chapt er 5 I dent ifies t he event t ypes needed t o discover policy violat ions

Chapt er 6 Collect s dat a and generat es alert s, and t unes syst em s using cont ext

Chapt er 7 Prevent s crit ical gaps in your event collect ion and m onit oring Following t he core chapt ers are t he closing chapt er and a t rio of appendixes:

Chapt er 8 Provides case st udies and real exam ples t o illust rat e t he concept s present ed in t he six core chapt ers

Appendix A Provides det ailed inst ruct ions for im plem ent ing Net Flow collect ion based on Cisco's deploym ent

Appendix B Provides a sam ple service level agreem ent ( SLA) for m aint aining securit y event feeds from net work devices

Appendix C Offers st at ist ical proofs for calculat ing and calibrat ing upt im e for securit y m onit oring configurat ions

P.3 . Con ve n t ion s Use d in Th is Book The following t ypographical convent ions are used in t his book:

I t alic I ndicat es new t erm s, URLs, em ail addresses, filenam es, file ext ensions, pat hnam es, direct ories, and Unix ut ilit ies

Const ant widt h I ndicat es com m ands, opt ions, swit ches, variables, at t ribut es, keys, funct ions, t ypes, classes, nam espaces, m et hods, m odules, propert ies, param et ers, values, obj ect s, event s, event handlers, XML t ags, HTML t ags, m acros, t he cont ent s of files, and t he out put from com m ands

Constant width bold Shows com m ands and ot her t ext t hat should be t yped lit erally by t he user

Constant width italic

Shows t ext t hat should be replaced wit h user- supplied values

N ot e : This icon signifies a t ip, suggest ion, or general not e.

W arning: This icon indicat es a warning or caut ion.

P.4 . Usin g Code Ex a m ple s This book is here t o help you get your j ob done. I n general, you m ay use t he code in t his book in your program s and docum ent at ion. You do not need t o cont act us for perm ission unless you're reproducing a significant port ion of t he code. For exam ple, writ ing a program t hat uses several chunks of code from t his book does not require perm ission. Selling or dist ribut ing a CD- ROM of exam ples from O'Reilly books does require perm ission. Answering a quest ion by cit ing t his book and quot ing exam ple code does not require perm ission. I ncorporat ing a significant am ount of exam ple code from t his book int o your product 's docum ent at ion does require perm ission. We appreciat e, but do not require, at t ribut ion. An at t ribut ion usually includes t he t it le, aut hor, publisher, and I SBN. For exam ple: "Securit y Monit oring, by Chris Fry and Mart in Nyst rom . Copyright 2009 Chris Fry and Mart in Nyst rom , 978- 0- 596- 51816- 5." I f you feel your use of code exam ples falls out side fair use or t he perm ission given here, feel free t o cont act us at perm [email protected] .

P.5 . Sa fa r i® Book s On lin e N OTE

When you see a Safari ® Enabled icon on t he cover of your favorit e t echnology book, t hat m eans t he book is available online t hrough t he O'Reilly Net work Safari Bookshelf. Safari offers a solut ion t hat 's bet t er t han e- books. I t 's a virt ual library t hat let s you easily search t housands of t op t ech books, cut and past e code sam ples, download chapt ers, and find quick answers when you need t he m ost accurat e, current inform at ion. Try it for free at ht t p: / / safari.oreilly.com .

P.6 . Com m e n t s a n d Qu e st ion s Please address com m ent s and quest ions concerning t his book t o t he publisher: O'Reilly Media, I nc. 1005 Gravenst ein Highway Nort h Sebast opol, CA 95472 800- 998- 9938 ( in t he Unit ed St at es or Canada) 707- 829- 0515 ( int ernat ional or local) 707- 829- 0104 ( fax)

We have a web page for t his book, where we list errat a, exam ples, and any addit ional inform at ion. You can access t his page at : ht t p: / / www.oreilly.com / cat alog/ 9780596518165/ To com m ent or ask t echnical quest ions about t his book, send em ail t o: bookquest [email protected] For m ore inform at ion about our books, conferences, Resource Cent ers, and t he O'Reilly Net work, see our websit e at : ht t p: / / w w w .oreilly.com /

P.7 . Ack n ow le dgm e n t s We're kind of shy about put t ing our nam es on t his book. Chris and I did all t he writ ing, but t he ideas we're describing didn't originat e wit h us. They represent t he work st art ed by Gavin Reid, Cisco CSI RT's boss and FI RST rep, back in 2003. Gavin built t he CSI RT t eam , assem bled from proven net work engineers, syst em adm inist rat ors, and applicat ion developers. You'll find exam ples of script s writ t en by Dust in, Mike, and Dave, t uning developed by Jeff, Jayson, and Nit in, invest igat ions led by Chip and Kevin, and procedures writ t en by Lawrence. I n m any ways, t he whole t eam wrot e t his book. They're t he ones who deployed t he gear, wrot e t he t ools, hired t he st aff, built t he processes, and invest igat ed t he incident s t hat form t he basis for t he ideas present ed here. The book seem ed fine unt il Jeff Bollinger looked at it . He discovered all kinds of inconsist encies and t echnical gaps, and was kind enough t o t ell us about t hem before we published t he book. Jeff gave room for Devin Hilldale t o school us on st yle and gram m ar. Devin point ed out t he inconsist encies t hat derive from m ult iple aut hors, and helped sm oot h out t he writ ing st yle. He t old m e t o st op leaving t wo spaces aft er periods, but m y eight h grade t yping t eacher st ill cont rols m y fingers. Mark Lucking gave input t hroughout t he book, drawing from his experience in inform at ion securit y for banking. Good securit y requires good com m unit y. Cisco CSI RT part icipat es in securit y organizat ions of our peers in indust ry and governm ent . We share int elligence, t rack em erging t hreat s, and assist one anot her wit h incident response and invest igat ions. Mem bership in t rust ed securit y organizat ions such as FI RST and NSTAC NSI E provides access t o inform at ion in a currency of t rust . FI RST requires all prospect ive m em bers be nom inat ed by at least t wo exist ing m em bers. Candidat es m ust host an invest igat ive sit e visit by a FI RST m em ber, and be approved by a t wo- t hirds st eering com m it t ee vot e. I n Chapt er 8 , we shared valuable insight s from t wo case st udies. Thanks t o Scot t McI nt yre of KPN- CERT, and t o t he securit y m anagem ent at Nort hrop Grum m an: Georgia Newhall, George Bakos, Grant Jewell, and Rob Renew. ( Rob and Scot t : hope t o see you in Kyot o for FI RST 2009! ) This book will help you j ust ify m oney t o spend on securit y m onit oring. Read t he whole t hing, and apply all six st eps from t he core chapt ers t o use t hose resources efficient ly.

Ch a pt e r 1 . Ge t t in g St a r t e d I t was m id- January 2003. Things were going well in m y role as a net work engineer support ing dat a cent er net works at Cisco. My t eam celebrat ed on January 21 when our sit e vice president powered off t he last Avaya PBX; t he Research Triangle Park ( RTP) cam pus t elephony was now 100% VoI P. We had j ust com plet ed several WAN circuit and hardware upgrades and were beginning t o see t he highest availabilit y num bers ever for our rem ot e sit es. Then, on January 25 ( a Sat urday at t he RTP cam pus) , t he SQL Slam m er worm wreaked havoc on net works around t he world. Slam m er, also known as Sapphire, t arget ed vulnerable MS- SQL servers using self- propagat ing m alicious code. Securit y professionals surely rem em ber t he event well. The worm 's propagat ion t echnique creat ed a pot ent denial- of- service ( DoS) effect , bringing down m any net works as it spread. The only at t ribut e dist inguishing t he Slam m er worm from norm al SQL t raffic was a large num ber of 376- byt e UDP packet s dest ined for port 1434. [ 1 ] [ 1]

ht t p: / / www.cert .org/ advisories/ CA- 2003- 04.ht m l

I SPs used ingress/ egress filt ering t o block t raffic, but by t hen it was t oo lat e t o prevent syst em com prom ise; rat her, it was a m it igat ion m easure t o prot ect t he I nt ernet backbone: The Sapphire Worm was t he fast est com put er worm in hist ory. As it began spreading t hroughout t he I nt ernet , it doubled in size every 8.5 seconds. I t infect ed m ore t han 90 percent of vulnerable host s wit hin 10 m inut es. [ 2 ] [ 2]

ht t p: / / www.caida.org/ publicat ions/ papers/ 2003/ sapphire/ sapphire.ht m l

The rat e of replicat ion and m ult it ude of com prom ised syst em s on com pany net works began t o sat urat e net work links wit h propagat ion at t em pt s. Net work adm inist rat ors saw t his issue on som e of t he WAN links in t he Unit ed St at es when t heir pagers began t o light up like Christ m as t rees wit h ut ilizat ion alert s, followed by link down Sim ple Net work Managem ent Prot ocol ( SNMP) t raps. I nit ially, t he problem was t hought t o be relat ed t o a DS3 net work card we had j ust replaced in one of our Sout heast region WAN rout ers; however, as t he issue appeared in ot her regional office WAN links, it becam e clear t hat t his was not an isolat ed incident . We had experienced t he net work problem s caused by virus out breaks such as Code Red ( which at t acked vulnerable Microsoft I I S web servers) , but none approached t he severit y of net work im pact t hat Slam m er did. A few Slam m er host s were able t o generat e enough t raffic t o t ake down WAN links, causing int erm it t ent connect ivit y problem s in our rem ot e sit es globally. Ult im at ely, a m aj orit y of t he com prom ised syst em s were t raced t o unpat ched lab servers. I dent ifying and m it igat ing t hese host s was no easy t ask:

Too few net work int rusion det ect ion syst em s ( NI DSs) were deployed, and no one was responsible t o view or follow up on alert s for infect ed syst em s. Net work t elem et ry ( such as Net Flow) and anom aly det ect ion were insufficient t o ident ify infect ed syst em s. There was no way t o priorit ize t he response; t he only dat a we had were I P addresses and DNS nam es of affect ed m achines. We didn't have cont ext ual inform at ion such as " dat a cent er host " versus " user LAN host " versus " lab host ." Over t he next 48 hours, global net working t eam s ident ified infect ed syst em s using a m anual process t hat involved deploying t he recom m ended access cont rol list s ( ACLs) on rem ot e WAN rout ers[ 3 ] t o block packet s. Mat ches on t he deny access cont rol ent ries ( ACEs) for UDP 1434 indicat ed an infect ed host at t he sit e. We could not ident ify t he source I P address t hat was creat ing t he deny ent ries, as adding t he " log" clause t o t he end of t he deny ACE spiked t he rout er's CPU and drast ically degraded net work perform ance. The next st ep required net work engineers t o analyze swit ch port ut ilizat ion in real t im e, searching for t he infect ed host t o disable it s port . This m anual process required subst ant ial m an- hours t o address. [ 3]

ht t p: / / www.cisco.com / warp/ public/ 707/ cisco- sn- 20030125- worm .sht m l

I f we had im plem ent ed a few of t he recom m endat ions det ailed in t his book, our net working t eam could have cont ained t he t hreat m uch m ore rapidly. A t uned NI DS deploym ent would have enabled us t o locat e t he infect ed I P addresses im m ediat ely, priorit izing response based on t heir nam ed net work associat ion ( dat a cent er servers, lab host s, or deskt op syst em s, as you'll see in Chapt er 6 ) . Even prior t o t he availabilit y of t he NI DS signat ure, we could have used Net Flow t o ident ify infect ed host s based on recognized t raffic pat t erns, as we'll discuss in Chapt er 3 . A priorit ized, planned response would have occurred based on t his inform at ion, wit h appropriat e m it igat ion m easures applied t o t he im pact ed syst em s. The I P inform at ion from Net Flow alone could have allowed for quick m anual inspect ion of t he rout er ARP t ables and associat ed MAC- t o- I P address m apping. Arm ed wit h t hat m apping, t he net work engineers could have quickly disabled port s on t he access swit ches, shut t ing down worm propagat ion.

This book det ails infrast ruct ure and fram eworks t hat would have furt her helped when Nachi broke out several m ont hs lat er. Since we couldn't see t he fut ure, however, Nachi creat ed t he sam e effect and was addressed wit h t he sam e m anual process as Slam m er.

1 .1 . A Ra pidly Ch a n gin g Th r e a t La n dsca pe We've heard it before: " gone are t he days of script kiddies and t eenagers out t o wreak havoc j ust t o show off." The lat e 1990s and early 2000s produced a st aggering num ber of DoS at t acks. Malware, t he engine for t he DoS at t ack, has progressed from sim ple program s t hat at t ack a single vulnerabilit y t o com plex soft ware t hat at t acks m ult iple OS and applicat ion vulnerabilit ies. Let 's look at t he descript ion of t he Nachi worm 's m et hod of infect ion ( circa 2003) : This worm spreads by exploit ing a vulnerabilit y in Microsoft Windows. ( MS03- 026) Web servers ( I I S 5) t hat are vulnerable t o an MS03- 007 at t ack ( port 80) , via WebDav, are also vulnerable t o t he virus propagat ing t hrough t his exploit . [ 4 ] [ 4]

ht t p: / / vil.nai.com / vil/ cont ent / v_100559.ht m

Here's inform at ion on a very popular virus from 2006 called SDBot : The worm propagat es via accessible or poorly- secured net work shares, and som e variant s are int ended t o t ake advant age of high profile exploit s: WEBDAV vulnerabilit y ( MS03- 007) LSASS vulnerabilit y ( MS04- 011) ASN.1 vulnerabilit y ( MS04- 007) Workst at ion Service vulnerabilit y ( MS03- 049) PNP vulnerabilit y ( MS05- 039) I m ail I MAPD LOGI N usernam e vulnerabilit y Cisco I OS HTTP Aut horizat ion vulnerabilit y Server service vulnerabilit y ( MS06- 040) When it at t em pt s t o spread t hrough default adm inist rat ive shares, for exam ple: PRI NT$ E$ D$ C$ ADMI N$ I PC$ Som e variant s also carry a list of poor usernam e/ password com binat ions t o gain access t o t hese shares. Weak Passwords and Configurat ions Several variant s are known t o probe MS SQL servers for weak adm inist rat or passwords and configurat ions. When successful, t he virus could execut e rem ot e syst em com m ands via t he SQL server access. [ 5 ] [ 5]

ht t p: / / vil.nai.com / vil/ cont ent / v_139565.ht m

This m ore com plex form of m alware has com ponent s t o m ake it persist ent bet ween reboot s and t o cloak it self from

det ect ion by ant ivirus program s. I t even includes obfuscat ion t echniques t o prevent offline analysis! Many m alware program s include a com ponent t o st eal inform at ion from t he infect ed syst em and relay it back t o it s creat or, leveraging a rem ot e cont rol com ponent ( com m only called a botnet ) , which provides a vast array of capabilit ies t o com m and t he com prom ised syst em . Group all of t hese t rait s t oget her—decent ralized com m and and cont rol st ruct ures ( such as web- based or peer- t o- peer [ P2P] st ruct ures) , and encrypt ion and polym orphism ( so t hat t he m alware can m odify it self upon propagat ion t o anot her syst em , evading det ect ion by ant ivirus soft ware) —and you can easily see why ant ivirus t echnology rarely lives up t o it s prom ise.

1 .1 .1 . Fa ilu r e of An t ivir u s Soft w a r e Hopefully, you no longer rely solely on ant ivirus soft ware t o det ect and prot ect your end- user syst em s. Rat her, a defense- in- dept h st rat egy includes ant ivirus soft ware, adding OS and applicat ion pat ch m anagem ent , host - based int rusion det ect ion, and appropriat e access cont rols ( we said " hopefully" ) . I f you are st ill relying exclusively on ant ivirus soft ware for prot ect ion, you will be very disappoint ed. For exam ple, in sum m er 2008, m any of our em ployees received a well- craft ed phishing cam paign t hat cont ained a realist ic- looking em ail regarding a m issed shipm ent delivery from UPS:

-----Original Message----From: United Parcel Service [mailto:[email protected]] Sent: Tuesday, August 12, 2008 10:55 AM To: [email protected] Subject: Tracking N_ 6741030653 Unfortunately we were not able to deliver postal package you sent on July the 21st in time because the recipient's address is not correct. Please print out the invoice copy attached and collect the package at our office Your UPS

At t ached t o t his em ail was a t roj an t hat m ore t han 90% of t he 37 ant ivirus soft ware program s were unable t o det ect . Table 1- 1 shows t he t est result s yielded by analysis of t he t roj an binary. Ta ble 1 - 1 . Tr oj a n bin a r y a n a lysis t e st r e su lt s Ant ivirus Re su lt

Ant ivirus

Re su lt

AhnLab- V3

-

Kaspersky

-

Ant iVir

-

McAfee

-

Aut hent ium

W32/ Downldr2.DI FZ

Microsoft

-

Avast

-

NOD32v2

-

AVG

-

Norm an

-

Bit Defender

-

Panda

-

CATQuickHeal

-

PCTools

-

Clam AV

-

Prevx1

-

Dr Web

-

Rising

-

eSafe

-

Sophos

-

eTrust - Vet

-

Sunbelt

Troj anSpy.Win32.Zbot .gen ( v)

Ew ido

-

Sym ant ec

-

F- Prot

-

TheHacker

-

F- Secure

-

TrendMicro

-

Fort inet

-

VBA32

-

GDat a

-

ViRobot

-

I karus

Win32.Out break.UPSRechnung VirusBust er

-

K7Ant iVirus

-

-

Webwasher- Gat eway

As you can see from t he t est result s, t hese ant ivirus product s, which det ect m alware via " known bad" signat ures, failed t o ident ify t he t roj an. Such t echnology fails prim arily because an insignificant change t o t he virus will m ake it undet ect able by exist ing signat ures. Vendors are im proving t heir t echniques—by including heurist ic/ behavioralbased det ect ion, for exam ple—but t hey st ill fall far short of providing " com plet e" system securit y. An excellent source for m ore inform at ion regarding viruses, t heir capabilit ies, and why t hey are able t o hide from det ect ion is John Aycock's book, Com put er Viruses and Malware ( Springer) . The prevalence and advanced capabilit ies of m odern m alware should be reason enough t o closely m onit or for it s exist ence in your net work. I f it isn't , perhaps it s use by Mafia- like organizat ions of crim inals for profit via ident it y t heft , ext ort ion, and espionage is m ore convincing.

1 .1 . A Ra pidly Ch a n gin g Th r e a t La n dsca pe We've heard it before: " gone are t he days of script kiddies and t eenagers out t o wreak havoc j ust t o show off." The lat e 1990s and early 2000s produced a st aggering num ber of DoS at t acks. Malware, t he engine for t he DoS at t ack, has progressed from sim ple program s t hat at t ack a single vulnerabilit y t o com plex soft ware t hat at t acks m ult iple OS and applicat ion vulnerabilit ies. Let 's look at t he descript ion of t he Nachi worm 's m et hod of infect ion ( circa 2003) : This worm spreads by exploit ing a vulnerabilit y in Microsoft Windows. ( MS03- 026) Web servers ( I I S 5) t hat are vulnerable t o an MS03- 007 at t ack ( port 80) , via WebDav, are also vulnerable t o t he virus propagat ing t hrough t his exploit . [ 4 ] [ 4]

ht t p: / / vil.nai.com / vil/ cont ent / v_100559.ht m

Here's inform at ion on a very popular virus from 2006 called SDBot : The worm propagat es via accessible or poorly- secured net work shares, and som e variant s are int ended t o t ake advant age of high profile exploit s: WEBDAV vulnerabilit y ( MS03- 007) LSASS vulnerabilit y ( MS04- 011) ASN.1 vulnerabilit y ( MS04- 007) Workst at ion Service vulnerabilit y ( MS03- 049) PNP vulnerabilit y ( MS05- 039) I m ail I MAPD LOGI N usernam e vulnerabilit y Cisco I OS HTTP Aut horizat ion vulnerabilit y Server service vulnerabilit y ( MS06- 040) When it at t em pt s t o spread t hrough default adm inist rat ive shares, for exam ple: PRI NT$ E$ D$ C$ ADMI N$ I PC$ Som e variant s also carry a list of poor usernam e/ password com binat ions t o gain access t o t hese shares. Weak Passwords and Configurat ions Several variant s are known t o probe MS SQL servers for weak adm inist rat or passwords and configurat ions. When successful, t he virus could execut e rem ot e syst em com m ands via t he SQL server access. [ 5 ] [ 5]

ht t p: / / vil.nai.com / vil/ cont ent / v_139565.ht m

This m ore com plex form of m alware has com ponent s t o m ake it persist ent bet ween reboot s and t o cloak it self from det ect ion by ant ivirus program s. I t even includes obfuscat ion t echniques t o prevent offline analysis! Many m alware program s include a com ponent t o st eal inform at ion from t he infect ed syst em and relay it back t o it s creat or, leveraging a rem ot e cont rol com ponent ( com m only called a botnet ) , which provides a vast array of capabilit ies t o com m and t he com prom ised syst em . Group all of t hese t rait s t oget her—decent ralized com m and and cont rol st ruct ures ( such as web- based or peer- t o- peer [ P2P] st ruct ures) , and encrypt ion and polym orphism ( so t hat t he

m alware can m odify it self upon propagat ion t o anot her syst em , evading det ect ion by ant ivirus soft ware) —and you can easily see why ant ivirus t echnology rarely lives up t o it s prom ise.

1 .1 .1 . Fa ilu r e of An t ivir u s Soft w a r e Hopefully, you no longer rely solely on ant ivirus soft ware t o det ect and prot ect your end- user syst em s. Rat her, a defense- in- dept h st rat egy includes ant ivirus soft ware, adding OS and applicat ion pat ch m anagem ent , host - based int rusion det ect ion, and appropriat e access cont rols ( we said " hopefully" ) . I f you are st ill relying exclusively on ant ivirus soft ware for prot ect ion, you will be very disappoint ed. For exam ple, in sum m er 2008, m any of our em ployees received a well- craft ed phishing cam paign t hat cont ained a realist ic- looking em ail regarding a m issed shipm ent delivery from UPS:

-----Original Message----From: United Parcel Service [mailto:[email protected]] Sent: Tuesday, August 12, 2008 10:55 AM To: [email protected] Subject: Tracking N_ 6741030653 Unfortunately we were not able to deliver postal package you sent on July the 21st in time because the recipient's address is not correct. Please print out the invoice copy attached and collect the package at our office Your UPS

At t ached t o t his em ail was a t roj an t hat m ore t han 90% of t he 37 ant ivirus soft ware program s were unable t o det ect . Table 1- 1 shows t he t est result s yielded by analysis of t he t roj an binary. Ta ble 1 - 1 . Tr oj a n bin a r y a n a lysis t e st r e su lt s Ant ivirus Re su lt

Ant ivirus

Re su lt

AhnLab- V3

-

Kaspersky

-

Ant iVir

-

McAfee

-

Aut hent ium

W32/ Downldr2.DI FZ

Microsoft

-

Avast

-

NOD32v2

-

AVG

-

Norm an

-

Bit Defender

-

Panda

-

CATQuickHeal

-

PCTools

-

Clam AV

-

Prevx1

-

Dr Web

-

Rising

-

eSafe

-

Sophos

-

eTrust - Vet

-

Sunbelt

Troj anSpy.Win32.Zbot .gen ( v)

Ew ido

-

Sym ant ec

-

F- Prot

-

TheHacker

-

F- Secure

-

TrendMicro

-

Fort inet

-

VBA32

-

GDat a

-

ViRobot

-

I karus

Win32.Out break.UPSRechnung VirusBust er

-

K7Ant iVirus

-

-

Webwasher- Gat eway

As you can see from t he t est result s, t hese ant ivirus product s, which det ect m alware via " known bad" signat ures, failed t o ident ify t he t roj an. Such t echnology fails prim arily because an insignificant change t o t he virus will m ake it undet ect able by exist ing signat ures. Vendors are im proving t heir t echniques—by including heurist ic/ behavioral-

based det ect ion, for exam ple—but t hey st ill fall far short of providing " com plet e" system securit y. An excellent source for m ore inform at ion regarding viruses, t heir capabilit ies, and why t hey are able t o hide from det ect ion is John Aycock's book, Com put er Viruses and Malware ( Springer) . The prevalence and advanced capabilit ies of m odern m alware should be reason enough t o closely m onit or for it s exist ence in your net work. I f it isn't , perhaps it s use by Mafia- like organizat ions of crim inals for profit via ident it y t heft , ext ort ion, and espionage is m ore convincing.

1 .2 . W h y M on it or ? Organized crim e and insider t hreat s are changing t he securit y landscape, and provide am ple rat ionale for proact ive securit y m onit oring.

1 .2 .1 . Th e M iscr e a n t Econ om y a n d Or ga n ize d Cr im e An enorm ous am ount of m oney is being st olen every day—enough, in fact , t o drive coordinat ion and cooperat ion wit hin groups of crim inals. This illicit part nership has accelerat ed t he developm ent of sophist icat ed m alware ( used for t his purpose, it 's oft en called crim eware) . Most inform at ion securit y organizat ions, bot h governm ent and privat e, are ill- equipped t o handle such t hreat s wit h t heir exist ing t echnology and processes. A 2008 st udy by F- Secure Corporat ion predict ed t hat t he use of m alware for crim inal act ivit y would increase in count ries such as Brazil, China, t he form er Soviet Union, I ndia, Africa, and Cent ral Am erica. This is due t o an abundance of highly skilled people who lack opport unit ies t o use t hose skills in a legal m anner. [ 6 ] [ 6]

ht t p: / / www.f- secure.com / f- secure/ pressroom / news/ fsnews_20080117_1_eng.ht m l

Alt hough m ost of t his act ivit y is not direct ed at corporat ions, we have seen incident s t hat exploit knowledge of nam es or t eam / m anagem ent relat ionships, allowing t he creat ion of very believable phishing em ails. This t echnique is oft en referred t o as spearphishing. I n cont rast , t he act ions of m alicious insiders wit h access t o crit ical inform at ion and int ellect ual propert y m ake up what is referred t o as an insider t hreat.

1 .2 .2 . I n side r Th r e a t s St udies from t he U.S. Secret Service and t he U.S. Com put er Em ergency Response Team Coordinat ion Cent er ( CERT/ CC) validat e t he exist ence of insider t hreat s. Alt hough m any st ill debat e t he exact percent age, it appears t hat bet ween 40% and 70% of all incident s are relat ed t o insider t hreat s. This sizable am ount , coupled wit h t he insider's access and knowledge, m ust be m et wit h a proport ionat e am ount of m onit oring effort s t oward insider act ivit y. A few high- profile incident s should help t o drive t he insider t hreat m essage hom e: [ 7 ] [ 7]

Source: ht t p: / / www.privacyright s.org/ ar/ ChronDat aBreaches.ht m # 2008

Horizon Blue Cross Blue Shield I n January 2008, m ore t han 300,000 nam es and Social Securit y num bers were exposed when a lapt op was st olen. An em ployee who regularly works wit h m em ber dat a was t aking t he lapt op hom e.

Hannaford Bros. Co. I n May 2008, 4.2 m illion credit and debit card num bers were com prom ised. Close t o 1,800 cases of fraud were report ed relat ed t o t his securit y breach. I t was found t hat t he card num bers were harvest ed during t he t ransact ion process.

Com pass Bank I n March 2008, a dat abase cont aining nam es, account num bers, and cust om er passwords was breached. A form er em ployee st ole a hard drive cont aining 1 m illion cust om er records and used t hat inform at ion t o com m it fraud. He used a credit card encoder and blank cards t o creat e several new cards and wit hdraw m oney from m ult iple cust om er account s.

Count rywide Financial Corp. I n August 2008, t he FBI arrest ed a form er Count rywide Financial Corp. em ployee for st ealing personal inform at ion, including Social Securit y num bers. The insider was a senior financial analyst at a subprim e lending division. The alleged perpet rat or of t he t heft sold account inform at ion weekly in groups of 20,000 for $500. Not all of t he aforem ent ioned incident s were m alicious in nat ure, but all of t hem began wit h a violat ion of securit y

policy. Chapt ers Chapt er 2 and Chapt er 6 provide a fram ework for you t o det ect m alware and insider t hreat s. Chapt ers Chapt er 4 and Chapt er 5 will help you priorit ize your lim it ed m onit oring resources and choose t he event dat a t hat provides t he " biggest bang for t he buck."

1 .3 . Ch a lle n ge s t o M on it or in g Product lim it at ions, t he realit ies of operat ional m onit oring, event volum es, and t he necessit y of privacy prot ect ion are challenges faced by securit y professionals when const ruct ing a m onit oring approach.

1 .3 .1 . Ve n dor Pr om ise s " Just plug it in; it will sort out everyt hing for you! " This advice on set t ing up vendor XYZ's Securit y I nform at ion Manager ( SI M) syst em t o " aut om agically" correlat e securit y event s m ay work in sm all, st rict , well- m aint ained environm ent s. However, ut opian environm ent s such as t hese are rare in our experience and in t alking wit h our cust om ers. Securit y m onit oring is not like a Ron Popeil Showt im e Rot isserie; you can't " set it and forget it ." Securit y t echnology cannot aut om at ically provide t he cont ext ual inform at ion necessary for you t o priorit ize and focus your securit y m onit oring. Every environm ent is unique, but t he m et hods we discuss in Chapt er 3 will enable you t o build t his crit ical cont ext ual inform at ion int o all of your securit y t ools. " But wait , t here's m ore! "

1 .3 .2 . Ope r a t ion a l Re a lit ie s " Turn on audit ing for all your dat abase t ables." Dat abase operat ions in a busy ent erprise environm ent priorit ize perform ance and st abilit y, which gave us pause when considering such advice. What are t he pot ent ial perform ance im pact s? What risks does t his int roduce t o business operat ions, change cont rols, st abilit y, and upt im e? We began t o discuss t hese concept s t hrough em ail wit h an I T dat abase adm inist rat or. He st opped replying t o our em ails aft er we relayed t he " t urn on audit ing for all your t ables" advice! I ndeed, such int ensive dat abase audit ing in any but t he m ost rarely used environm ent would reduce syst em perform ance t o unaccept able levels. Our recom m endat ions in t his book are t est ed and proven by our own operat ional experience in I T, where we have bot h support ed ent erprise infrast ruct ures. We won't suggest m et hods t hat will negat ively im pact availabilit y, t hus harm ing your relat ionship wit h t he support st aff.

1 .3 .3 . Volu m e I n t he cont ext of m onit oring, logging dat a quickly degrades from essent ial lifeblood t o useless swam p wat er when t he volum e is t oo high. An im properly t uned NI DS or syslog daem on can creat e so m any m essages t hat it lit erally crashes your collect ion syst em s. Even if your collect or or SI M can handle t he flood of event dat a, a huge volum e of unsort ed event s will overwhelm your m onit oring st aff and drive t hem t o ignore t he dat a source. The guidelines in Chapt ers Chapt er 5 and Chapt er 6 will give you t he right direct ion for m aking event volum e m anageable even in t he m ost com plex environm ent s.

1 .3 .4 . Pr iva cy Con ce r n s You m ust t ake care t o com ply wit h local privacy laws and regulat ions, as t hey vary widely by count ry. The best advice we can give is t o ensure t hat your hum an resources depart m ent and legal counsel are aware of your m onit oring act ivit ies, form ally docum ent ing t heir approval for fut ure reference. This is t ypically done in a m onit oring st at em ent , which should be included in your com pany's accept able use policy.

1 .4 . Ou t sou r cin g You r Se cu r it y M on it or in g I n m any com panies, securit y is lit t le m ore t han a checkbox on a com pliance docum ent . " Em ployees, check! I T Services, check! Securit y, check! " And so on.... I f you've already out sourced t he whole of your securit y m onit oring t o a m anaged securit y services vendor so t hat you can check your com pliance boxes, st op reading and sell t his copy on eBay. You probably haven't even cracked t he binding, so you can list it " like new." I n our experience and wit h t alking t o cust om ers, it is ext rem ely rare t o find an out sourced securit y services vendor who really underst ands t he net work and securit y cont ext of it s client s, and as such rest rict s it s effect iveness t o t he sim plest of securit y problem s. I m agine t he following proposal: you want t o know when som eone copies cust om er dat a from your dat abase syst em s t o his deskt op. How would an out sourced securit y provider do t his? Rat her, how m uch would such a provider charge t o do t his? The service supplied by m ost providers is lim it ed t o regular report s of select ed NI DS alert s—t he sam e NI DS alert s select ed for every client —and affect ed I P addresses—not all t hat useful, in our opinion.

1 .5 . M on it or in g t o M in im ize Risk B2B, part ner , out source, ext ranet ; words t hat m ake securit y professionals cringe wit h disdain. Som et im es direct ors m ust accept high risk, such as connect ing a part ner net work before proper risk assessm ent can be com plet ed, due t o urgent business drivers. Oft en, however, such decisions are m ade by t hose wit hout aut horit y t o assum e such a high level of risk. Such decisions affect an ent ire corporat ion, and are oft en m ade wit h flawed or incom plet e inform at ion. I n response, t hose charged wit h inform at ion securit y are t em pt ed t o get frust rat ed and surrender t o chance. Such capit ulat ion is not necessary. I f you follow t he approach laid out in t his book, you can t ailor a m onit oring st rat egy based on t he " special" business sit uat ion, m inim izing or even m it igat ing t he addit ional risk. Require t arget ed securit y m onit oring, funded by t he risk- t aking sponsors, by saying, " I f you want t o vent ure int o t his risky proj ect , you will need t o fund addit ional m onit oring resources for hardware and headcount ."

1 .6 . Policy- Ba se d M on it or in g We want t o different iat e our fram ework for policy- based m onit oring ( som et im es we call it t arget ed m onit oring ) from m alware m onit oring, int rusion det ect ion, ext rusion det ect ion, and popular m onit oring fram eworks. Policybased m onit oring priorit izes m onit oring by enum erat ing and select ing crit ical syst em s, det ect ing policy deviat ions via t heir appropriat e event logs. I t requires analysis of generat ed event s against defined securit y policies wit hin t he cont ext of t he environm ent . The m et hods we describe will help you t o shift t he focus of your m onit oring resources t o t he m ost business- crit ical syst em s, bounding your alert s wit hin t he defined securit y policies for t hose syst em s.

1 .7 . W h y Sh ou ld Th is W or k for You ? We st rongly believe t hat t he fram eworks and m et hods present ed here are effect ive and sound, based on our experience wit hin one of t he m ost com plex and fluid ent erprise net works in t he world. We bot h have support ed crit ical syst em s whose upt im e direct ly im pact ed business revenue and em ployee product ivit y ( and ult im at ely, our careers) . This guidance is t he result of it erat ive im provem ent s, and should apply across all t echnologies in your exist ing securit y port folio. The bot t om line: if you im plem ent j ust som e of t he recom m endat ions m ade in t his book, you should im prove your m onit oring and incident response capabilit ies great ly. I f you im plem ent all of t he recom m endat ions, you will creat e a world- class securit y m onit oring capabilit y.

1 .8 . Ope n Sou r ce Ve r su s Com m e r cia l Pr odu ct s Bot h of us are em ployees of Cisco Syst em s, and we use t heir securit y product s. Because we are giving you advice based on our experience, you will find m any references t o Cisco product s. We use open source t ools when t hey m eet a specific need, and reference t hem ent husiast ically when t hey work well. Open source product s are feat ured in Richard Bej t lich's book, The Tao of Net work Securit y Monit oring ( Addison- Wesley Professional) , which covers t he use of securit y m onit oring t ools such as Snort , Bro, Argus, Sguil, and dozens of ot hers. I t is a great reference for t hose who have not already built , or are looking t o enhance, t heir m onit oring infrast ruct ure. This book int ends t o help readers get t he m ost out of t heir securit y m onit oring t ools, whichever product s t hey choose.

1 .9 . I n t r odu cin g Bla n co W ir e le ss To illust rat e our recom m endat ions, we show t heir im plem ent at ion wit hin a fict it ious com pany nam ed Blanco Wireless. Blanco is a m obile t elephony provider based in t he Unit ed St at es. As a m eans of account m anagem ent , Blanco collect s and st ores personal inform at ion, including nam es, addresses, phone num bers, Social Securit y num bers, credit rat ings, and ot her im port ant cust om er dat a. At t he end of each chapt er, we discuss how Blanco Wireless im plem ent s t he fram eworks and m et hods suggest ed in t he chapt er. Exam ples include diagram s and explanat ions of how Blanco will use t he chapt er's guidance t o develop securit y m onit oring.

Ch a pt e r 2 . I m ple m e n t Policie s for M on it or in g My first college apart m ent had a t errible cockroach problem . Upon ret urning from a dat e one evening, I was shocked t o see dozens of t hem scat t er away from an em pt y pizza box when I t urned on t he light s. Aft er t hat , it was t ough t o push away t he idea t hat cockroaches were everywhere—I expect ed t o see t hem in every corner of t he apart m ent . The first t im e I fired up Snort I was rem inded of t hat experience; suddenly I could see what was crawling t hrough t he net work, and I want ed t o fix it all at once. I t 's easy t o get sucked int o bug st om ping: once you see what 's on t he net work, you have t he urge t o fix and explain every securit y event you discover. Here's where t he analogy ends, t hough, for not everyt hing on t he wire is a cockroach. Much of t he t raffic is perfect ly fine, if ugly. Once you underst and t hat it s ugliness is not a securit y t hreat , you can safely let it t hrough. By narrowing your focus t o t he t ruly t hreat ening t raffic, you can t urn your full at t ent ion t o st om ping it out . Hist orically, securit y m onit oring t ools have dem onst rat ed t heir wort h by showing t he cockroaches: t hey illum inat e t he dark corners t o show you how well t hey're perform ing t heir t ask. Once you're convinced of a cockroach problem , you need a plan for dealing wit h t he problem , and t hat plan will likely involve prevent ion and det ect ion. A securit y guard could easily be fooled if his pract ice was t o invest igat e every m ovem ent det ect ed by securit y cam eras. He m ust work from a plan—when t o invest igat e, when t o log act ivit y for furt her analysis, and when t o ignore som et hing as norm al or insignificant . To organize his response, he m ust choose from a com binat ion of t he following approaches:

Blacklist ing Enum erat e t hreat ening event s and docum ent preparat ions t o det ect and address t hem . For exam ple, t he plan m ight enum erat e t hreat ening act ors ( such as people wit h weapons and ski m asks) and prevent t hem from ent ering t he prem ises. Signs forbidding specific it em s or act ivit ies, such as t he one in Figure 2- 1, are im plem ent ing a blacklist .

Anom aly m onit oring Enum erat e event s considered norm al, such as t he general work schedule and how em ployees are expect ed t o dress. Wat ch for event s t hat fall out side of what 's norm al. Signs such as t he suspicious m ail alert highlight ed in Figure 2- 2 im plem ent exam ples of anom aly m onit oring. N OTE

This is oft en called whit elist m onit oring, as it highlight s what is accept able and considers anyt hing not list ed t o be unaccept able.

Policy m onit oring Enum erat e a set of discernible crit eria from which you can evaluat e each person or vehicle approaching t he prem ises t o det erm ine t he response. Signs forbidding ent ry by all but aut horized personnel, as shown in Figure 2- 3, are exam ples of such m onit oring in act ion. Figu r e 2 - 1 . Ex a m ple of bla ck list m on it or in g

Figu r e 2 - 2 . Ex a m ple of a n om a ly m on it or in g

Figu r e 2 - 3 . Ex a m ple of policy m on it or in g

2 .1 . Bla ck list M on it or in g Creat ing a list of prohibit ed event s or it em s ( com m only called a blacklist ) is t he m ost st raight forward m et hod of securit y m onit oring. Wit h t he blacklist in hand, you can deploy t ools t o det ect prohibit ed it em s, and build procedures for rem ediat ing t hem . This t echnique is m ost effect ive under t he following condit ions:

You can reliably and accurat ely ident ify signs of dangerous or m alicious behavior Som e signs are accurat e indicat ions t hat som et hing is wrong: an airport securit y checkpoint screens for t he presence of banned it em s such as weapons and bom b chem icals. I f t he it em s you're screening for, however, are not accurat e signs of t rouble, it will bog down t he m onit oring process as your st aff m ust t ake t he t im e t o weed out t he false posit ives. For exam ple, because t he Transport at ion Safet y Adm inist rat ion ( TSA) was unable t o ident ify only dangerous liquids at securit y checkpoint s, it chose t o ban all liquids ( see Figure 2- 4 ) . This present ed a problem because m any liquids were perfect ly safe and necessary, such as baby form ula. The blacklist m ust also be lim it ed t o it em s t hat can be reliably ident ified. Soft ware firewalls running on deskt ops have hist orically been very poor at t his; t hey block t raffic or prom pt t he user for harm less connect ions in an effort t o dem onst rat e t hat t hey're st ill on t he j ob ( see Figure 2- 5 ) . Unfort unat ely, t his condit ions Aunt Mary int o answering OK or Allow t o every prom pt wit hout considering t he danger of doing so, negat ing t he firewall's purpose. I f a blacklist ed it em can be obscured in som e fashion, it cannot be reliably ident ified and will sail past det ect ion and prevent ion t ools.

You have a relat ively sm all list I f you have only a few it em s t o wat ch for, it 's reasonable t o keep t he list up- t o- dat e wit h filt ers t hat are properly t uned t o find t he right t riggers. I f t he list is t oo long, however, it becom es im possible t o keep it upt o- dat e and reliable. For exam ple, t he " do not fly" list is reasonably effect ive only because it represent s a t iny percent age of t he flying populat ion. I f t he list doubles or t riples in size, it m ay creat e chaos at securit y checkpoint s. Ant ivirus t ools have been successful because t hey can ident ify a sm all list of bad files from an infinit e list of harm less files. Most of t oday's web cont ent filt ers, oft en used by libraries and fam ilies t o keep out unsavory cont ent , police browsing by checking against a list of " known bad" dom ain nam es. This works only because t here are only a few t housand sit es t o filt er out , com pared t o t he m illions of available websit es on t he I nt ernet .

You cannot const rain event s t o a narrow set of accept able crit eria I f t he TSA could rest rict baggage t o a narrow list of preapproved it em s, air t ravel m ight be m uch safer, but at an unaccept able cost . To effect ively enforce t he rule, t he agency would have t o draft a list of reliably safe it em s. Passengers would be perm it t ed t o wear and carry it em s from only a sm all list of choices. Because t hat is not pract ical, t he agency screens for blacklist ed it em s. I n an ent erprise, however, it is possible t o cat alog all aut horized applicat ions and devices on your net work. Though it 's a vast t ask, cont rolled environm ent s such as dat a cent ers allow reasonable lim it s regarding which it em s are allowed and expect ed on t he net work. On t he net work perim et er, t his doesn't hold t rue. You m ust sort t hrough t he t raffic flowing t hrough your I nt ernet gat eways t o det erm ine how best t o secure yourself in t he t hick of it . Figu r e 2 - 4 . TSA r u le s for liqu ids a n d ge ls

Figu r e 2 - 5 . Ex a m ple of con fu sin g fir e w a ll m e ssa ge

Therefore, blacklist m onit oring has been t radit ionally applied t o t he perim et er due t o t he large volum e of dat a. Monit oring t ools m ust select a sm all percent age of t raffic upon which t o alarm . This t ype of m onit oring can be effect ive against t hreat s t hat are reliably ident ified, det ect ing t hem wit h signat ure pat t erns on t he wire. I t 's becom ing increasingly easy for m alware aut hors t o evade sim ple signat ure- based det ect ion. A single, inconsequent ial change t o a m alware binary can t hrow off t he signat ure pat t ern, and encoding or placing null byt es in t he payload can furt her obfuscat e t he m alicious t raffic.

2 .2 . An om a ly M on it or in g Monit oring for m eaningful deviat ions from norm al t raffic and event s is a prom ising t echnique. I t 's an em erging area of int rusion det ect ion, and m onit oring t hat uses art ificial int elligence and st at ist ical deviat ions t o det ect t raffic abnorm alit ies. When anom aly det ect ion is init ially deployed, t he t ools m ust first creat e a wat erm ark against t he t raffic t hat will be m easured. Sust ained st at ist ical deviat ions above or below t hat wat erm ark are t riggers for t he t ool t o analyze t he t raffic furt her and produce an alert for a net work securit y analyst . Product s such as Arbor Peakflow, which provides an early warning of denial- of- service ( DoS) t raffic and ot her anom alous pat t erns, have em ployed t his t echnique effect ively. I nt rusion det ect ion syst em s have a growing set of capabilit ies t o det ect anom alies in prot ocol usage, such as t unneling and nonencrypt ed t raffic over encrypt ed prot ocols. They're also good at det ect ing volum e- based incident s, such as port scans. St ill, t his t echnique can elicit a high rat e of false posit ives, and it doesn't oft en capt ure enough det ail t o conduct m eaningful incident response.

2 .3 . Policy M on it or in g The goal of policy m onit oring is t o com pare event s discovered on t he net work t o ensure t hat t hey are approved and accept able. For exam ple, in a sensit ive environm ent , a securit y guard would have a list of t hose who are perm it t ed t o ent er t he building aft er business hours ( t he policy) . The guard would have cause t o quest ion and det ain any nonlist ed person ent ering t he building aft er hours. A bet t er exam ple of policy m onit oring is applied t o count erfeit prot ect ion. I t 's com m on for ret ailers t o require t heir cashiers t o inspect large bills ( say, bills larger t han $20 in t he Unit ed St at es) before accept ing t hem . Policy- based m onit oring is being applied, as t he cashier inspect s t he currency for reliable indicat ions t hat it is bona fide before accept ing it . The only bills legal t o creat e or pass for sale are t hose m int ed by t he U.S. Treasury. The Treasury designs securit y feat ures int o t he currency t o help cashiers and ot hers evaluat e t he bill's int egrit y. To prove aut hent icit y, t he cashier can evaluat e cert ain hard- t o- falsify t rait s of t he bill, such as wat erm arks, holographic im ages, color- shift ing ink, and securit y t hreads. This requires t he cashier t o know and be able t o accurat ely ident ify such t rait s. Success depends on bot h t he currency's reliable, unique, and falsificat ion- proof securit y feat ures, and t he cashier's abilit y t o acknowledge t hese signs. Policy- based net work m onit oring is pract ical where accept able condit ions can be docum ent ed as policies. For exam ple, a securit y colleague once discovered a Windows server in t he dat a cent er m aking dozens of connect ions t o sit es out side his com pany's address space, and det erm ined t hat t hose sit es were bot net cont rollers. I f his com pany had applied policy m onit oring, t he server would have been allowed t o init iat e connect ions t o only a handful of sit es. You can enforce t his t ype of cont rol in a firewall, t hough som e net work engineers are ret icent t o im pact net work operat ions or perform ance by deploying t hem . When firewalls can't be applied, t he need for policybased net work m onit oring is am plified, and would have called out t his com prom ise im m ediat ely. A net work is a good candidat e for policy m onit oring if it m eet s t he following t wo condit ions:

You have reasonable cont rol over deployed syst em s and services Most ent erprises have st andards t hat rest rict t he servers and services allowed int o dat a cent ers. This allows t he net work m onit oring st aff t o docum ent a relat ively sm all set of expect ed prot ocols on t he wire. I t also allows adm inist rat ors t o rest rict allowable prot ocols so t hat policy m onit oring can succeed. An environm ent is ready for policy m onit oring if adm inist rat ors can art iculat e accept able t raffic pat t erns.

A m alicious payload can be easily disguised I f m alevolent t raffic can be easily obfuscat ed so t hat it slips past your int rusion det ect ion sensors and virus scanners, policy m onit oring will allow you t o see t hat t raffic in new ways. This can im prove t he fidelit y, accuracy, and t im eliness of your blacklist via a hybrid approach.

2 .4 . M on it or in g Aga in st D e fin e d Policie s To effect ively m onit or t he ent erprise, you m ust codify accept able behavior as policies, providing a reference point against which t o survey. These policies m ust be precise and concret e t o be successful. When m y daught er received her st age one driver's license, she was allowed t o drive only bet ween t he hours of 6 a.m . and 9 p.m . To m onit or for com pliance of such a policy, an officer need only check t he license st at us of a young adult against t he t im e of day when evaluat ing com pliance. The policy was clear and concise, and she knew exact ly what was expect ed of her. Of course, in m onit oring for det erm ined t hreat s, you should keep your policy det ails a closely guarded secret , as a t rue crim inal will disguise t raffic t o evade det ect ion. I n developing policies against which you can build m onit oring procedures, it 's helpful t o reference ext ernal st andards such as t hose published by t he I SO. The Sit e Securit y Handbook ( RFC 2196) suggest s t hat a solid securit y policy m ust have t he following charact erist ics:

I t m ust be capable of being im plem ent ed t hrough syst em adm inist rat ion procedures, publishing of accept able use guidelines, or ot her appropriat e m et hods. I t m ust be enforceable wit h securit y t ools, where appropriat e and wit h sanct ions, where act ual prevent ion is not t echnically feasible. I t m ust clearly define t he areas of responsibilit y for t he users, adm inist rat ors, and m anagem ent .

k e e p- t oge t h e r 2 .4 .1 . M a n a ge m e n t En for ce m e n t To m ake policies enforceable, you should base t hem on a docum ent ed " code of conduct " t o which all em ployees are held account able. Policies m ay need t o be referenced for disciplinary act ion, so linking t hem t o t he code of conduct is a t ransit ive m eans of enforcing behavior. The guidelines and rules developed from t hese policies m ust be det ailed enough t o yield rules t hat can be m onit ored. For exam ple, a policy requiring em ployees t o use encrypt ed channels t o discuss confident ial m at t ers is im possible t o m onit or and enforce because hum an cont ext is necessary t o det erm ine confident ialit y. Cont ext cannot be discovered aut om at ically; it requires det ailed analysis. As an alt ernat ive, a policy t hat requires em ployees t o use encrypt ion when sending m ail t o specific dom ains is enforceable, as it 's possible t o det erm ine encrypt ed channels for m ail prot ocols and t o discover t he dest inat ion of t he t raffic. When select ing policies for m onit oring, don't bot her wit h policies t hat you know m anagem ent won't enforce. For exam ple, your m onit oring syst em m ay be very adept at uncovering use of peer- t o- peer ( P2P) applicat ions on t he net work; it 's well underst ood t hat P2P net working is oft en used t o violat e copyright law and download illegal m at erial. You m ay even have a policy, as m any com panies do, against use of such soft ware on t he office net work. Managem ent , however, m ay not be willing t o rest rict em ployee freedom by enforcing such rules. Lacking enforcem ent , det ect ion of P2P net working and ot her recreat ional t raffic can becom e a dist ract ion from policy m onit oring. Focus inst ead on det ect ing policy violat ions you can assign for act ion. Once you det ect an event , you'll likely have an inform at ion- gat hering st ep t hat allows you t o validat e t he inform at ion and det erm ine furt her det ails about t he source and dest inat ion of t he act ivit y. Once t hat 's com plet e, you'll rout e t he case t o a support t eam t hat can fix t he problem . Em ployees m ust be aware of policies, especially t hose for which you will be t aking act ion. This is best done wit h an awareness cam paign t hat explains what is expect ed and how best t o com ply wit h t he policies. One of t he best ways t o " bake in" policy is t o build t ools and configurat ion guides t hat im plem ent policy det ails. When an em ployee changes a password or configures a web server, t he t ools and configurat ion guidance you've provided will be readily available t o lower resist ance t o com pliance.

2 .5 . Type s of Policie s Two t ypes of policies are used for m onit oring: regulat ory com pliance , which involves adherence t o ext ernally enforced cont rols, and em ployee policies , which govern t he securit y com pliance of em ployees.

2 .5 .1 . Re gu la t or y Com plia n ce Policie s All com panies are bound by som e form of I T legislat ion in t he count ries where t hey conduct business. This legislat ion places obligat ions and rest rict ions on t he com pany, and com pliance wit h t hese rules oft en requires act ive m onit oring. Exam ples of such laws include t he Sarbanes- Oxley Act of 2002 ( SOX) , which requires dem onst rat ion of int egrit y in account ing; t he Healt h I nsurance Port abilit y and Account abilit y Act of 1996 ( HI PAA) , which prot ect s t he privacy of personal healt h inform at ion; and California's Senat e Bill 1386 ( SB1386) , which prot ect s t he privacy of personal inform at ion. I n addit ion t o regulat ory com pliance, adherence t o indust ry st andards is a furt her necessit y , which requires verifiable com pliance wit h set s of best pract ices. Som e m ay be required by business part ners as a m eans of ensuring dat a handling, such as t he Visa PCI st andards. 2 .5 .1 .1 . Ex a m ple : COBI T con figu r a t ion con t r ol m on it or in g Cont rol Obj ect ives for I nform at ion and relat ed Technology ( COBI T) is a set of st andards t hat t he I nform at ion Syst em s Audit and Cont rol Associat ion ( I SACA) and t he I T Governance I nst it ut e ( I TGI ) int roduced in 1992. I T m anagem ent m ay subscribe t o t he cont rol obj ect ives set fort h by COBI T, and require t he developm ent of m onit oring procedures t o m aint ain com pliance. Few of t he cont rol obj ect ives can be effect ively m onit ored in real t im e, but a good exam ple of one t hat can is Design and Support 9 ( DS9) , which requires " m anaging t he configurat ion." I n Sect ion 4 ( DS 9.4) , t he cont rol specifically requires t he verificat ion and audit ing of configurat ion inform at ion. ( COBI T 4.1, I TGI , page 135.) To accom plish t his, you m ust ensure t hat changes execut ed on crit ical syst em s and net work devices have been approved and docum ent ed in a change cont rol syst em . To m onit or for violat ion of such cont rols, you m ust reconcile det ect ed changes t o crit ical syst em s wit h t he records in your configurat ion cont rol reposit ory. This requires access t o t he change logs from t he devices, along wit h elem ent s such as t he t im e of t he change, t he com ponent changed, and t he person who m ade t he change. To reconcile t his inform at ion, you m ust have access t o t he configurat ion cont rol syst em . Consider COBI T configurat ion m onit oring on a com pany's rout ers—som e of t he m ost im port ant and sensit ive devices on t he net work. To m onit or t hese devices for unaut horized changes, we m ust first configure t he rout ers t o log a m essage upon changes t o t he configurat ion. On Cisco I OS rout ers, we can accom plish t his by enabling syslog out put and sending m essages t o an ext ernal syslog collect or for m onit oring and analysis . For exam ple, t he following will set up logging t o a syslog server at 10.83.4.100:

router> enable Password: router# configure terminal router(config)# logging 10.83.4.100 When it 's configured t o enable syslog out put , t he rout er m ust be configured t o m essage upon each configurat ion change. This will t ell us when it was changed, and who changed it :

router(config)# archive router(config-archive)# log config router(config-archive-log-config)# logging enable router(config-archive-log-config)# notify syslog I f it 's set up correct ly, t he rout er will be logging t o our collect or at 10.83.4.100. Let 's t ake a look at t he set up t o see whet her we did it correct ly:

router>show logging Syslog logging: enabled (12 messages dropped, 83 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

Console logging: disabled Monitor logging: level debugging, 0 messages logged, xml disabled, filtering disabled Buffer logging: level informational, 118416 messages logged, xml disabled, filtering disabled Logging Exception size (4096 bytes) Count and timestamp logging messages: disabled No active filter modules. Trap logging: level informational, 118420 message lines logged Logging to 10.83.4.100 (udp port 514, audit disabled, link up), 118420 message lines logged, xml disabled, filtering disabled

Now, every t im e t he configurat ion on t hat device changes, it will send an alert . Here's a sam pling of t he kind of m essage it will generat e upon such changes:

router# show archive log config all idx sess user@line 1 1 mnystrom@vty0 2 1 mnystrom@vty0 3 2 mnystrom@vty0

Logged command | logging enable | logging size 200 | hostname rtp2-prod

By forwarding t hese alert s t o our m onit oring syst em , it will receive an alert every t im e t he configurat ion changes on t his device. Figure 2- 6 shows an exam ple of such an alert . N OTE

To effect ively respond t o incident s generat ed by such logs, t he inform at ion m ust be com plem ent ed by Aut hent icat ion, Aut horizat ion, and Account ing ( AAA) records, which correlat e em ployee aut hent icat ion t o such devices. Figu r e 2 - 6 . Se cu r it y a le r t for con figu r a t ion ch a n ge

We m ust reconcile t he alert in Figure 2- 6 wit h t he configurat ion m anagem ent syst em t o det erm ine whet her it m at ches an approved change record for t hat device. For t his device, it m ust fall wit hin t he approved window and have been execut ed by t he approved im plem ent er. I f t he change is not an approved change, t he analyst m ust engage t he user called out in t he alert m essage, t o see whet her t he change can be explained. A problem on t he

device m ay have required an adm inist rat or t o t roubleshoot or m ake t em porary changes t o t he device. Such changes would not be reflect ed in an approved change request , and should be easy t o reference in a t rouble t icket docum ent ing t he problem . I f t he change cannot be reconciled, t his is a policy violat ion and a report ing procedure m ust be execut ed. N OTE Alert ing and responding t o unexpect ed configurat ion changes is a lot of t rouble. To avoid t riggering incident s t oo frequent ly, t une your alert ing wit h generous except ions t hat allow for t im e surrounding approved change request s, docum ent ed out ages, and known m aint enance windows. You should also not ify t he support st aff t hat t heir configurat ion changes are being m onit ored, and t hat t hey will be required t o provide a j ust ificat ion for out of- policy changes. This is like a securit y cam era in a convenience st ore—it brings t he posit ive side effect of guiding com pliance wit h policy. Figure 2- 7 provides an exam ple of an approved request . Not e t he im port ant elem ent s of Figure 2- 7 :

Device nam e I m plem ent er ( user I D) Tim e window ( St art Tim e and End Tim e colum ns) Approval st at us We can now m onit or t hese elem ent s by wat ching for device changes t hat fall out side t his t im e window or were not perform ed by t his aut horized user.

Nearly all user- based m onit oring presupposes t hat t he user I D is unique t o t he user. I f it 's a shared user I D, it will likely prove very difficult t o t race t he act ivit y t o an individual wit hout a lot of ext ra work, such as t racing t o t he source I P address of t he act ivit y.

Figu r e 2 - 7 . Scr e e n ca pt u r e of ch a n ge m a n a ge m e n t syst e m

2 .5 .1 .2 . Ex a m ple : SOX m on it or in g for fin a n cia l a pps a n d da t a ba se s Sect ion 404 of SOX requires I T m anagem ent t o im plem ent cont rols on applicat ions t hat are a key part of t he com pany's financials. This is t o m it igat e t he pot ent ial risk of m isst at ing t he com pany's financial num bers. To ensure t he int egrit y of financial applicat ions, especially t hose wit h direct relevance t o SOX cont rols, you m ust m onit or for dat a int egrit y wit hin such applicat ions. A rat ional policy would prohibit all direct dat abase access for such sensit ive applicat ions, except for scheduled adm inist rat ion by adm inist rat ors ( and possibly dat a m aint enance for applicat ion changes, archiving, et c.) . To be pract ical, t he allowed host s and users, along wit h t he approved t im e windows, m ust be docum ent ed in a policy. Securit y m onit oring could em ploy net work t raffic analysis t o wat ch for act ivit y t hat falls out side t he approved policy. 2 .5 .1 .3 . Ex a m ple : M on it or in g H I PAA a pplica t ion s for u n a u t h or ize d a ct ivit y Tit le I I of HI PAA addresses securit y and privacy of healt h dat a. Am ong m any ot her safeguards, it st at es t hat

" I nform at ion syst em s housing [ prot ect ed healt h inform at ion] m ust be prot ect ed from int rusion. When inform at ion flows over open net works, som e form of encrypt ion m ust be ut ilized. I f closed syst em s/ net works are ut ilized, exist ing access cont rols are considered sufficient and encrypt ion is opt ional." To m onit or for com pliance wit h t his safeguard, you could posit ion a dat a capt ure device, such as a NI DS, on t he net work t o ensure t hat dat a leaving t he applicat ion/ syst em is encrypt ed before leaving t he closed net work. Many NI DSs can det ect unencrypt ed dat a over a supposedly encrypt ed channel, and could alert on discovery of such dat a. Tracing t his t o t he source of t he connect ion would allow you t o log t his as a policy violat ion. 2 .5 .1 .4 . Ex a m ple : I SO 1 7 7 9 9 m on it or in g I SO 17799 ( a.k.a. I EC 27002) is an inform at ion securit y code of pract ice, covering best pract ice recom m endat ions on inform at ion securit y m anagem ent for use by t hose who are responsible for init iat ing, im plem ent ing, or m aint aining I nform at ion Securit y Managem ent Syst em s ( I SMSs) . [ 8 ] An obj ect ive of I SO 17799 is sect ion access cont rol, described in Sect ion 8, which requires t hat you prevent unaut horized access t o inform at ion syst em s. [ 8]

ht t p: / / www.iso.org/ iso/ iso_cat alogue/ cat alogue_t c/ cat alogue_det ail.ht m ?csnum ber= 39612

One challenge facing ent erprises t rying t o secure syst em s from unaut horized access is t he generic account . Shared applicat ions such as web servers, applicat ion servers, and dat abases run as a generic user account , t ypically wit h lim it ed privileges. Unless t hey are secured and m onit ored, t hese account s can becom e weak point s in im plem ent ing t his policy. Adm inist rat ors m ay need t o occasionally log int o t hese account s t o updat e applicat ions. Monit oring for unaut horized access will require you t o m onit or syst em logs and audit syst em configurat ions. Syslog from t he syst em s you wish t o m onit or will allow you t o det ect syst em access and changes. To det ect account access, you should configure t he syst em t o det ect login and sudo access t o t he generic account s. Then, you should reconcile t hese changes against approved change request s, as described in t he previous exam ple. An audit ing t echnique could involve checking snapshot s of device configurat ions, specifically, t he account regist ry such as /etc/passwd , t o ensure t hat direct login is disabled for such account s and t hat sudo is appropriat ely rest rict ed. 2 .5 .1 .5 . Ex a m ple : Pa ym e n t Ca r d I n du st r y D a t a Se cu r it y St a n da r d ( PCI D SS) m on it or in g Maj or credit card com panies developed PCI DSS as a guideline t o help organizat ions t hat process card paym ent s prevent credit card fraud, hacking, and various ot her securit y vulnerabilit ies and t hreat s. A com pany processing, st oring, or t ransm it t ing paym ent card dat a m ust be PCI DSS- com pliant or risk losing it s abilit y t o process credit card paym ent s. [ 9 ] [ 9]

ht t ps: / / www.pcisecurit yst andards.org/ securit y_st andards/ pci_dss.sht m l

PCI DSS requires t he prot ect ion of cardholder dat a. As an exam ple, Requirem ent 4 of t he st andard st at es t hat you m ust " Encrypt t ransm ission of cardholder dat a across open, public net works." To effect ively m onit or for com pliance, an organizat ion could set up a NI DS t o det ect unencrypt ed dat a em anat ing from t he applicat ions used for card processing. For addit ional prot ect ion, addit ional signat ures could im plem ent a regular expression m at ch t o find credit card num bers t raversing t he net work. You can use a regex pat t ern such as t his:

^((4\d{3})|(5[1-5]\d{2}))(-?|\040?)(\d{4}(-?|\040?)){3}|^(3[4,7]\d{2}) (-?|\040?)\d{6}(-?|\040?)\d{5} t o det ect t he com m on num eric pat t ern used by credit cards. Wit h t his regular expression, a NI DS ( or any packet capt ure device wit h a full copy of net work t raffic) could cat ch unencrypt ed t ransm ission of t hese card num bers. Should t he NI DS det ect such a pat t ern, it 's a likely violat ion of t he encrypt ion policy out lined by Sect ion 4 of PCI DSS.

2 .5 .2 . Em ploye e Policie s Com pliance- based policies are developed from wit hout , and drive an organizat ion's policies, audit ing, and m onit oring. I n cont rast , em ployee policies are t ypically derived from t he com pany's et hical st andards or code of business conduct , and are oft en set by hum an resources, legal, and inform at ion securit y depart m ent s. These policies are designed t o place som e lim it s on em ployees t o preserve et hical st andards, lim it financial loss, prot ect corporat e revenue, and m aint ain a safe working environm ent . 2 .5 .2 .1 . Ex a m ple : Un iqu e login for pr ivile ge d ope r a t ion s

To m aint ain account abilit y, an organizat ion m ust be able t o correlat e act ivit y t o an individual em ployee. For t he organizat ion t o do t his effect ively, it m ust require em ployees t o use unique logins when accessing shared resources. Especially when perform ing privileged operat ions, em ployers m ust ensure t hat sensit ive operat ions are at t ribut able t o an individual. To accom plish t his t ask, an em ployer needs policies t hat require em ployees t o use t heir individually assigned account s when accessing shared resources such as Unix servers, dat abase servers, and so on. When em ployees are perform ing privileged operat ions, em ployers should direct t he em ployees t o first log in wit h t heir individual account s before " swit ching user" for privileged operat ions. On Unix, sudo is t he recom m ended com m and for users execut ing privileged operat ions; on an Oracle dat abase, a user can connect as a privileged account ( such as SYSTEM) . Upon exam ining t he log m essages, t he securit y analyst can t race t he act ions direct ly t o t he individual execut ing t he com m ands. Shared account s should never be accessible direct ly, as t hey obscure t he t rue ident it y of t he individual. On a Unix server, a sim ple m et hod for m onit oring violat ions of t his policy requires only t hat t he server record each login via syslog. The m onit oring applicat ion or st aff m ust t hen screen for inst ances of t he user " root " direct ly logging int o t he syst em , as shown in t he following code snippet , and conduct furt her invest igat ion about t he act ivit y :

Mar 28 16:19 bw-web1 sshd(pam_unix)[13698]: session opened for user root by (uid=0)

N OTE Should you require securit y logs t o invest igat e a m alicious incident , act ivit y will likely correlat e t he m ost dam aging event s t o t he privileged user ( root or Adm inist rat or) . Tracing t he act ivit y back t o an individual will require t hat backt racking t hrough t he logs t o correlat e it wit h a login by a unique user. I f t he privileged user logged in direct ly, t he invest igat ion m ust t ake on m uch m ore com plexit y. 2 .5 .2 .2 . Ex a m ple : Rogu e w ir e le ss de vice s The insecure wireless net work at a Marshall's discount clot hing st ore near St . Paul, Minnesot a, m ay have allowed high- t ech at t ackers t o gain a beachhead in ret ail giant TJX Com panies' com put er net work, result ing in t he t heft of inform at ion on at least 45.6 m illion credit and debit cards, according t o t he Wall St reet Journal . [ 1 0 ] [ 10]

ht t p: / / www.securit yfocus.com / brief/ 496/

Though it 's not clear whet her t he problem at Marshall's was a rogue wireless device or j ust a poorly configured wireless deploym ent , it 's clear t hat wireless t echnology ext ends ent erprise net works well beyond t he guarded perim et er. Wireless access point s are cheap, are easy t o connect , and can creat e a dangerous back door int o t he organizat ion if t hey are not properly configured. Securit y professionals m ust vigilant ly drive rogue wireless devices out of t he corporat e net work. Technologies now exist t o allow t he wireless infrast ruct ure t o defend it self by denying access t o rogue wireless devices. Reliably finding and blocking rogue devices from t he net work requires sophist icat ed infrast ruct ure. Corporat e policy m ust require wireless devices t o be properly configured and deployed, and only by t he I T st aff. A st andard should provide det ails for secure configurat ion, and t he key t o successful m onit oring is t o wat ch for wireless devices t hat do not m eet t his st andard. Several Wireless I nt rusion Det ect ion Syst em s ( WI DSs) are available, which use an overlay net work of RF det ect ors t o discover wireless access point s. These can be correlat ed t o net work locat ion, allowing t he net work securit y m onit or t o com pare t he net work locat ion wit h t hose of approved wireless access point s. Effect ive use of t his solut ion requires a " regist rat ion" of approved wireless access point s. Docum ent ing approved, support ed services, and t heir net work locat ion is a vit al part of m aking policy m onit oring work, and we cover it in great er dept h in Chapt er 3 . N OTE WI DSs st ore a dat abase of aut horized access point s as a basis for det ect ion of rogue devices. WI DSs use a variet y of m eans t o ident ify access point s, including MAC addresses, I P addresses, and RF fingerprint s. The t echnology t o det ect a unique RF signat ure was developed at Carlet on Universit y in Ot t awa, Canada, in 2006. I n Figure 2- 8 , t he WI PS cont roller ( a.k.a. WI DS cont roller; t he P st ands for prevent ion ) m aint ains a dat abase of aut horized access point s. The L3 swit ch has soft ware t o read t he report ed access point fingerprint s via t he connect ed access point s. These fingerprint s are st ored and analyzed in t he WI PS cont roller, which will t ell t he L3 swit ch t o deny I P connect ivit y t o t he rogue access point . Figu r e 2 - 8 . A w ir e le ss in t r u sion pr e ve n t ion syst e m ( W I PS)

2 .5 .2 .3 . Ex a m ple : D ir e ct I n t e r n e t con n e ct ion fr om pr odu ct ion se r ve r s I n Richard Bej t lich's book, Ext rusion Det ect ion ( Addison- Wesley Professional) , Bej t lich describes t he im port ance of wat ching out bound t raffic, t hat is, connect ions init iat ed from inside t he com pany t oward t he I nt ernet . Doing so, he argues, allows t he net work securit y m onit oring st aff t o wat ch for com m unicat ions wit h known com m and and cont rol servers, a sure sign of a com prom ised asset . Bej t lich argues t hat product ion servers should be wat ched closely for t heir connect ions: I n som e cases, det ect ing any out bound connect ion at all is a sign of com prom ise. For exam ple, a properly adm inist ered web server should never have t o m ake an out bound connect ion t o any foreign host . ( By 'foreign,' I m ean a m achine not associat ed wit h t he local organizat ion.) DNS queries should be sent t o t he local DNS server. Updat es should be pushed t o t he syst em from a local pat ch or package builder, or pulled from t he local pat ch or package builder. [ 1 1 ] [ 11]

Ext rusion Det ect ion , Chapt er 3, pages 92 and 93.

Connect ions t o t he I nt ernet from product ion servers should be rare and well docum ent ed. Soft ware updat es can be st aged on local servers, and adm inist rat ors should use personal syst em s t o check m ail or research t echniques t o fix problem s. I f a product ion server has been com prom ised, it will soon begin t o shut t le inform at ion t o a server under t he hacker's cont rol. Server policy should t herefore forbid product ion servers from init iat ing I nt ernet connect ions. Once net work securit y

m onit oring is in place, you can set it up t o cat ch any such act ivit y. I nit ially, m any of t he discovered incident s will prove t o be accident al usage, and can be m it igat ed accordingly as policy violat ions. Once m onit oring is fully im plem ent ed, it will prove t o be an early warning syst em for cat ching com prom ised servers and will be a surprisingly easy and effect ive t ool for securing syst em s. Session dat a records such as Net Flow provide t he perfect t ool for such m onit oring, and we cover t hem in m ore det ail in Chapt er 3 . For now, t hink of Net Flow as being sim ilar t o a phone bill; it list s connect ions bet ween I P addresses and port s, wit h a record of when and for how long t hey occurred ( see Figure 2- 9 ) . You can analyze Net Flow recorded at perim et er gat eways t o discover dat a cent er subnet s init iat ing connect ions t o I nt ernet addresses. A single connect ion is a clear policy violat ion, and you can invest igat e and m it igat e it accordingly. Figu r e 2 - 9 . N e t Flow r e cor ds in dica t in g con n e ct ion s

2 .5 .2 .4 . Ex a m ple : Tu n n e le d t r a ffic I nt ernet Relay Chat ( I RC) was developed in 1988 as an I nt ernet chat program . I RC servers allow for m any sim ult aneous conversat ions and groups via t he use of channels. Since servers and channels are long- lived, users can connect and disconnect at will, and can part icipat e in a never- ending conversat ion wit h ot her users on t he channel. I n recent years, I RC has becom e a popular t ool for t eam s t o com m unicat e, as all conversat ion is open and readable by all users on t he channel. Because of it s server/ channel design and support for program m at ic access, I RC has becom e t he de fact o applicat ion for hackers t o rem ot ely cont rol t heir com prom ised syst em s. Com m only used for cont rolling bot net s, I RC is widely used as t he cont rol channel for progressively sophist icat ed applicat ion program m ing int erfaces ( API s) . Due t o it s com m on use as a hacker cont rol plane, default I RC net work port s are com m only blocked by firewalls and reliably discovered by NI DS sensors. To circum vent t hese cont rols, hackers have begun t o disguise and even encrypt t he t raffic. Securit y researcher Dan Kam insky has fam ously writ t en and spoken on t he t opic of t unneling inform at ion over DNS several t im es in t he past few years, and has int roduced t ools such as Ozym anDNS t o m ake it easy. Bot net I RC t raffic usually has an inst ant ly recognizable form at . I t cont ains random ly generat ed usernam es wit h em bedded num eric ident ifiers, along wit h an indicat ion of geographic locat ion. The form at looks like t his:

PASS NICK USER MODE JOIN

letmein [M00|JOR|05786] XP-4132 * 0 :owned-host [M00|JOR|05786] -ix #

The bot net cont roller uses t his inform at ion t o ident ify and com m and his bot s. I n cont rast , user- direct ed I RC t raffic will cont ain m ore hum an- readable usernam es and host nam es. I f you have business needs t hat require access t o I RC servers on t he I nt ernet ( which prevent you from blocking I RC t raffic alt oget her) , you can deploy int rusion det ect ion t echniques t o det ect t hese pat t erns and block t he I RC bot net t raffic. This, of course, requires analysis and m onit oring procedures t o discern bet ween m alicious and benign t raffic. Tools t hat inspect net work t raffic can discover t ellt ale signs of t unneling, norm ally by checking t he t raffic against t he expect ed st ruct ure for t he prot ocol assigned t o t hat port . You also can apply policy- based m onit oring t o discover t his t raffic. One m eans for discovering such t unneling is t o det ect DNS request s originat ing from locat ions ot her t han t he DNS servers in t he ent erprise. By way of policy, client s should only use int ernal DNS servers for nam e resolut ion. I nt ernet - bound DNS t raffic originat ing from host s ot her t han t hese DNS servers would be a policy violat ion, and a possible sign of t unneled t raffic. Upon det ect ing t his t raffic, you could t hen analyze it t o det erm ine whet her it 's t ruly dangerous or is an anom aly t hat requires furt her t uning of net work m onit ors. Here's an exam ple of a Snort alert indicat ing pot ent ial DNS t unneling, requiring furt her invest igat ion:

Apr 2818:34:15 [**] snort[1000001]: Potential DNS Tunneling [**] [Classification: Potentially Bad Traffic] [Priority: 2] (UDP) 31.33.73.57:22652 ->10.13.55.7:53

2 .6 . Policie s for Bla n co W ir e le ss The fict it ious com pany Blanco Wireless will serve as a plat form t o illust rat e t he st ages and t echniques of im plem ent ing policy m onit oring. As part of account adm inist rat ion, Blanco m ust st ore sensit ive inform at ion such as Social Securit y num bers and direct billing det ails. Due t o t he sensit ive nat ure of such inform at ion, Blanco has developed several policies t o prot ect it self and it s cust om ers' dat a.

2 .6 .1 . Policie s Blanco em ploys t he following policies t o m aint ain com pliance wit h governm ent regulat ions, safeguard it s m ost sensit ive dat a, and provide invest igat ive support should any of t he dat a be com prom ised. These are, of course, not exhaust ive. Rat her, t hey serve as illust rat ions for how t o apply policy m onit oring. 2 .6 .1 .1 . D a t a Pr ot e ct ion Policy I n keeping wit h California law and Blanco Wireless's com m it m ent t o cust om er privacy, em ployees are required t o m aint ain st rict confident ialit y of all personally ident ifiable inform at ion ( PI I ) :

Scope This applies t o all PI I st ored on product ion servers in t he Blanco Wireless net work.

Configurat ion requirem ent s

St orage PI I m ust be encrypt ed in st orage and t ransm it t ed only over encrypt ed net work connect ions.

Access Dat abases cont aining PI I m ust be accessed only via an approved m et hod:

An applicat ion whose purpose is t o broker such access. An approved dat abase m anagem ent server. Direct dat abase access via deskt op program s such as TOAD is st rict ly prohibit ed.

Dat abase securit y Dat abases st oring PI I m ust be configured according t o Blanco Wireless's Dat abase Configurat ion Guide, which prescribes t he applicat ion of all severit y 1 and 2 direct ives cont ained wit hin Pet e Finnigan's Oracle Dat abase Securit y Checklist .[ 1 2 ] [ 12]

ht t p: / / www.sans.org/ score/ checklist s/ Oracle_Dat abase_Checklist .pdf

2 .6 .1 .2 . Se r ve r Se cu r it y Policy The purpose of t he Server Securit y Policy is t o est ablish st andards for t he base configurat ion of int ernal server equipm ent t hat Blanco Wireless owns and/ or operat es. Effect ive im plem ent at ion of t his policy will m inim ize unaut horized access t o Blanco's propriet ary inform at ion and t echnology:

Scope This policy applies t o all product ion servers at Blanco Wireless, including web, applicat ion, and dat abase servers.

Configurat ion requirem ent s The following direct ives are required of all servers at Blanco, and should be det ailed in every configurat ion or " hardening" guide used by adm inist rat ors:

Out bound connect ions Product ion servers deployed int o Blanco dat a cent ers m ay init iat e connect ions only t o ot her product ion servers wit hin t he Blanco ent erprise. Connect ions m ay never be init iat ed from product ion servers t o t he I nt ernet . I f deployed int o an approved DMZ, servers m ay only respond t o request s in accordance wit h t heir deployed funct ion.

Net work services Servers in t he dat a cent er m ust regist er t heir host ed net work services in t he Blanco Ent erprise Managem ent Syst em prior t o deploym ent . Regist rat ion is not required for approved m anagem ent services such as SSH and syslog, which are necessary for adm inist rat ion and m onit oring.

Account s Servers m ust be configured t o lim it available account s t o t hose who have direct , adm inist rat ive responsibilit y on each specific server. Servers m ust never allow direct login by nonadm inist rat ive personnel such as developm ent support st aff m em bers or end users. Those wit h a need t o affect applicat ion or dat a configurat ion should use t he I T- support ed deploym ent applicat ions t o effect needed changes on t he servers.

Privileged access Direct login t o t he syst em m ust occur t hrough individually assigned account I Ds. Access t o privileged account s, such as root , m ust be perm it t ed only via " swit ch user" com m ands such as sudo or su upon successful login. Direct login as a generic privileged account such as root is st rict ly prohibit ed.

Adm inist rat ion Rem ot e adm inist rat ion m ust be perform ed only over crypt ographically secured, encrypt ed net work connect ions. Adm inist rat ive int erfaces m ust never be exposed direct ly t o t he I nt ernet , but m ust be available only t o t hose who have properly aut hent icat ed int o t he Blanco Wireless net work.

Logging The following event s m ust be logged t o a separat e log server t o support invest igat ions and m onit oring:

Login Privileged operat ions, including sudo and su St at us change for net worked services ( st art / st op)

2 .6 .2 . I m ple m e n t in g M on it or in g Ba se d on Policie s I n t he next few chapt ers, we will det ail a m et hod for analyzing and docum ent ing Blanco's net work, find t he best areas t o t arget our m onit oring, and provide pract ical guidance on how t o deploy m onit oring specifically t o wat ch for policy violat ions. Based on t he Blanco policies out lined in t his chapt er, here are t he specific it em s we will m onit or t o effect policy m onit oring:

St orage/ t ransm ission of PI I We will m onit or dat a cent er gat eways t o wat ch for signs t hat Social Securit y num bers are being t ransm it t ed over unencrypt ed links.

Dat a access We will m onit or for unaut horized SQL* Net connect ions int o our sensit ive dat abases.

Out bound connect ions We will m onit or for connect ions init iat ed from sensit ive servers t o ensure t hat t hey are approved except ions.

Dat abase securit y We will audit TNS list eners t o ensure t hat t hey m eet hardening crit eria.

Net work services We will audit open port s on servers t o ensure t hat t hey are regist ered as required by t he Server Securit y Policy, referenced in t he preceding sect ion.

Account s We will m onit or syslog for account logins and com pare t hem against a dat abase of syst em adm inist rat ors t o ensure t hat logins are proper and aut horized.

Privileged access We will m onit or syslog for direct privileged logins t o servers, and m onit or SQL* Net connect ions for direct syst em login t o dat abases.

Adm inist rat ion We will audit product ion servers from out side t he com pany t o uncover any rem ot e adm inist rat ion prot ocols ( such as SSH, VNC, and Rem ot e Deskt op) , a violat ion of t he Server Securit y Policy.

2 .7 . Con clu sion There are a wide variet y of approaches for select ing t he policies t o m onit or. Once policies are select ed, you m ust det erm ine t he set t ing—t he environm ent in which t hese policies are t o be applied. You'll need a reliable m ap of your net work, one t hat highlight s t he underlying funct ions, applicat ions, and users. I n t he next chapt er, we'll explain how t o develop and docum ent a cont ext ual underst anding of your own net work.

Ch a pt e r 3 . Kn ow You r N e t w or k I m agine going t o bat t le wit hout an underst anding of t he t errain, roads, buildings, weat her, or even your own fight ing force's t act ics and capabilit ies. This is t he sit uat ion faced by m any inform at ion securit y professionals when t hey init ially at t em pt t o m onit or t heir net work environm ent . Knowing your net work is akin t o underst anding your m ilit ary capabilit ies, st rengt hs, and weaknesses when preparing for an enem y at t ack. I n inform at ion securit y, t he enem y will change t act ics cont inually, but you have a " hom e field advant age" because t he bat t leground is your net work . Hist ory proves t hat blindly charging int o or defending t he unknown will alm ost cert ainly end in defeat . One of t he best ways t o express t his concept com es from Richard Bej t lich, inform at ion securit y professional and aut hor of The Tao of Net work Securit y Monit oring. I n a January 2007 post on his blog, [ 1 3 ] Bej t lich describes t he " Self- Defeat ing Net work" as having t he following charact erist ics: [ 13]

ht t p: / / www.t aosecurit y.blogspot .com / 2007/ 01/ self- defeat ing- net work.ht m l

Unknown Unm onit ored Uncont rolled Unm anned Trust ed Alt hough you m ay not have cont rol of or influence over t hese charact erist ics, you m ust m ake every effort t o Know Your Net work! Doing so will help you succeed in m ost of your securit y- relat ed endeavors. I n t his chapt er, we will explore t wo prim ary m et hods of learning about a net work: net work t axonom y and net work t elem et ry.

3 .1 . N e t w or k Ta x on om y I m agine you receive a report from your m onit oring st aff t hat " I P address 10.10.10.20 was seen perform ing a buffer overflow at t ack against I P address 10.50.10.43." What does t his m ean? Do t hese I P addresses belong t o you? I f so, have you even deployed t hat range of addresses? Are t hey on a product ion net work, lab net work, ext ranet / part ner net work, or dat a cent er net work? I s t his address space NAT'd or proxied? The answers t o t hese quest ions direct ly im pact how you will respond t o t he report ; t hey det erm ine ownership, securit y policy, crit icalit y, and whet her a response is required. To efficient ly process securit y alert s, you m ust build t his cont ext int o t he alert s dat a as it is generat ed. Knowing your net work will also im prove ot her areas of incident response, enabling you t o " separat e t he wheat from t he chaff." You can use t he net work t axonom y dat a you've collect ed and enum erat ed for your specific environm ent t o provide cont ext ual inform at ion relat ed t o each securit y incident . You can furt her use it t o populat e obj ect groups in your SI M and in your int rusion det ect ion, access cont rol, net work analysis ( anom aly det ect ion) , and process aut om at ion t asks. First , however, you m ust classify your net work t ypes and t hen enum erat e t hem in a m eaningful way. Let 's st art wit h I P addresses.

3 .1 .1 . N e t w or k Type Cla ssifica t ion I P net work t ype classificat ion refers t o t he process of ident ifying your organizat ion's varying net work t ypes along wit h t heir unique at t ribut es. This process varies widely from com pany t o com pany, and even am ong divisions wit hin an organizat ion. I f you were t o consult 10 different net work vendors you would get 10 different explanat ions of t his process. Rat her t han t rying t o describe a " one- size- fit s- all" explanat ion of what 's involved, we'll discuss a fram ework for net work classificat ion based on net work funct ion . A net work's funct ion is t he best charact erist ic for defining securit y policy governing a net work. A net work's funct ional descript ion should explain who t he net work's end users are. Each net work will be configured t o govern users' behavior wit h a variet y of securit y policies, role- based access, and opt im izat ions t hat will not fit int o a sim ple fram ework. Therefore, you m ust concent rat e on discovering only t he general funct ion of a net work when applying a classificat ion t o it . Figure 3- 1 is an exam ple of net work classificat ion. N OTE

The net work t ype classificat ion principles we'll discuss focus on ent erprise net works. However, you can easily apply t hem t o service provider net works as well. Be aware, however, t hat when classifying net works, service providers oft en favor aut onom ous syst em [ 1 4 ] groupings inst ead of funct ional groupings. [ 14]

ht t p: / / www.light reading.com / docum ent .asp?doc_id= 4054 Figu r e 3 - 1 . N e t w or k cla ssifica t ion

3 .1 .1 .1 . Ex t e r n a l n e t w or k s I n our net work classificat ion fram ework, ext ernal net works represent t hose t hat reside out side t he corporat e firewall. Depending on t he size, needs, and charact erist ics of your organizat ion, you likely have a num ber of net works bet ween t he I nt ernet and your int ernal net work. To sim plify our discussion and t he fram ework, we'll assum e a m edium - sized business wit h a DMZ ( also called a point of dem arcat ion or perim et er ) net work presence bet ween t he I nt ernet and t he corporat e firewall. The I nt ernet net work classificat ion should be t he last one you define; it will consist of everyt hing t hat falls out side

your ot her net work classificat ions. This giant " net work of net works" includes your cust om ers, com pet it ors, organized crim inals, governm ent ent it ies , and everyt hing in bet ween…and you'll rarely be able t o ident ify one from anot her ! The DMZ net work classificat ion describes t he net work ( or, in som e cases, collect ion of net works) t hat cont ains your e- com m erce host s and syst em s relat ed t o core business services such as SMTP and DNS. This net work classificat ion will have a few subclassificat ions based on t he st rict er securit y policies required for t he syst em s exposed on it , since t he prim ary funct ion of t he DMZ is t o allow services reachable from t he I nt ernet . 3 .1 .1 .2 . I n t e r n a l n e t w or k s I n our net work classificat ion fram ework, int ernal net works represent t hose t hat reside inside t he perim et er, behind t he corporat e firewall. I nt ernal net work classificat ion is t he m ost com plex of t he t hree m ain classificat ions; it 's com posed of several core subclassificat ions relat ed t o t he following funct ions, m ost com m only found in ent erprise net works:

Dat a cent er A dedicat ed, highly available net work environm ent for crit ical com put e services including diverse services such as code developm ent , account ing syst em s, and I P t elephony. Usually support ed and operat ed by a dedicat ed corporat e I T group.

Ext ranet Connect ions bet ween your int ernal net work and your vendors, suppliers, and part ners, t ypically wit h lim it ed access. Usually support ed and operat ed by corporat e I T, wit h support lim it ed t o t he boundary of dem arcat ion bet ween t he corporat ion and t he ext ernal ent it y.

Rem ot e access Net works providing connect ivit y t o users from out side t he corporat e firewall. Access is provided via legacy dial- based services ( I SDN, m odem ) as well as newer I P- based solut ions t raversing t he public I nt ernet , such as virt ual privat e net works ( VPNs) . Rem ot e access net works are usually support ed and operat ed by corporat e I T.

Lab Net works used for t est ing and developm ent . These are t ypically unt rust ed ( t o t he corporat e net work) , lack securit y cont rols, and are not support ed or operat ed by corporat e I T.

Deskt op and wireless St andardized net works t hat provide connect ivit y t o em ployees while at a corporat e worksit e. Usually support ed and operat ed by corporat e I T. Fort unat ely, m any I T t eam s have already conduct ed m uch of t he net work enum erat ion and classificat ion required for our solut ion. The best and m ost com m on source of such invaluable dat a is cont ained wit hin your exist ing I P address m anagem ent ( I PAM) solut ion . [ 1 5 ] [ 15]

ht t p: / / encyclopedia2.t hefreedict ionary.com / I PAM/

3 .1 .2 . I P Addr e ss M a n a ge m e n t D a t a Hist orically, gat hering dat a about your net work required soft ware t ools t hat discovered what was " out t here" in your infrast ruct ure by blindly scanning your I P address ranges. Com m ercial t ools such as HP OpenView and open source soft ware such as Cheops gat her such dat a using a com binat ion of Sim ple Net work Managem ent Prot ocol ( SNMP) queries, supplem ent ing t he dat a wit h net work port scans t o discover t he net work. Such t ools can ident ify device t ypes, m ap net work t opologies, and recognize device OS and applicat ion det ails. However, t hey do not provide t he cont ext ual inform at ion needed t o priorit ize t heir collect ed dat a, except via painst aking m anual annot at ion .

I n light of t he lim it at ions of such aut om at ed discovery t ools, we t urn t o an I PAM solut ion for ext ract ing cont ext ual dat a. I PAM is m ore t han a t ool; it also refers t o a fram ework for m anaging and allocat ing t he net work address space for syst em ent it ies. Securit y is not oft en associat ed wit h I P address m anagem ent , as t he t ools are built prim arily for t he com plex t ask of allocat ing I P address space wit hin a com pany's diverse net works. I P address m anagem ent solut ions norm ally ext end int o DNS and DHCP services. Several open source and com m ercial solut ions exist t o st ore I PAM dat a, but a useful solut ion for m onit oring incident response m ust include places t o st ore t he following at t ribut es for each I P address:

I P subnet and m ask Owner/ cont act inform at ion Subnet descript ion Subnet funct ion Subnet locat ion Subnet parent Hierarchical present at ion Figure 3- 2 shows an exam ple of I P addresses st ored in Cisco Net work Regist rar—a com m ercial I PAM solut ion. Exam ple 3- 1 dem onst rat es t he represent at ion of net work hierarchy and subnet t agging m ade possible by an I PAM solut ion. Figu r e 3 - 2 . Cisco N e t w or k Re gist r a r

Ex a m ple 3 - 1 . Ex a m ple I PAM da t a

10.9.0.0/16 SITE1 Campus |-- 10.9.0.0/19 Data Centers |-- 10.9.32.0/19 Site 1 Desktop Networks | |-- 10.9.34.0/24 Building 1 3rd floor | |-- 10.9.57.0/24 Building 4 3rd Floor | | |-- 10.9.57.0 - Network | | |-- 10.9.57.26 userid-win2k.blanco.com Network Prefix/Length 10.9.57.0/24 Parent Block 10.9.32.0/19 Description Building 4 3rd Floor Type Subnet Status Active Function Desktop LAN Location SITE1 N OTE

I f your I T net working t eam has done a good j ob wit h I P address m anagem ent and classificat ion, buy t hem a beer! They've j ust m ade your life a whole lot easier! I n Exam ple 3- 1 , you saw t hat 10.9.0.0/ 16 is an address block assigned t o t he SI TE1 Cam pus . This block cont ains dat a cent er subnet s and deskt op subnet s. You can use t his inform at ion t o provide cont ext for NI DS alert s, anom aly det ect ion dat a, Net Flow, and anyt hing else t hat provides an I P address but lacks cont ext ual inform at ion. For exam ple, using t he I PAM dat a in Exam ple 3- 1 , which of t he following t wo alert s ( shown in Exam ples Exam ple 3- 2 and Exam ple 3- 3 ) would you t reat wit h higher priorit y? Ex a m ple 3 - 2 . N I D S a le r t 1

evIdsAlert: eventId=1196869618127124428 severity=high vendor=Cisco originator: hostId: ids-1 appName: sensorApp appInstanceId: 462 time: 2008/01/26 15:41:54 2008/01/26 15:41:54 UTC signature: description=B02K-UDP id=4055 version=S2 subsigId: 2 marsCategory: Penetrate/Backdoor/Trojan/Connect interfaceGroup: vs0 participants: attacker: addr: locality=OUT 10.100.99.244 port: 4500 target: addr: locality=IN 10.9.57.26 port: 4500 Ex a m ple 3 - 3 . N I D S a le r t 2

evIdsAlert: eventId=1196869618127124428 severity=high vendor=Cisco originator: hostId: ids-1 appName: sensorApp appInstanceId: 462 time: 2008/01/26 15:41:54 2008/01/26 15:41:54 UTC signature: description=B02K-UDP id=4055 version=S2 subsigId: 2 marsCategory: Penetrate/Backdoor/Trojan/Connect interfaceGroup: vs0 vlan: 0 participants: attacker: addr: locality=OUT 10.100.99.244 port: 4500 target: addr: locality=IN 10.9.1.12 port: 4500

According t o t he I PAM dat a in Exam ple 3- 2 , alert 1 shows t hat t he t arget of t he at t ack is in Building 4 of Sit e1's deskt op net works, probably a user PC. Alert 2 in Exam ple 3- 3 shows t hat when I PAM dat a is t aken int o considerat ion, t he t arget of t he at t ack is in a dat a cent er, m aking t his alert a higher priorit y t han alert 1. Alt hough t his exam ple is great ly sim plified, it st ill shows t he power of adding cont ext t o alert s wit h I PAM dat a. To drive t his point furt her, which of t he following warnings would you rat her receive?

"IP address 10.100.99.244 attempted a B02K-UDP attack against 10.9.1.12" or :

"IP address 10.100.99.244 attempted a B02K-UDP attack against a datacenter server at 10.9.1.12"

I f possible, get access t o t he underlying dat abase of t he I PAM applicat ion t o allow direct dat a queries; it 's m uch quicker t han using a GUI . I t will not only save t im e, but it will also assist you wit h aut om at ing host dat a collect ion. Even if your I PAM dat a is not in an applicat ion, but rat her is m aint ained in an Excel spreadsheet , it 's st ill a valuable cont ext for your incident dat a. A spreadsheet , however, is difficult t o keep up- t o- dat e and lacks cachet as an aut horit at ive source of dat a. Rat her, we highly recom m end an I PAM solut ion t hat is scalable and can be kept up- t odat e easily. Chapt er 6 will discuss I PAM dat a in m ore det ail, leveraging it for t uning your I PS and SI M t ools. To get you st art ed, here are som e of t he m ore popular I PAM product s on t he m arket t oday:

BlueCat Net works Prot eus/ Adonis ( ht t p: / / www.bluecat net works.com / ) I ncognit o I P/ Nam e/ DNS Com m ander ( ht t p: / / www.incognit o.com / ) BT I NS I PCont rol ( ht t p: / / www.ins.com / ) Carnegie Mellon Net Reg ( ht t p: / / www.net .cm u.edu/ net reg/ ) Alcat el- Lucent Vit alQI P ( ht t p: / / www.alcat el- lucent .com / vit alqip/ ) I Pplan ( ht t p: / / ipt rack.sourceforge.net / ) NeuSt ar Met aI nfo ( ht t p: / / www.m et ainfo.com / ) Net work t axonom y can assist your underst anding of your net work as a foundat ion for pract ical securit y m onit oring. Net work t elem et ry builds upon such knowledge, and is one of t he m ost powerful t echnologies available for securit y m onit oring. I t illust rat es net work act ivit y at op your net work t axonom y foundat ion.

3 .2 . N e t w or k Te le m e t r y Telem et ry conj ures im ages of sat ellit es and aeronaut ics. I t is a t echnology t hat allows t he rem ot e m easurem ent and report ing of inform at ion of int erest t o t he syst em designer or operat or. I t 's derived from a word wit h Greek root s: " t ele" m eans rem ot e, and " m et ron" m eans " m easure." When we apply t elem et ry t o t he net working world, we're referring t o m et adat a pert aining t o I P com m unicat ions bet ween num erous syst em s. Several net work equipm ent vendors support t he abilit y t o collect and export t his net work t raffic m et adat a for analysis. The net work t elem et ry t ool we have used m ost ext ensively is Cisco's Net Flow.

3 .2 .1 . N e t Flow Net Flow m easures I P net work t raffic at t ribut es bet ween any t wo or m ore I P addresses based on OSI Layer 3 and Layer 4 inform at ion. Cisco init ially creat ed Net Flow t o m easure net work t raffic charact erist ics such as bandwidt h, applicat ion perform ance, and ut ilizat ion. Hist orically, it was used for billing and account ing, net work capacit y planning, and availabilit y m onit oring. As m ent ioned in Chapt er 2 , Net Flow records are like what you see on a phone bill ( see Figure 33) , whereas packet capt ure ( a.k.a. net work prot ocol analyzers, sniffers, and deep packet inspect ion) is like what a wiret ap collect s. Much like a phone bill, Net Flow t ells you who called, when t hey called, and for how long t he conversat ion last ed ( see Figure 3- 4) . Though not it s prim ary use, securit y is a m ore recent applicat ion of t his key net work t elem et ry t echnology. Net Flow can provide non- repudiat ion, anom aly det ect ion, and invest igat ive capabilit ies unlike any ot her net work t elem et ry t echnology. Here's Cisco's descript ion of Net Flow: [ 1 6 ] [ 16]

ht t p: / / www.cisco.com / en/ US/ docs/ ios/ solut ions_docs/ net flow/ nfwhit e.ht m l

A flow is ident ified as a unidirect ional st ream of packet s bet ween a given source and dest inat ion—bot h defined by a net work- layer I P address and t ransport - layer source and dest inat ion port num bers. There are several versions of Net Flow wit h version 5 being t he m ost widely used. Net Flow version 9 adds TCP flags and som e addit ional fields, as well as a t em plat e- based form at called Flexible Net Flow. For t he discussions and exam ples in t his book, we will use Net Flow version 5. However, t he concept s and m et hods we describe should work for any Net Flow version. Figu r e 3 - 3 . A ph on e bill

Figu r e 3 - 4 . An e x a m ple of N e t Flow ( ht t p:/ / w w w .cisco.com / e n/ US/ docs/ ne t _ m gm t / ne t flow _ colle ct ion_ e ngine / 3 .0 / use r / guide / nfcfor m .ht m l)

Net Flow can be enabled as a feat ure on any net work device t hat will st ore OSI Layer 3 and Layer 4 I P packet at t ribut es in a t able or cache as a record. The am ount of t im e any given record will st ay on a net work device depends prim arily on t he am ount of m em ory assigned for t he cache and t he num ber of new net work connect ions being t racked ( how busy t he rout er is) . You can view t hese records, or flows, direct ly from t he rout er com m and- line int erface or you can export t hem t o ext ernal st orage for viewing lat er. Figure 3- 5 shows a basic overview of how t hese records are placed in a t able and viewed or st ored. Figu r e 3 - 5 . N e t Flow ope r a t ion

Table 3- 1 shows t he Net Flow version 5 header form at .

Ta ble 3 - 1 . N e t Flow ve r sion 5 h e a de r for m a t Byt e s Con t e n t s Descript ion 0–1

Version

Net Flow export form at version num ber

2–3

Count

Num ber of flows export ed in t his packet ( 1–30)

4–7

SysUpt im e

Current t im e in m illiseconds since t he export device boot ed

8–11

unix_secs

Current num ber of seconds since 0000 UTC 1970

12–15 unix_nsecs

Residual nanoseconds since 0000 UTC 1970

16–19 flow_sequence Sequence count er of t he t ot al flows seen 20

Engine_t ype

The t ype of flow- swit ching engine

21

Engine_id

The slot num ber of t he flow- swit ching engine

22–23 Reserved

Unused ( zero) byt es

Table 3- 2 shows t he Net Flow version 5 record form at . Ta ble 3 - 2 . N e t Flow ve r sion 5 r e cor d for m a t Byt e s Con t e n t s Descript ion 0–3

srcaddr

Source I P address

4–7

dst addr

Dest inat ion I P address

8–11

next hop

I P address of t he next hop rout er

12–13 I nput

SNMP index of t he input int erface

14–15 Out put

SNMP index of t he out put int erface

16–19 dPkt s

Packet s in t he flow

20–23 dOct et s

Tot al num ber of Layer 3 byt es in t he flow packet s of t he flow

24–27 First

SysUpt im e at t he st art of t he flow

28–31 Last

SysUpt im e at t he t im e t he last packet of t he flow was received

32–33 Srcport

TCP/ UDP source port num ber

34–35 Dst port

TCP/ UDP dest inat ion port num ber

36

Pad1

Unused

37

Tcp_flags Cum ulat ive binary OR of TCP flags

38

Prot

I P prot ocol t ype

39

Tos

I P t ype of service

40–41 Src_as

Aut onom ous syst em num ber of t he source

42–43 Dst _as

Aut onom ous syst em num ber of t he dest inat ion

44

Src_m ask Source address prefix m ask bit s

45

Dst _m ask Dest inat ion address prefix m ask bit s

46–47 Pad2

Unused

The m ost im port ant fields in Table 3- 2 are t he source and dest inat ion I P addresses, source and dest inat ion port , I P prot ocol, packet s, oct et s, and first and last dat a fields. The cont ent of t he byt es in t hese fields of t he flow record gives you t he " sessionized" dat a necessary for effect ive securit y m onit oring. Alt hough underst anding t he specific byt e order of a Net Flow packet is not t erribly im port ant t o securit y m onit oring, underst anding how t o collect t he dat a is im port ant . Analysis of export ed Net Flow packet dat a will help you underst and t he cont ext of any securit y incident . 3 .2 .1 .1 . Ex por t in g N e t Flow for colle ct ion Many net work hardware vendors support Net Flow, including Cisco, Juniper ( it s im plem ent at ion is called J- Flow) , Huawei ( it s im plem ent at ion is called Net St ream ) , and ot hers. When using Net Flow, you m ust first decide where in t he net work you will collect t he dat a. At a m inim um , you m ust collect Net Flow at perim et er rout ers t o m aint ain a view int o t he ingress ( ent ering) and egress ( leaving) t raffic t raversing your net work. You can do t his at your I SP gat eway rout ers (see Figure 3- 6) or, if you have a larger and m ore com plex DMZ presence, at your DMZ dist ribut ion layer rout ers ( see Figure 3- 7) . Figu r e 3 - 6 . N e t Flow colle ct ion a t I SP ga t e w a y r ou t e r s

Figu r e 3 - 7 . N e t Flow colle ct ion a t D M Z ba ck bon e r ou t e r s

I ngress and egress t raffic collect ion of Net Flow is t he key t o using net work t elem et ry successfully! You'll need t his dat a t o see what t raffic is t raversing your net work perim et er, and t o underst and what direct ion it is flowing. Net Flow collect ion requires t wo com ponent s: t he Net Flow sender and t he Net Flow collect or. The Net Flow sender can be any net working device t hat creat es and export s Net Flow. Many com m ercial and open source vendors offer collect ion soft ware. The following exam ples discuss Net Flow export from Cisco rout ers and collect ion wit h t he popular flow- t ools package, which is an open source effort from Ohio St at e Universit y. Configuring Cisco rout ers t o export Net Flow is a sim ple t ask, but t he st eps will vary slight ly by hardware plat form and Net Flow version. You should always collect bidirect ional flow dat a; ot herwise, you m ay end up wit h only half t he conversat ions. Exam ple 3- 4 shows a sim ple Net Flow configurat ion. I t illust rat es how t o enable Net Flow at t he global level, defining a source int erface for export , and defining a dest inat ion collect or I P address and UDP port .

Ex a m ple 3 - 4 . Sim ple N e t Flow con figu r a t ion for a Cisco r ou t e r

Router1(config)#ip route-cache flow Router1(config)#ip flow-export source Loopback 0 Router1(config)#ip flow-export destination 10.1.1.1 9999 Router1(config)#interface FastEthernet0/0 Router1(config-if)#ip route-cache flow

Not ice t hat t he source int erface used in Exam ple 3- 4 is a Loopback int erface. This is preferable, as t he I P address assigned t o t he Loopback int erface is less likely t o change as net works are readdressed or m odified. This I P address will probably be used in firewall rule set s t o allow export of DMZ Net Flow int o a collect or on your int ernal net work, as depict ed previously in Figures Figure 3- 3 and Figure 3- 4. The dest inat ion address of 10.1.1.1 is t he address of t he Net Flow collect or, which we expect t o be list ening on UDP port 9999. 3 .2 .1 .2 . Pe r for m a n ce con side r a t ion s for N e t Flow colle ct ion You m ay be concerned about t he im pact of Net Flow export from your net work devices. Cisco has docum ent ed t he perform ance im pact of Net Flow on m any of it s plat form s, and you can reference t his inform at ion in it s " Net Flow Perform ance Analysis" whit e paper, found at t he following web page: ht t p: / / www.cisco.com / en/ US/ t ech/ t k812/ t echnologies_whit e_paper0900aecd802a0eb9.sht m l Based on Cisco's t est ing, t he variable t hat carries t he m ost weight on all plat form s is t he num ber of flows. The m ore flows t raversing t he rout er, t he m ore CPU resources are used. Ot her variables include I OS version and whet her t he Net Flow processes are handled in hardware or soft ware. I n our experience, Net Flow has a negligible im pact on rout er CPU resources. As always, however, you will want t o invest igat e, t est , and verify t he im pact of any change t o your product ion rout er configurat ion. I f you find t hat t he im pact on CPU resources is t oo great for your environm ent , all is not lost : Net Flow has a feat ure, called Sam pled Net Flow, which m any Cisco rout ers support . Sam pled Net Flow allows a rout er t o sam ple one out of every X packet s. For t he purposes of using Net Flow t o invest igat e securit y incident s, however, it is not recom m ended t hat you im plem ent Sam pled Net Flow; you will likely m iss vit al forensic dat a. 3 .2 .1 .3 . W h e r e t o colle ct N e t Flow For t he securit y analysis we're describing, you should collect Net Flow from devices t hat represent a choke point or aggregat ion point in your net work. Such a choke point delineat es connect ion point s—t ypically a " dist ribut ion layer" device in net work parlance. Exam ples include t he following:

A pair of redundant dat a cent er gat eways All t raffic going int o and out of your dat a cent er

A DMZ backbone rout er All t raffic going int o and out of your DMZ

An ext ranet backbone rout er All t raffic going int o and out of your ext ranet Figu r e 3 - 8 . Ba sic flow - t ools com pon e n t s

Ensure t hat you have Net Flow enabled on all appropriat e rout er int erfaces, and verify t hat you have t he full flow m ask, as illust rat ed in previous sect ions. 3 .2 .1 .4 . OSU flow - t ools OSU flow- t ools cont ains t ools t o collect , send, process, and generat e report s from Net Flow. An ent ire book could be dedicat ed t o t his one t ool set ; inst ead, we will provide sim ple exam ples of t heir pract ical applicat ion for securit y m onit oring. Appendix A cont ains det ailed inst ruct ions for com plet ing a basic Net Flow collect ion set up wit h OSU flowt ools. To effect ively deploy and use OSU flow- t ools, you m ust underst and four basic flow- t ools com ponent s: flow- capt ure, flow- cat , flow- filt er, and flow- print , as shown in Figure 3- 8. Here's a brief descript ion of what each com ponent handles:

Flow- capt ure List ens on t he defined UDP port , processes Net Flow packet s, and st ores t he dat a t o disk in a highly com pressed binary form at

Flow- cat Ut ilit y for concat enat ing t he capt ured dat a; used when searching t he st ored dat a

Flow- filt er Powerful searching and report ing ut ilit y allowing select ion of specific flows based on crit eria such as source and dest inat ion I P prot ocols, address, and port

Flow- print Ut ilit y t o display flow dat a form at t ed in ASCI I The abilit y t o det erm ine whet her a conversat ion bet ween syst em s occurred is vit al for securit y invest igat ions. Each t im e a syst em sends an I P packet t hrough a rout er wit h Net Flow export enabled, you have a digit al fingerprint of t hat t raffic. The following are som e real- world exam ples using t hese digit al fingerprint s t o confirm such conversat ions wit h high

confidence. These fingerprint s are vit al confirm at ions of act ivit y—from a single packet at t ack t o t he leak of crit ical inform at ion. 3 .2 .1 .4 .1 . I de n t ifyin g in fe ct e d h ost s pa r t icipa t in g in bot n e t s Suppose t hat a new virus variant has been released, and your exist ing ant ivirus product is unable t o ident ify it . A Windows adm in in your organizat ion has sent t he following em ail t o your t eam :

Security Team, My server, WADC-WIN2K, has been having serious performance problems in the past 12 hours. The CPU keeps spiking and when I do a 'netstat' command I see lots of connections to hundreds of other hosts on TCP 135. This is not normal for this server. I ran Kaspiramicro Antivirus and didn't see any viruses listed. I know that it was recommended for me to upgrade all of my servers because of the recent vulnerabilities published, but I forgot this one. Perhaps it is related? Could you take a look? Thanks, Srinivisan Nguyen Navratalova Senior Systems Administrator Blanco Wireless, Inc.

Your Windows securit y lead invest igat or checks t he syst em and finds som e odd- looking .exe files t hat were writ t en t o C: \ WI NDOWS\ SYSTEM32 t he evening before. Som e regist ry ent ries appear t o run t he execut able on syst em st art up. He also spot s a connect ion on TCP 31337 t o a server in a known hacker net work, according t o several securit y list s, at 31.33.73.57. Upon subm it t ing t he binary t o t he ant ivirus soft ware vendor, analysis confirm s t hat it is indeed a new variant of a virus, one t hat uses t he server's lat est ( unpat ched) vulnerabilit y as a propagat ion vect or. This virus is found t o be a variant of AngryMint St orm / 32, known for it s popularit y am ong t he organized cybercrim e group t hat runs t he world's largest I nt ernet Relay Chat ( I RC) bot net s. Your Windows securit y lead invest igat or sends t he inform at ion t o you in an em ail:

Hey guys, Looks like that server you had me look at is infected with a new variant of the AngryMintStorm virus we've been seeing lately. It has an IRC botnet component that allows these miscreants to control the system remotely. You should see connections going to omg.i.pwnt.your.server.secretdns.org, which currently resolves to IP address 31.33.73.57. We've got a new virus definition file on the way from our Kaspriamicro, but there's no telling how many systems were hit with this. Good Luck, Jonathon "Jack" Bowers Senior Windows Security Engineer

To ident ify which syst em s on your net work were infect ed wit h t he virus, you can use Net Flow. A digit al fingerprint will exist for any I P address t hat at t em pt ed t o com m unicat e wit h t he virus's I RC com m and and cont rol server ( C&C) I P address. I n such a sit uat ion, it 's com m on t o t reat any syst em having com m unicat ed wit h 31.33.73.57 on TCP port 31337 as com prom ised. You should rem ediat e t he sit uat ion im m ediat ely; t here is no legit im at e reason for a com put er on your net work t o com m unicat e wit h ht t p: / / om g.i.pwnt .your.server.secret dns.org. You can use t he collect ed Net Flow

and OSU flow- t ools t o ident ify any syst em t hat has com m unicat ed wit h t he m alicious server via your net work. First , creat e an access cont rol list ( ACL) file nam ed flow.acl wit h t he I P address of t he C&C. This synt ax is very sim ilar t o t he ACLs used on net work firewalls. This file will have access cont rol ent ries t hat are used t o m at ch flow t raffic:

ip access-list standard botnet permit host 31.33.73.57 The preceding line shows an ACL nam ed botnet t hat will be used t o m at ch I P address 31.33.73.57. Now, use t he flowcat, flow-filter, and flow-print com m ands t o search t he Net Flow collect ed from t he perim et er DMZ backbone rout ers. You're looking for digit al fingerprint s of connect ions bet ween t he I RC C&C server and syst em s on your net work:

[nfchost]$ flow-cat /var/local/netflow/files/2008-1-12/ft* | flow-filter -Sbotnet -o -Dbotnet | flow-print -f5

The flow-cat /var/local/netflow/files/2008-1-12/ft* com m and concat enat es t he com pressed binary file( s) cont aining t he Net Flow, which was specified in t he file|directory option. The flow-filter –Sbotnet –o –Dbotnet com m and filt ers t he flows based on source or dest inat ion m at ches, which were specified as bot net C&C servers in t he flow.acl file creat ed earlier. The flow-print –f5 com m and out put s t he m at ching flow records in an easy- t o- read, one- line, 32- colum n form at . I n plain English, t his com m and says " Show m e all Net Flow sourced from or dest ined t o t he bot net C&C I P address," or m ore sim ply, " Who is t alking t o t he bad guys?" Here's t he out put of t he preceding com m and:

Start End 0112.08:39:49.91 0112.08:40:34.51 0112.08:40:33.59 0112.08:40:42.29

SrcIPaddress 10.10.71.100 31.33.73.57

SrcP 8343 31337

DstIPaddress 31.33.73.57 10.10.71.100

DstP 31337 8343

You now have a digit al fingerprint of I P address 10.10.71.100 t alking t o 31.33.73.57 at 08: 39 on January 12. You need t o rem ediat e 10.10.71.100 im m ediat ely. This single query can quickly ident ify all syst em s on your net work infect ed wit h the AngryMint St orm / 32 zero- day virus. N OTE

Bot net int elligence can be gained via m alware analysis wit h t ools such as Norm an SandBox,[ 1 7 ] conversat ional heurist ic analysis soft ware such as Bot Hunt er, [ 1 8 ] and dat a shared wit hin t he securit y com m unit y via t he Snort bleeding- edge NI DS signat ure dat abase. [ 1 9 ] [ 17]

ht t p: / / www.norm an.com / m icrosit es/ nsic/

[ 18]

ht t p: / / www.bot hunt er.net /

[ 19]

ht t p: / / www.em ergingt hreat s.net /

You can ext end t his m et hod t o ot her form s of m alware or phishing. For inst ance, t he following exam ple leverages Net Flow t o ident ify fraud vict im s. I m agine t he im pact on your em ployees if t hey received t he phishing em ail shown in Figure 3- 9, causing t he inadvert ent disclosure of t heir bank account credent ials or ot her sensit ive inform at ion. One of your savvier em ployees sends you t he following em ail:

Security Team,

I and many other employees received the attached email today. It looks like phishing to me. Since so many of our employees use BigBank for their banking I wanted to bring this to your attention. Regards, Frank N. Beyahns Sales Director Blanco Wireless

A quick look at t he em ail shows t hat t he URL present ed t o t he user (ht t p: / / 31.3.33.7/ .PayPal/ cm d/ cgibin/ webscrcm d_login.php) does not belong t o BigBank, but rat her t o a server on your fam iliar hacker net work. The hackers have set up a websit e t hat looks j ust like BigBank's websit e so t hat t hey can harvest legit im at e credent ials and st eal m oney from cust om ers who are t ricked by t he phishing em ail. You now know t hat anyone clicking on t he link will leave a digit al fingerprint ; Net Flow will record t raffic dest ined t o TCP port 80 on I P address 31.3.33.7. I n addit ion, BigBank let your securit y t eam know t hat t wo m ore sit es were involved in t his phishing act ivit y at I P addresses 64.38.64.20 and 139.192.10.10. You can build a filt er t o m at ch against t he m alicious host I P addresses:

ip access-list standard phish permit host 31.3.33.7 ip access-list standard phish permit host 64.38.64.20 ip access-list standard phish permit host 139.192.10.10 The flow-filter com m and has m any opt ions. I n t his case, you're going t o filt er based on your nam ed ACL ( phish ) and dest inat ion port ( 80) :

[nfchost]$ flow-cat /var/local/netflow/files/2007-11-28/ft* | flow-filter -P80 | flow-print -f5 Start End 1128.09:39:49.91 1128.09:40:34.51 1128.09:45:33.59 1128.09:46:42.29 1128.14:15:13.05 1128.14:16:21.15

SrcIPaddress 10.10.51.75 10.10.55.12 10.10.30.119

SrcP 1250 2080 1349

-Dphish

DstIPaddress 31.3.33.7 139.192.10.10 139.192.10.10

Figu r e 3 - 9 . Ph ish in g e m a il r e ce ive d by e m ploye e s

DstP 80 80 80

You need t o check your DHCP, Windows Act ive Direct ory, or ot her logs t o see who had I P addresses 10.10.51.75, 10.10.55.12, and 10.10.30.119 at t he t im es list ed. Arm ed wit h t hat inform at ion, you can not ify t hose users of t he phishing act ivit y and t ake appropriat e act ion wit h BigBank.

I f t he phishing page in our exam ple was host ed on a sit e t hat also includes popular legit im at e cont ent ( such as Facebook, MySpace, iBiblio, et c.) , ident ificat ion of com prom ised syst em s becom es m ore com plicat ed. You need t o augm ent Net Flow wit h proxy server logs or ot her event sources t hat show t he URLs accessed by affect ed users. 3 .2 .1 .4 .2 . Flow a ggr e ga t ion Net Flow can also ident ify policy violat ions, our core t hem e. For exam ple, your com pany would probably wish t o prevent dat a from being copied from a dat abase server t o a host out side your net work. Since you know your net work well, and have Net Flow deployed t o your dat a cent er dist ribut ion gat eways, you can set up aut om at ed report ing t o send an alert , for exam ple, when a dat a t ransfer great er t han 500 MB occurs from your dat abase host t o an out side syst em . I n net works wit h high t raffic volum e, a file t ransfer can be recorded part ially in one flow- file and part ially in anot her. Since t he OSU flow- t ools collect or ( and m ost ot her collect ors) writ es t o a file every X m inut es as defined by t he opt ions specified in t he flow-capture com m and, you will need a way t o aggregat e flow dat a over m ult iple files, as shown in Figure 3- 10. Figu r e 3 - 1 0 . Flow a ggr e ga t ion of a file t r a n sfe r

I n Figure 3- 10, you can see a 554 MB file t ransfer from a server t o a m iscreant 's syst em occurring over a 15- m inut e

period. To det ect such a t ransfer, we'll use Net Flow t o ident ify any syst em t hat t ransfers m ore t han 500 MB of dat a in a single session. Because your Net Flow collect or writ es a new flow dat a file every five m inut es, t he dat a cont ained in any single file m ay not show t hat a policy violat ion occurred. Assum e t hat flow- file 1 shows a 200 MB file t ransfer, flow- file 2 also shows a 200 MB t ransfer, and flow- file 3 cont ains t he dat a for t he last 154 MB of t he file t ransfer. None of t hese would alone t rigger t he 500 MB t hreshold. To see t he pot ent ial 500 MB+ policy violat ion, you need t o aggregat e t he flow dat a. One solut ion t o t his flow- t ools lim it at ion leverages yet anot her open source suit e of Net Flow t ools, called nfdum p. [ 2 0 ] Using t he net work t axonom y dat a cont ained in t he I PAM t ool, we can narrow our search t o Oracle dat abase subnet s, locat ed in 10.20.20.0/ 24 and 10.20.23.0/ 24, referenced in t his flow-cat com m and: [ 20]

ht t p: / / www.nfdum p.sourceforge.net /

[nfchost]$ flow-cat /var/local/netflow/files/2008/1/5/ft* | ft2nfdump | nfdump -a A srcip,dstip 'bytes > 500000000 and ( src net 10.20.20.0/24 or src net 10.20.23.0/24) Date start Duration Proto 2007-1-05 12:39:11 1151.3 TCP

SrcIP:Port DstIP:Port Packets Bytes 10.20.20.8:22 10.10.50.32:4942 234729 554.9 M

The preceding out put shows t hat on January 5, 2007, at 12: 39 p.m ., a syst em using I P address 10.10.50.32 t ransferred a 554 MB file from an Oracle server at 10.20.20.8 via SSH ( TCP 22) . Follow- up on t his t ransfer wit h t he em ployee revealed t hat a consult ant from Art ist s Consult ing had downloaded a dat abase cont aining cust om er inform at ion t o his lapt op. This is in violat ion of Blanco's accept able use policy. The consult ant 's lapt op was t aken for forensic im aging and invest igat ion. Net Flow at t he DMZ perim et er was queried t o det erm ine whet her t he file had been copied out of Blanco's net work. The following query shows all t raffic for t he consult ant 's com put er at 10.10.50.32 on January 5:

ip access-list standard artistcon permit host 10.10.50.32 [nfchost]$ flow-cat /var/local/netflow/files/2007-1-5/ft* | flow-filter -Sartistcon -o -Dartistcon | flow-print -f5

The result s of t his query will t ell you whet her t he file was copied out of Blanco's net work . 3 .2 .1 .4 .3 . Re pu dia t ion a n d n on r e pu dia t ion You can use Net Flow t o repudiat e t he occurrence of a net work conversat ion bet ween t wo or m ore syst em s. A query of Net Flow for specific I P addresses during t he relevant t im e fram e can show zero result s and prove t hat a conversat ion didn't exist . Conversely, you can use Net Flow t o prove t he occurrence of a net work conversat ion bet ween t wo or m ore syst em s. This is very useful in sit uat ions where you suspect t hat a part icular event has occurred and you want t o prove t hat it did. A query of t he Net Flow for specific I P addresses during t he relevant t im e fram e can show result s, proving t hat a conversat ion did happen. Not e t hat we're using repudiat ion and nonrepudiat ion in t he cont ext of t ransact ions happening on your net work, not in t he cont ext of a court of law. 3 .2 .1 .5 . Ch oosin g a N e t Flow colle ct or As we already m ent ioned, m any solut ions—including com m ercial and open source—are available for Net Flow analysis. Here are som e com m on feat ures t o consider when select ing a solut ion for Net Flow collect ion:

Dat a com pression

Replicat ion capabilit y St orage and processing perform ance The Net Flow collect ion soft ware should support com pression of t he Net Flow. The higher t he com pression, t he longer t he dat a is ret ained. OSU flow- t ools im plem ent s zlib com pression which, wit h default set t ings, has shown a com pression rat io of around 4: 1. Alt hough t here are no known form ulas for calculat ing disk space requirem ent s, experience wit h OSU flow- t ools' flow-capture com ponent in a product ion DMZ environm ent is im pressive. Wit h 1.2 gigabit s per second of bandwidt h and a 200,000 packet - per- second ( pps) dat a rat e, a 600 GB disk can m aint ain approxim at ely t welve weeks of com pressed Net Flow. The abilit y t o relay flow dat a t o ot her syst em s in real t im e is im port ant , as m any t ools can use Net Flow. Exam ples include SI M t ools such as Cisco Securit y Monit oring, Analysis, and Response Syst em ( MARS) , anom aly det ect ion solut ions such as Arbor Peakflow, and opt im izat ion t ools such as Net QoS Perform ance Cent er and Lancope St ealt hWat ch. To provide m axim um flexibilit y for t he Net Flow analysis solut ion, t he collect or should st ore dat a in st andard, nonpropriet ary form at s, or provide ut ilit ies t o convert dat a t o st andard form at s. This will enable you t o use a variet y of t ools t o analyze your st ored Net Flow. Here are som e com m on considerat ions when select ing a Net Flow analysis solut ion:

Speed of t he query St orage of t he raw dat a Access t o t he raw dat a Several fact ors det erm ine t he speed of a query, including t he capabilit ies of t he hardware on which t he analysis soft ware is running and proper query bounding ( e.g., it is probably not a good idea t o query Net Flow records for all HTTP t raffic from all host s in a 24- hour period against a sizable e- com m erce environm ent ; you would quickly run out of m em ory and disk space) . I n addit ion, t he abilit y t o access raw flow dat a can allow you t o develop cust om t ools t o serve t he unique requirem ent s of your environm ent . N OTE St oring raw dat a in a highly com pressed file form at is preferable t o st oring it in a dat abase. A busy net work will quickly cause t he dat abase t o grow by m ult iple t erabyt es wit hin days. OSU flow- t ools com pression allows us t o st ore approxim at ely t hirt y days of flows for an environm ent wit h 200,000 pps sust ained on 600 GB of disk space. I n sum m ary, Net Flow is an inexpensive, easy- t o- deploy, widely support ed, and powerful t ool t hat has m any uses for securit y m onit oring, incident invest igat ion, and net work forensics. The digit al fingerprint s provided by Net Flow are inv aluable.

3 .2 .2 . SN M P Sim ple Net work Managem ent Prot ocol ( SNMP) is a useful com ponent in a well- rounded net work t elem et ry suit e. SNMP analysis can provide addit ional visibilit y int o t raffic pat t erns, net work pat h ut ilizat ion, and device healt h ( as we'll discuss in Chapt er 7 ) . We'll t ake a narrow view of SNMP, lim it ing it s use for net work t elem et ry and securit y m onit oring. Net work int erface ut ilizat ion is t he single m ost useful SNMP at t ribut e for net work t elem et ry; when graphed over t im e, it provides an easy- t o- underst and , visual view of your net work environm ent . 3 .2 .2 .1 . M RTG There are m any com m ercial and open source soft ware t ools for collect ing and analyzing SNMP dat a. I n t his sect ion, we will explain how t o use t he open source t ool MRTG. The Mult i Rout er Traffic Grapher ( MRTG) is free soft ware creat ed by Tobias Oet iker for m onit oring net work t raffic on dat a net work links. Here is Oet iker's descript ion of MRTG: [ 2 1 ] [ 21]

ht t p: / / oss.oet iker.ch/ m rt g/ index.en.ht m l

You have a rout er, you want t o know what it does all day long? Then MRTG is for you. I t will m onit or SNMP net work devices and draw pret t y pict ures showing how m uch t raffic has passed t hrough each int erface. Rout ers are only t he beginning. MRTG is being used t o graph all sort s of net work devices as well as everyt hing else from weat her dat a t o vending m achines. MRTG is writ t en in perl and works on Unix/ Linux as well as Windows and even Net ware syst em s. MRTG is free soft ware licensed under t he Gnu GPL.

3 .2 .2 .1 .1 . M RTG e x a m ple I m agine you are t asked wit h deploying a NI DS int o a dat a cent er environm ent . How will you design t he solut ion? One of t he m ost basic requirem ent s when deploying a NI DS involves sizing it t o properly analyze t he aggregat e net work bandwidt h. SNMP provides an abilit y t o t rack link ut ilizat ion t o discover aggregat e bandwidt h. I n addit ion, net work ut ilizat ion pat t erns can provide visibilit y int o st at es or occurrences of net work t raffic t hat fall out side norm al ranges for t he environm ent . A quick glance at net work ut ilizat ion graphed over t im e can ident ify anom alies in act ivit y. For exam ple, analysis of t he graph in Figure 3- 11 shows t hat t he NI DS det ect ed a sizable spike in net work t raffic Sat urday night . I s t his norm al? Should anyone be on t his part icular net work lat e at night on a Sat urday? This sit uat ion m ay warrant furt her invest igat ion, possibly using Net Flow t o ident ify large file t ransfers. Figu r e 3 - 1 1 . SN M P qu e r ie s for n e t w or k in t e r fa ce da t a gr a ph e d ove r t im e w it h M RTG

The dat a t hat result s when you graph net work t raffic in t his m anner can be considered m et a- m et adat a; t hat is, in cont rast wit h Net Flow we see only aggregat e ut ilizat ion st at ist ics, not individual conversat ions. Net Flow does not underst and t he concept of int erface bandwidt h or speed; it underst ands only byt e ( or oct et ) count . SNMP dat a can give you a high- level view of t raffic pat t erns t hat m ay point t o t he need for furt her invest igat ion wit h Net Flow, which adds furt her det ail.

3 .2 .3 . Rou t in g a n d N e t w or k Topologie s You m ust keep net work and rout ing t opologies and policies in m ind when int erpret ing securit y alert s; t hey allow you t o infer expect ed t raffic pat h t raversal. To accom plish t his, answer t hese quest ions regarding your rout ing t opologies:

Which address blocks are publicly rout ed? Which address blocks are act ive/ deployed? From which I SP connect ion are t he address blocks advert ised? I f you see securit y alert s sourced from or dest ined t o an address block t hat has not been deployed in your net work, you can infer several t hings. I f t he alert shows an at t ack sourced from an inact ive or yet - t o- be- deployed address block, it is likely spoofed. I f t he alert is dest ined t o an inact ive or yet - t o- be- deployed address block, it is likely a net work scan or a noisy worm . Underst anding rout ing t opologies can also help you set expect at ions for t raffic flow and pat h t raversal. For exam ple, I know t he default rout e advert ised t o m y Ohio sales office com es from our New York branch's I SP connect ion. I can infer t hat any out bound NI DS alert s or Net Flow showing a source I P address from m y Ohio office should t raverse t he New York DMZ.

3 .3 . Th e Bla n co W ir e le ss N e t w or k Blanco's securit y t eam has worked wit h it s I T st aff t o bet t er underst and t heir com pany's net work using I PAM dat a, Net Flow, and general rout ing inform at ion.

3 .3 .1 . I P Addr e ss Assign m e n t Blanco has a sim ple address space, docum ent ed wit h t he open source I Pplan soft ware. The subnet s we will use in our exam ples for t his and subsequent chapt ers are shown in Figure 3- 12 and appear highlight ed in t he following code snippet :

10.10.0.0/16 Redwood City Campus |-- 10.10.0.0/19 Data Centers |-- 10.10.32.0/19 Site 1 Desktop Networks | |-- 10.10.32.0/24 Building 1 1st floor | |-- 10.10.33.0/25 Building 1 2nd floor | |-- 10.10.33.128/25 Building 2 10.10.0.0/19 Data Centers |-- 10.10.0.0/20 Building 3 Data Center | |-- 10.10.0.0/25 Windows Server Subnet | |-- 10.10.0.128/25 Oracle 10g Subnet | |-- 10.10.1.0/26 ESX VMWare Farm | |-- 10.10.1.64./26 Web Application Servers Figu r e 3 - 1 2 . Bla n co W ir e le ss su bn e t da t a in I Ppla n soft w a r e

3 .3 .2 . N e t Flow Colle ct ion I n keeping wit h best pract ices, Blanco collect s Net Flow from it s Cisco rout ers in bot h t he DMZ backbone and t he dat a cent er gat eways. Blanco uses t he OSU flow- t ools package t o collect and analyze Net Flow for m onit oring and incident response.

3 .3 .3 . Rou t in g I n for m a t ion Blanco has a class C net work allocat ed, which is used for cust om er- facing web services. This net work exist s in Blanco's single DMZ net work wit h t wo I nt ernet connect ions provisioned from t wo com pet ing I SPs. I n addit ion, all net work t raffic dest ined for out side Blanco's aut onom ous syst em ( AS) t raverses it s one DMZ net work. I nt ernally, Blanco has deployed RFC 1918 address space in t he 10.10.0.0/ 16 block; no ot her port ion of 10.0.0.0/ 8 has been

deployed ( see Figure 3- 13 ) . Figu r e 3 - 1 3 . Bla n co W ir e le ss n e t w or k dia gr a m

3 .4 . Con clu sion St ruct ured, docum ent ed net work knowledge is foundat ional t o cont ext - based securit y m onit oring. By deploying t ools for docum ent ing and underst anding your net work environm ent , you can begin t o priorit ize securit y alert s based on how t hey affect your net work. Chapt er 4 will provide a t hird and final foundat ion, guiding you t o select broad t arget s t o priorit ize your m onit oring against .

Ch a pt e r 4 . Se le ct Ta r ge t s for M on it or in g Reese: Anyone who's t rying t o use t he override would have t o hack t hrough t he react or's firewall. That would t ake m assive com put er power, which NRC could see com ing a m ile away. Jack: I s it possible t hat NRC could m iss t his due t o unusually heavy t raffic on t he I nt ernet ? Reese: I don't follow. Jack: Well, less t han an hour ago m illions of com put ers around t he world were st ream ing t he t rial of t he Secret ary of Defense. I s it possible t hat t his t rial was j ust som e kind of a Troj an horse t o disguise a m assive at t ack on nuclear power plant s' firewalls? Reese: That m ight be possible. [ 2 2 ] [ 22]

24, " Episode 4 x 06: 12: 00 P.M.–1: 00 P.M." Original air dat e: January 24, 2005. Tradem ark & © 2005 Twent iet h Cent ury Fox Film Corporat ion. Originally t ranscribed for TwizTV.

Successful ent erprise securit y m onit oring dem ands focus. I n t his fict it ious at t ack from Fox's popular t elevision series 24, t he net work securit y t eam ( NRC) m issed a t arget ed, crit ical at t ack due t o a decoy surge in net work t raffic. The depict ed scenario dem onst rat es how difficult it is t o m onit or effect ively when all syst em s are given " equal" at t ent ion. Focused m onit oring requires considerable t uning and quick recognit ion of benign t raffic. For exam ple, an Oracle 11i- based ERP syst em will generat e t housands of m essages per hour from it s various com ponent s. Wit hout t uning and a st rong underst anding of t he environm ent , t he net work m onit oring st aff will wast e t heir t im e following dead ends. Exam ple 4- 1 cont ains an exam ple of a Snort alert t hat could result from m onit oring an Oracle syst em . Ex a m ple 4 - 1 . Sn or t a le r t fr om m on it or in g a n Or a cle syst e m [ 2 3 ]

[**] [1:2650:2] ORACLE user name buffer overflow attempt [**] [Classification: Attempted User Privilege Gain] [Priority: 1] [Xref => http://www.appsecinc.com/Policy/PolicyCheck62.html] Event ID: 62662 Event Reference: 62662 11/21/06-17:16:04.892127 172.16.0.136:34833 -> 172.16.5.12:1521 TCP TTL:64 TOS:0x0 ID:32755 IpLen:20 DgmLen:100 DF ***AP*** Seq: 0xB700F908 Ack: 0xF1594793 Win: 0x3309 TcpLen: 32 TCP Options (3) => NOP NOP TS: 2796076485 835448929 00 30 00 00 06 00 00 00 00 00 03 68 FB 01 00 00 .0.........h.... 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 01 ................

[ 23]

See ht t p: / / www.webservert alk.com / archive251- 2006- 11- 1737633.ht m l.

I s t his a crit ical alert ? The words buffer overflow at t em pt are sure t o get your at t ent ion, yet t his alert is benign—oft en t riggered by Java applicat ions using a st range dat abase connect ion st ring. [ 2 4 ] By focusing your t uning and your m onit oring, you can ignore and t une out t hese benign event s. Across t he ent erprise, various logging and m onit oring com ponent s generat e m illions of addit ional exam ples of such alert s. For effect ive m onit oring, each alert will require det ailed analysis, oft en involving support and applicat ion st aff m em bers t o explain t he anom alous t raffic. This kind of t uning is an im port ant part of securit y m onit oring, and it 's a cont inuous t ask. Policy- based m onit oring can't spare you t he t edium of such t uning; rat her, it will narrow your focus t o crit ical syst em s and t o event s t hat indicat e policy violat ions. To apply policy- based m onit oring, you m ust first select which syst em s you want t o m onit or. [ 2 4 ] OSDir Mail Archive, m sg# 00171: " False Posit ive in 2650.2 ( ORACLE user nam e buffer overflow at t em pt ) ."

This chapt er will describe t he various approaches for select ing m onit oring t arget s. Once select ed, we'll drill deeper int o t he t arget s, providing guidance for how t o discover t he com ponent s wit hin t hem , and docum ent ing t he det ails necessary t o configure event feeds.

4 .1 . M e t h ods for Se le ct in g Ta r ge t s To effect ively select syst em s t o m onit or ( i.e., your t arget s) , you m ust est ablish your priorit ies. Your approach will be based on t he risk or perceived value of t he t arget s and t he inform at ion wit hin t hem . You can develop a t axonom y of approaches in count less ways. Here's t he range we'll consider in t his chapt er:

Business im pact analysis Different iat ing syst em s based on level of crit icalit y in t he cont ext of availabilit y

Revenue im pact Applicat ions t hat drive orders and delivery

Expenses im pact Syst em s t hat m anage cont ract ual obligat ions t o part ners and cust om ers

Legal requirem ent s St at ut es and cont ract s t hat highlight which inform at ion and syst em s t o prot ect

Sensit ivit y profile Syst em s t hat access privileged, rest rict ed inform at ion

Risk profile Syst em s t hat can't be properly prot ect ed using exist ing cont rols

Visibilit y profile Syst em s which, if at t acked or com prom ised, m ight prove em barrassing t o t he com pany or dam aging t o visit ors

4 .1 .1 . Bu sin e ss I m pa ct An a lysis The process of conduct ing a business im pact analysis ( BI A) is part of t he disast er recovery planning funct ion in m ost ent erprises. I t 's a m et hodology for ident ifying crit ical processes and syst em s t o est ablish t heir priorit y for backup and disast er recovery. Many books describe m et hodologies for t his exercise, including guides for t he Cert ified I nform at ion Syst em s Securit y Professional ( CI SSP) exam . Here's a sam ple process t aken from Robert a Bragg's CI SSP Training Guide : [ 2 5 ] [ 25]

1. 2. 3. 4. 5.

Robert a Bragg. CI SSP Training Guide ( Que Publishing, 2002) , pp. 454–458.

I dent ify t im e- crit ical business processes. I dent ify support ing resources ( personnel, facilit ies, t echnology, com put ers, soft ware, net works, equipm ent , vit al records, dat a, et c.) for t he crit ical processes. Det erm ine Maxim um Tolerable Downt im es ( MTDs) . Ret urn t o business unit s for validat ion. Provide t he final report , including MTDs and recom m endat ions for next st eps, t o senior m anagem ent .

Bragg recom m ends a series of int erviews wit h business st akeholders t o det erm ine t he effect of downt im e or disrupt ion on business processes. BI A breaks down t he im pact over t im e, chart ing t he result s int o a t able for m ore det ailed analysis and discussion, t o guide priorit izat ion. The key input t o t hese priorit izat ion discussions is t he im pact t o t he business bot t om line, which invit es considerat ion of sales, revenue, financial int erest , fines/ penalt ies, and even cost s t o your cust om ers and part ners.

This inform at ion is t hen verified in st eps 4 and 5. Hopefully, your com pany's disast er planning t eam has already perform ed a BI A. I n any case, t he generat ed list provides an excellent st art ing point for t arget ing your securit y m onit oring.

4 .1 .2 . Re ve n u e I m pa ct An a lysis The BI A focuses heavily on t he revenue im pact of disrupt ions, so let 's analyze revenue im pact a bit furt her. Any applicat ion used in t aking or fulfilling orders, for exam ple, is t ied direct ly t o revenue ( for com panies t hat sell goods, not services) . For an e- com m erce com pany, Bragg suggest s t hat t he MTD is exact ly 0, as even t iny am ount s of downt im e can result in st aggering losses. According t o Cavusoglu, Mishra, and Raghunat han, " Firm s t hat solely depend on t he I nt ernet as a revenue generat ing m echanism pay higher prices in case of a securit y breach t han firm s t hat have m ult iple sales channels. Securit y of I T syst em s for net firm s is ext rem ely im port ant for a net firm 's success." [ 2 6 ] [ 2 6 ] Huseyin Cavusoglu, Birendra Mishra, and Srinivasan Raghunat han. " The Effect of I nt ernet Securit y Breach Announcem ent s on Market Value of Breached Firm s and I nt ernet Securit y Developers" ( The Universit y of Texas at Dallas School of Managem ent , 2002) , p. 14.

Of course, not all com panies m ake t heir m oney selling goods. Revenue for service- based com panies depends on t he com panies' abilit y t o perform t he t ask for which t hey've been hired. The aut o body shop, t he U.S. Post al Service, and eBay don't receive revenue if t hey can't fix, deliver, or auct ion your product . When considering syst em s t hat are crit ical t o revenue generat ion and recognit ion, order ent ry and fulfillm ent syst em s will have t he largest im pact on revenue for m ost com panies :

Order ent ry syst em s Whet her orders are ent ered direct ly by cust om ers or t hey're t aken by a sales professional, t he syst em s used for t aking orders are crit ical t o revenue generat ion. Typically, m any syst em s are involved in t he process, including invent ory, order st at us, order booking, credit card processing, and ret urn processing syst em s. These syst em s m ust not only be prot ect ed from unplanned downt im e, but also are t arget s for fraud and t heft .

Order fulfillm ent syst em s According t o U.S. Generally Accept ed Account ing Principles ( GAAP) , revenue is not recognizable ( placed on t he books) unt il t he goods or services are delivered. Therefore, m anufact uring, shipping, and service scheduling/ t racking syst em s are key part s of revenue. Alt hough t hese syst em s are not t ypically exposed t o t he I nt ernet as direct ly as order ent ry syst em s ( and t herefore are not as direct ly in t he pat h for fraud and hacking) , t hey are crit ical for m aint aining com pliance wit h regulat ory requirem ent s and revenue r ecognit ion.

4 .1 .3 . Ex pe n se I m pa ct An a lysis Applicat ions t hat support paym ent s are crit ical t o m anaging expenses. I f t hese applicat ions are unable t o funct ion properly, your com pany will likely incur financial penalt ies. Though t he cause of disrupt ion is norm ally accident al, it 's now becom ing feasible for m alicious com pet it ors t o use dist ribut ed denial- of- service ( DDoS) t ools against you t o im pact your revenue or cause painful expenses. Such was t he case in March 2008, when t he gam ing sit e Gala Coral was knocked offline for m ore t han 30 m inut es by a DDoS at t ack, likely at t he direct ion of a com pet it or. [ 2 7 ] [ 27]

See ht t p: / / news.digit alt rends.com / news/ st ory/ 15992/ gam ing_co_hit _by_m aj or_ddos_at t ack/ .

Payroll and t im e- keeping syst em s direct ly im pact expenses. Monit oring t hese syst em s can help prevent fraud and abuse, as was uncovered in an audit of Ashm ont Cleaning Com pany, where an em ployee was changing syst em records of his work hours t o receive ext ra overt im e pay, and st ole m ore t han $30,000 before get t ing caught as t he result of an audit ( Belloli & McNeal 2006) . [ 2 8 ] Two years ago, an unnam ed com pany discovered t hat an em ployee was using it s purchase order syst em t o st eal from t he com pany by rout ing orders t o shell com panies, which t he em ployee and her husband cont rolled. These cases dem onst rat e t hat m onit oring financial syst em s such as t hose t hat process purchasing and payroll requires m ore t han j ust wat ching for availabilit y problem s. Expert s in t he rules and except ions of business processes m ust review report s and workflow, using periodic report s t o cat ch problem s early. Syst em m onit oring can augm ent hum an review by cat ching abuse of account syst em privileges or at t em pt s t o guess passwords for account s wit h elevat ed syst em privileges. [ 28]

See ht t p: / / goliat h.ecnext .com / com s2/ gi_0199- 5592079/ Fraudulent - overt im e- access- t o- t he.ht m l .

4 .1 .4 . Le ga l Re qu ir e m e n t s A com pany m ay incur punit ive expenses from t he governm ent or from part ners in legal cont ract s if it s syst em s do not perform t heir proper funct ions in t erm s of operat ing and prot ect ing sensit ive inform at ion. As dim ensions of legal im pact , let 's consider regulat ory com pliance and cont ract ual obligat ion and how t hey can affect your com pany. 4 .1 .4 .1 . Re gu la t or y com plia n ce My neighbor m anages product ion for a Germ an m anufact urer of indust rial heat exchangers. One spring evening as we were sharing our experiences of being " called int o a crisis" at t he office, he t opped our st ories by explaining t he huge penalt ies he faces if he's caught violat ing local environm ent al regulat ions. His em ployees m ust m aint ain high purit y st andards when discharging wast e wat er used in t he m anufact uring plant . I f an audit or discovers a breach of com pliance, t he com pany will incur shat t ering m onet ary fines and corporat e officers will face personal liabilit y, including a prison sent ence, upon convict ion. Every com pany m ust com ply wit h som e am ount of regulat ion governing it s business operat ions. Com pliance is oft en dem onst rat ed in rout ine audit s and report ing; act ive m onit oring is seldom necessary t o achieve com pliance. Regulat ions are oft en developed t o prot ect inform at ion, and such syst em s t ypically cont ain t he m ost sensit ive inform at ion in your ent erprise. Act ive m onit oring is oft en required by regulat ion, and is a pract ical part of prot ect ing regulat ed syst em s and dat a. Most regulat ory cont rols are aim ed at prot ect ing t he int egrit y and confident ialit y of financial inform at ion. The Sarbanes- Oxley Act of 2002 ( SOX) , which was enact ed aft er a series of account ing scandals, specifies a set of legal cont rols and audit s in an effort t o prevent furt her scandals. SOX Sect ion 404 is aim ed at assessing t he cont rols bounding financial report ing, and encourages com panies t o regulat e and m onit or t he processes for st oring and accessing financial inform at ion. Effect ive m onit oring for financial syst em s should concent rat e on t he dat abases and applicat ions used for such processing. This includes m eans for det ect ing unaut horized changes and abuse of privileges, as well as m onit oring for dat abase adm inist rat ors and ot hers wit h adm inist rat ive access. 4 .1 .4 .2 . Ex a m ple : Gr a m m - Le a ch Blile ly Act A U.S. regulat ion aim ed at personal privacy, t he Gram m - Leach- Blilely Act ( GLBA) cont ains a " Safeguards Rule" which requires com panies t o prot ect non- public personal inform at ion: I n furt herance of t he policy in subsect ion ( a) of t his sect ion, each agency or aut horit y described in sect ion 6805 ( a) of t his t it le shall est ablish appropriat e st andards for t he financial inst it ut ions subj ect t o t heir j urisdict ion relat ing t o adm inist rat ive, t echnical, and physical safeguards— ( 1) t o insure t he securit y and confident ialit y of cust om er records and inform at ion; ( 2) t o prot ect against any ant icipat ed t hreat s or hazards t o t he securit y or int egrit y of such records; and ( 3) t o prot ect against unaut horized access t o or use of such records or inform at ion which could result in subst ant ial harm or inconvenience t o any cust om er. [ 2 9 ] [ 29]

U.S. Code Tit le 15,6801: Prot ect ion of Nonpublic Personal I nform at ion.

The GLBA applies only t o financial inst it ut ions, but requires prot ect ion for syst em s t hat st ore nonpublic personal inform at ion, such as cust om er dat abases, online banking, and brokerage syst em s. 4 .1 .4 .3 . Ex a m ple : Pa ym e n t Ca r d I n du st r y D a t a Se cu r it y St a n da r d The Paym ent Card I ndust ry Dat a Securit y St andard ( PCI DSS) prescribes m onit oring for credit card dat a in t he sect ion, " Regularly Monit or and Test Net works." I n Sect ion 10.6, it specifically requires " Review logs for all syst em com ponent s at least daily. Log reviews m ust include t hose servers t hat perform securit y funct ions like int rusion det ect ion syst em ( I DS) and aut hent icat ion, aut horizat ion, and account ing prot ocol ( AAA) servers ( for exam ple, RADI US) . Not e: Log harvest ing, parsing, and alert ing t ools m ay be used t o achieve com pliance wit h Requirem ent 10.6." The PCI DSS requires t hat specific syst em s, including AAA servers, be m onit ored for abuse. 4 .1 .4 .4 . Ex a m ple : St a n da r ds for cr it ica l in fr a st r u ct u r e pr ot e ct ion Som e form s of com pliance are volunt ary and very specific t o t he indust ry in which t he com pany operat es. Though not m andat ory, com pliance wit h such guidelines is oft en required by execut ive m anagem ent t o avoid ext ernal scrut iny and lawsuit s. The " Nort h Am erican Elect ric Reliabilit y Council ( NERC) Crit ical I nfrast ruct ure Prot ect ion Com m it t ee ( CI PC) Securit y Guidelines for t he Elect ricit y Sect or" [ 3 0 ] provides requirem ent s for securing, bot h physically and elect ronically, crit ical syst em s for elect ric ut ilit ies. The guideline prescribes det ailed securit y configurat ions and m onit oring for any business syst em t hat m ust connect t o a cont rol syst em . This st andard

requires you t o discover t he syst em s connect ing t o your cont rol syst em s, such as t rouble t icket ing or invent ory syst em s, and m onit or t hem for securit y breaches. [ 30]

NERC. Securit y Guidelines for t he Elect ricit y Sect or ( May 2005) , p. 4.

4 .1 .4 .5 . Con t r a ct u a l obliga t ion Com panies t hat offer packaged inform at ion t echnology services are oft en bound t o cont ract requirem ent s for availabilit y, int egrit y, and perform ance. Such com panies m ust m onit or t heir syst em s very carefully, t o help ensure t hat cont ract requirem ent s are m et before t heir cust om ers discover a cont ract breach and exercise t heir cont ract ual privileges, which oft en include financial penalt ies. I n recent years, large com panies have t urned increasingly t o Applicat ion Service Providers ( ASPs) t o offload applicat ion processing. A com m on requirem ent specified in such cont ract s includes securit y m onit oring. [ 3 1 ] For sensit ive applicat ions and dat a, securit y m onit oring is vit ally im port ant , and such cont ract s oft en require m onit oring of specific segm ent s or syst em s as part of t he agreem ent . [ 3 1 ] Cisco Syst em s. Evaluat ing Applicat ion Service Provider Securit y for Ent erprises ; ht t p: / / www.cisco.com / web/ about / securit y/ int elligence/ asp- eval.ht m l ( 2008) .

4 .1 .5 . Se n sit ivit y Pr ofile Sensit ive dat a is t he m ost com m on and obvious place t o t arget m onit oring. I nform at ion m ay be considered sensit ive because it is associat ed wit h an individual's privacy, it has com pet it ive value ( int ellect ual propert y) , or it has governm ent - labeled handling requirem ent s ( classified inform at ion) . 4 .1 .5 .1 . Syst e m s t h a t a cce ss pe r son a lly ide n t ifia ble in for m a t ion ( PI I ) The t hriving m alware underground is fueled largely by t he abilit y t o st eal, m arket , and sell t he privat e inform at ion of individuals. This inform at ion has a m arket value because t hieves can use it t o creat e fake ident it ies, giving t hem t he opport unit y t o st eal t housands of dollars from t heir vict im s via t he vict im s' bank, credit card, or brokerage. Wit h t he advent of out sourcing and ASPs, it 's becom ing feasible for com panies t o relinquish all responsibilit y for st oring sensit ive dat a about em ployees and cust om ers. For com panies t hat m ust m aint ain such syst em s, it should be fairly obvious which applicat ions access such sensit ive inform at ion. Recent legislat ion in m any count ries out lines t he definit ion of PI I and t he required prot ect ions. I t usually involves a unique personal ident ifier com bined wit h ot her dat a, such as a passport num ber st ored along wit h t he person's nam e. Here are exam ples of syst em s t hat com m only st ore PI I :

Payroll syst em s, which oft en cont ain bank account inform at ion for direct deposit of paychecks, along wit h t ax ident ificat ion num bers and Social Securit y num bers Hum an resources syst em s, which oft en cont ain Social Securit y num bers, im m igrat ion ident ifiers, and ot her unique ident ifiers for use in com plying wit h em ploym ent law Syst em s t hat access credit card det ails, such as cust om er dat abases and purchasing syst em s Medical records syst em s, which cont ain inform at ion prot ect ed under t he Healt h I nsurance Port abilit y and Account abilit y Act of 1996 ( HI PAA) , such as individuals' healt h hist ory 4 .1 .5 .2 . Syst e m s t h a t a cce ss con fide n t ia l in for m a t ion Since 2005, report s of hackers using social engineering t echniques t o st eal com pet it ive int elligence have repeat edly surfaced. These t echniques have involved sending t arget ed t roj ans t o individuals wit hin t he vict im com pany ( com m only known as spearphishing ) , plant ing " back doors" t hat are lat er used t o st eal t he dat a. I n 2006, for exam ple, CERT/ CC warned t hat " The st ealt hy at t acks have frequent ly been sent t o a specific person at t he t arget ed organizat ion and show t hat at t ackers are researching t he best way t o convince t he vict im t hat t he docum ent cont aining t he Troj an horse is real." [ 3 2 ] Dat a has also been st olen wit h m ore t radit ional at t ack t echniques such as guessing weak passwords. SAP allegedly used such t echniques in 2007, when it s em ployees were accused of st ealing volum es of Oracle's soft ware and support m at erials in an at t em pt t o gain an upper hand in engaging cust om ers. [ 3 3 ] [ 32]

See ht t p: / / www.securit yfocus.com / news/ 11222/ .

[ 33]

See ht t p: / / www.securit yfocus.com / news/ 11453/ .

Com panies should classify propriet ary inform at ion and label it for rest rict ed dist ribut ion. Such inform at ion represent s t he m ost crit ical secret s a com pany m aint ains, and it can include exam ples such as t he following:

Product designs, including blueprint s, schem at ics, soft ware design specificat ions, and st eps in a m anufact uring process Com ponent s and assem bly inst ruct ions List s of cust om ers, cont act s, and order hist ory Sales figures and leads Orders in process Plans for m ergers and acquisit ions Product s/ ideas under developm ent Pat ent s not yet filed or approved Don't overlook t he value of m onit oring t he com put ers and account s of your senior execut ives. These syst em s are surely accessing highly privileged inform at ion, m aking t hem t he likely t arget s of spearphishers. 4 .1 .5 .3 . Syst e m s t h a t a cce ss cla ssifie d in for m a t ion Handling classified inform at ion ( as cont rolled by official m arkings and procedures of a governm ent agency, bound by law) requires special handling and prot ect ions. For exam ple, t he U.S. Depart m ent of Defense direct ive num ber 5220.22- M set s fort h requirem ent s described in t he Nat ional I ndust rial Securit y Program Operat ing Manual ( NI SPOM) . [ 3 4 ] This direct ive requires t hat all TOP SECRET and SECRET inform at ion be m onit ored wit h a NI DS. I t also requires special m onit oring for any syst em connect ed t o a classified inform at ion syst em , and t hat such connect ions are m ade via a Cont rolled I nt erface ( CI ) , described as follows: [ 3 4 ] Nat ional I ndust rial Securit y Program Operat ing Manual ( NI SPOM) , DoD 5220.22- M, Sect ion 5- 307 ( February 2006) .

8- 702. Cont rolled I nt erface Requirem ent s. The CI shall have t he following propert ies: a. Adj udicat ed Differences. The CI shall be im plem ent ed t o m onit or and enforce t he prot ect ion requirem ent s of t he net work and t o adj udicat e t he differences in securit y policies . Based on t his requirem ent , a connect ion t o a classified inform at ion syst em requires m onit oring and enforcem ent t o prot ect t he inform at ion it cont ains. This serves as an excellent exam ple of a syst em for which t arget ed m onit oring is required.

4 .1 .6 . Risk Pr ofile Due t o availabilit y requirem ent s and vendor packaging, som e syst em s cannot be regularly pat ched ( and som e cannot be pat ched at all) . These syst em s present an increased risk t o t he ent erprise, and you should m onit or t hem carefully. You can use prevent ive cont rols such as firewalls and int rusion prevent ion syst em s ( I PSs) t o augm ent t heir securit y, but for sensit ive or crit ical syst em s, act ive securit y m onit oring is a necessary t ool t o furt her reduce risk. For exam ple, Supervisory Cont rol and Dat a Acquisit ion ( SCADA) syst em s are used t o cont rol and m anage a nat ion's crit ical infrast ruct ure, including energy, wat er, t ransport at ion, and com m unicat ion syst em s. I n recent years, t hese syst em s have been convert ed t o com m odit y operat ing syst em s and net works ( such as Windows and TCP/ I P) , m oving away from propriet ary syst em s. Connect ions t o SCADA syst em s from ext ernal net works have placed t hem in t he line of fire for securit y t hreat s. Weaknesses in t hese syst em s have been repeat edly dem onst rat ed, and t he role t hese syst em s play in crit ical infrast ruct ure m akes t hem ext rem ely vulnerable t o elect ronic warfare and t errorist at t acks. Figure 4- 1 shows a screenshot from a SCADA wat er plant pum ping syst em . Figu r e 4 - 1 . Scr e e n sh ot fr om a SCAD A w a t e r pla n t pu m pin g syst e m ( se e ht t p:/ / w w w .nor da t a sys.com / scr e e ns.ht m )

SCADA syst em s are com m only plagued by t hree broad vulnerabilit ies, as described at a Black Hat Federal Briefing: [ 3 5 ] [ 3 5 ] RG & DM ( X- Force I nt ernet Securit y Syst em s) . " SCADA Securit y and Terrorism , We're Not Crying Wolf," Black Hat Federal Briefings 2006.

No aut hent icat ion These syst em s have aut om at ed com ponent s connect ed and running aut o nom ously . This m akes t hese syst em s ripe for worm s and ot her self- replicat ing at t acks.

No pat ching Com m only due t o int olerance for downt im e, t hese syst em s run for m any years wit hout int errupt ion or at t ent ion for pat ching.

No isolat ion/ firewalling As syst em s are int erconnect ed and t he net work is ext ended, t hese syst em s are increasingly reachable via t he I nt ernet and ot her less- prot ect ed net works. Because t hese problem s are endem ic t o SCADA syst em s, securit y m onit oring is required t o m it igat e t he subst ant ial risk t hey present . Som e vulnerabilit y scanning t ools cont ain built - in SCADA probes, as illust rat ed in t he Nessus screen capt ure in Figure 4- 2 . Figu r e 4 - 2 . N e ssu s sca n opt ion s for SCAD A syst e m s

4 .1 .6 .1 . Risk a sse ssm e n t s An inform at ion t echnology securit y assessm ent ( com m only called a risk assessm ent ) is a focused st udy int ended t o locat e I T securit y vulnerabilit ies and risks. The risk assessm ent norm ally includes an evaluat ion of bot h physical and t echnical risks, and uses personal int erviews, vulnerabilit y scans, and on- sit e observat ion t o evaluat e risk. Risk assessm ent s are aim ed at evaluat ing an organizat ion's com pliance wit h a set of securit y st andards. A com m on securit y fram ework used as a st andard in such assessm ent s is I SO 17799, which defines im port ant adm inist rat ive securit y guidelines for organizat ions . NI ST Special Publicat ion 800- 30, " Risk Managem ent Guide for I nform at ion Technology Syst em s," describes nine st eps for conduct ing inform at ion securit y risk assessm ent s, as shown in Figure 4- 3 . Figu r e 4 - 3 . N I ST r isk a sse ssm e n t pr oce ss

This process provides st ruct ured analysis of t hreat s, vulnerabilit ies in deployed infrast ruct ure, and available cont rols t o produce a report of securit y risks ( gaps) and recom m ended cont rols. Where recom m ended cont rols are considered t oo cost ly or im pract ical , securit y m onit oring can oft en provide an accept able alt ernat ive for m it igat ing securit y risks. I n t he securit y reviews I 've conduct ed, business unit s have oft en been unwilling t o im plem ent st rict cont rols t o m it igat e risk, fearing it will slow progress on t im e- sensit ive proj ect s. I n t hose cases, I 've required t he business unit 's execut ive t o accept t he risk by signing a form al docum ent explaining t he risks of t he proj ect . This solut ion is effect ive for raising awareness, but it leaves t he risks unm it igat ed. Securit y m onit oring offers a unique com prom ise, allowing t he business owner t o m it igat e risks by sponsoring t arget ed m onit oring. Risk assessm ent s com m only discover present risks via scans by specialized t ools, producing report s sim ilar t o t he Qualys report shown in Figure 4- 4 .

Figu r e 4 - 4 . Ex a m ple Qu a lys sca n fr om r isk a sse ssm e n t

Your risk assessm ent should t est com pliance wit h your est ablished policies, as described in Chapt er 2 . I n addit ion, it should incorporat e net work address dat a, as described in Chapt er 3 , allowing you t o aim specific t est s according t o where syst em s are placed on t he net work. For exam ple, if your policy requires use of I T- sanct ioned virt ual privat e net work ( VPN) connect ions for all rem ot e adm inist rat ion, you should audit t he syst em s in t he dat a cent er for rogue connect ions such as m odem s.

4 .1 .7 . Visibilit y Pr ofile Your m ost visible syst em s are likely t hose exposed t o t he I nt ernet , providing access t o cust om ers, shareholders, part ners, and t he press. This includes your com pany websit e, blogs, support t ools, and ot her cust om er- accessible syst em s. According t o st at ist ics from t he 12t h Annual Com put er and Crim e Survey, [ 3 6 ] websit e defacem ent s have slight ly increased in recent years, but rem ain a very sm all percent age of securit y incident s. Many com panies are generat ing t heir cont ent dynam ically, even rebuilding t heir st at ic cont ent on a night ly basis. These advances seem t o reduce t he need for m onit oring t he st at ic web cont ent present ed by your com pany. However, as not ed in t he sidebar, " A New At t ack on Websit es," defacem ent s are only one dim ension of securit y t hreat s for your st at ic cont ent .

[ 3 6 ] Robert Richardson. " 2007 Com put er Crim e and Securit y Survey" ; ht t p: / / www.gocsi.com / form s/ csi_survey.j ht m l ( March 2008) .

A N e w At t a ck on W e bsit e s Because m ost websit e defacem ent s are int ended t o draw at t ent ion t o t he at t ack, t hey are com m only at t ribut ed t o vandals and polit ical act ivist s. A m ore serious and pract ical concern regarding t he int egrit y of your websit e involves com prom ise for t he purpose of m alware dist ribut ion. Hist orically, such sit es were lim it ed t o t he seedy side of t he I nt ernet , where vict im s were lured by t he prom ise of salacious im ages or videos. That 's now changing, as lessprot ect ed sit es com m only visit ed and im plicit ly t rust ed by users are new t arget s for hackers t o use in dist ribut ing m alware. As st at ed in t he Google Technical Report , " All Your iFram es Point t o Us," [ 3 7 ] a sam pling of m illions of random URLs dem onst rat ed t hat , t hough adult sit es cont inued t o cont ain t he largest proport ion of m alicious cont ent , several ot her cat egories had appreciable percent ages of m alware as well. According t o t he iFram es paper, t wo t ypes of sit es are used in m alware dist ribut ion: landing sit es, used t o exploit or redirect a vict im 's browser, and m alware dist ribut ion sit es, used t o download t he payload t o t he vict im 's host . Should an adversary com prom ise your sit e, he will likely t urn it int o a landing sit e by adding an invisible iFram e t o one of t he st at ic web pages. The iFram e cont ains JavaScript exploit code t o com prom ise your visit ors' browsers. I f your sit e host s user- cont ribut ed cont ent such as page com m ent s or a Wiki, t he adversary can deploy exploit code direct ly wit hout needing t o first com prom ise your sit e. Such t echniques are com m only used in clickj acking and banner ad m alware at t acks. Wit h clickj acking, t he at t acker overlays invisible m alicious cont ent on a page so t hat when t he user clicks a link, he is act ually clicking a URL under t he hacker's cont rol. Wit h banner ads, at t ackers deploy m ult iple redirect s via a series of banner ads, event ually direct ing users t o a sit e host ing m alware, which is aut om at ically inst alled on t he user's com put er. This at t ack is possible because t he prim ary ( first redirect ) banner ads are benign, drawing no at t ent ion when placed on leading search engines and popular sit es.

[ 3 7 ] Provos, Mavrom m at is, Raj ab, and Monrose. " All Your iFram es Point t o Us," ht t p: / / research.google.com / archive/ provos- 2008a.pdf ( March 2008) .

Figu r e 4 - 5 . Ex a m ple of w e bsit e de fa ce m e n t de t e ct ion by Pe r isca n

Several t ools and services can m onit or for websit e defacem ent s. These t ools download each page and linked page, com put ing and st oring a hash of t he cont ent s. They t hen periodically check t he pages against t he st ored hash, calling out differences. Should t he syst em discover a discrepancy, an adm inist rat or is not ified t o analyze t he change and can t hen det erm ine whet her t he change was aut horized. Sit e m onit oring syst em s such as Cat bird and Periscan ( t he lat t er shown in Figure 4- 5 ) will wat ch your websit e for changes and alert you, allowing you t o det erm ine whet her t he change was aut horized. For high- visibilit y syst em s such as your com pany's websit e, t hese t ools allow you t o augm ent t he securit y m onit oring of t radit ional t ools such as a NI DS.

4 .2 . Pr a ct ica l Con side r a t ion s for Se le ct in g Ta r ge t s My Nokia sm art phone drives m e crazy alert ing m e t o t hings t hat I don't need t o know, t hat I can't do anyt hing about , and t hat int erfere wit h basic operat ions ( such as m aking phone calls) . One part icularly annoying m essage, " Packet Dat a St art ed," oft en appears j ust as I 'm beginning t o dial a num ber, forcing m e t o acknowledge t he m essage and rest art m y num ber dialing. Nokia t hought fully included t his feat ure t o keep users wit hout unlim it ed dat a plans inform ed t hat t heir phone is about t o incur dat a charges. I n m y case, I have no need t o see t he m essage, since I have an unlim it ed dat a plan. Don't configure your syst em s like m y sm art phone, collect ing event s for which you don't int end t o t ake act ion. I f you're able t o fully m onit or a syst em , but you can't do anyt hing about t he event s t hat are generat ed, why bot her m onit oring it ? Event collect ion is always necessary t o support invest igat ions. Even when you're not act ively m onit oring event s, you m ust collect t he event s t o support incident response. For t arget ed m onit oring, however, event s t hat you cannot m it igat e are a dist ract ion and should not be alert ed. For exam ple, Figure 4- 6 shows an Oracle alert from a Securit y I nform at ion Manager ( SI M) syst em . Figu r e 4 - 6 . Or a cle Applica t ion Se r ve r a le r t

Th e I m por t a n ce of Se con da r y Eve n t s " Because of t he probabilit y t hat an int ruder will t am per wit h logs on a com prom ised syst em , it is im port ant t o safely collect copies of log m essages in a cent ralized locat ion." [ 3 8 ] Policy- based m onit oring recom m ends a narrow, t arget ed approach t o securit y m onit oring t hat filt ers out all unrelat ed event s. I t requires deep analysis and t uning t o weed out ext raneous event s. What should you do wit h t he event s shunned from your m onit oring syst em ? St ore t hem in a safe place t o support incident response and invest igat ions . When responding t o a securit y incident , you will be focused on specific syst em s affect ed during a specific period. " Secondary" event s t hat m ight norm ally be t uned out of a m onit oring syst em t ake on new im port ance. These event s are useful t o illum inat e a problem , aid in m it igat ion, t race act ivit y, and at t ribut e act ions. Collect as m any event s as you can st ore, and keep t hem for as long as you can. Archive all t he securit y event s you've collect ed t o a server so t hat you can search t hem easily t o support your incident response and invest igat ion effort s. You can forward select ed event s in parallel t o your securit y m onit oring syst em for processing, where t hey will be analyzed or dropped based on t he m onit oring philosophy and procedures. Though policy- based m onit oring recom m ends t uning out m any event s, it 's st ill im port ant t o st ore all event s ( m onit oring and secondary event s) in archival st orage for as long as possible.

[ 38]

Richard Beij t lich. Ext rusion Det ect ion ( Addison- Wesley Professional, 2005) .

This event seem s t o det ail an at t ack against an Oracle applicat ion server. I f you know t hat you're not running Oracle applicat ion soft ware, or t hat no one at t he com pany can analyze t he t arget syst em s for signs of at t ack or m it igat e t he problem , you're wast ing your t im e alert ing about t he event . Polit ical blockades are anot her source of frust rat ion for securit y m onit oring. Alt hough t he securit y t eam m ay do an excellent j ob discovering securit y problem s, get t ing t he support t eam s t o address t he problem s m ay be polit ically infeasible. Don't wast e your t im e t arget ing such syst em s, if you can avoid it . Take peer- t o- peer ( P2P) soft ware, for exam ple. I t 's relat ively easy t o det ect , and you can likely t rack down t he individuals using it on t he corporat e net work ( e.g., t o download m ovies) . I f m anagem ent and hum an resources are unwilling t o enforce policies governing it s use, however, t here's no benefit t o t racking t hem down in t he first place.

4 .3 . Re com m e n de d M on it or in g Ta r ge t s To help you det erm ine t he best t arget s for securit y m onit oring, you m ust build on your securit y policies and docum ent ed net work t opology, as we described in Chapt ers Chapt er 2 and Chapt er 3 . Arm ed wit h t hose decisions and docum ent ed knowledge, you should conduct a st ruct ured assessm ent of t he syst em s t hat com prise your com pany. 1.

2.

Conduct a BI A. Most ent erprises have a t eam focused on business cont inuit y and disast er preparat ion. Cont act t hem and ask for t he result s of t he m ost recent BI A, or ask t hem t o conduct one in preparat ion for securit y m onit oring. The BI A will produce, am ong ot her t hings, a list of crit ical I T syst em s. This is a good place t o find t arget s for inform at ion securit y m onit oring. The BI A will call out t im e- crit ical business processes and MTDs. Ordered by least am ount of MTD, t his list can becom e a priorit y order for applying securit y m onit oring. Syst em s ident ified in such an assessm ent will likely include t hose responsible for revenue generat ion and t hose wit h high visibilit y profiles. Conduct an I nform at ion Technology Securit y Assessm ent ( I TSA) . This form al appraisal will analyze t he securit y of your I T syst em s t o det erm ine areas of risk. I t should use t he policies and net work knowledge t hat you've docum ent ed as a benchm arking st andard. To t hat end, it will incorporat e exam inat ion of regulat ory com pliance, cont ract ual/ legal requirem ent s, and syst em s t hat access sensit ive dat a. The I TSA will produce a list of act ion it em s as well as an assessm ent of risk present ed by your I T syst em s. Using t he result s of t his assessm ent , you can develop a list of syst em s t hat require t arget ed m onit oring, especially where prevent ive cont rols are im pract ical t o apply.

When you use t he BI A and I TSA, a list of syst em s will em erge for which you can t arget securit y m onit oring. The list will focus your m onit oring on t he business priorit ies and concret e risks your com pany faces. Based on our experience, t he best t arget s for focused securit y m onit oring are t hose t hat can cause t he m ost harm , by way of dat a loss or revenue loss. Most com panies should t herefore m onit or syst em s t hat do t he following:

Access sensit ive dat a Syst em s governed by regulat ory com pliance requirem ent s t hat st ore int ellect ual propert y, or t hat st ore privat e dat a have enorm ous value, and should receive your prim ary at t ent ion.

Present a high risk profile Syst em s t hat present special risk t o your com pany should be given careful at t ent ion for securit y m onit oring. Norm ally, t hese syst em s are ident ified during a risk assessm ent or ext ernal audit .

Generat e revenue Syst em s t hat are responsible for or direct ly im pact revenue generat ion are obvious places t o focus your securit y m onit oring.

4 .4 . Ch oosin g Com pon e n t s W it h in M on it or in g Ta r ge t s Once you've select ed t he I T syst em s t hat need m onit oring, you m ust analyze t he com ponent m akeup of t hese t arget ed syst em s t o select event feeds ( which we'll cover in dept h in t he next chapt er) . To det erm ine t he com ponent m akeup, you should break down each syst em int o it s core elem ent s, including t he dat abases, web servers, applicat ion servers, and various host s upon which t hese solut ions run. Depending on t he com ponent s in your solut ion, you m ight collect syslog m essages from Unix/ Linux servers ( and from Windows servers, if t hey're configured wit h add- on packages) , m onit or t he AUD$ t able on Oracle dat abases, analyze t he access_log from Apache web servers, and so on. Your syst em will also depend on com plem ent ary services such as aut hent icat ion servers ( LDAP, Act ive Direct ory, NI S+ , et c.) , caching servers, net work at t ached st orage ( NAS) , and so fort h. Analyzing your policies will help you det erm ine which com plem ent ary services you should m onit or t o com plet e your t arget ed m onit oring plan. You should even consider net work devices t hat serve access t o your syst em ; t ools such as Net Flow and syslog can help t race device configurat ion changes, am ong m any ot her t hings.

4 .4 .1 . Ex a m ple : ERP Syst e m To illust rat e t he process of select ing com ponent s for m onit oring, consider an inst allat ion of SAP R/ 3, which by definit ion is a t hree- t ier archit ect ure com posed of present at ion servers, applicat ion servers, and a dat abase server. I n a t ypical inst allat ion, t he present at ion and applicat ion servers would be load- balanced across t wo or m ore Linux servers. The dat abase server is oft en a single Oracle dat abase inst ance, load- balanced t ransparent ly using Oracle's nat ive funct ionalit y t o st raddle inst ances across physical servers. Assum ing a t ot al of five servers ( t wo present at ion servers, t wo applicat ion servers, and one dat abase server) , t he following logs ( also illust rat ed in Figure 4- 7) provide rich event sources for securit y m onit oring:

Host syslog from all five servers NI DS logs from dat a cent er gat eways Audit files from each applicat ion server Oracle audit logs from t he Oracle 10 g dat abase, including specialized audit logging for m ast er record t ables Account ing audit t rails ( st ored in dat abase t ables) [ 3 9 ] [ 3 9 ] Dr. Pet er J. Best . " Audit Trail Analysis for Fraud Cont rol wit h SAP R/ 3." CACS 2005 I SACA Oceania Conference.

Net Flow logs int o/ from t he dat a cent er Figu r e 4 - 7 . W h e r e t o colle ct logs for SAP R/ 3 m on it or in g

4 .4 .2 . Ga t h e r in g Com pon e n t D e t a ils for Eve n t Fe e ds I n Chapt er 5 , we'll det ail how t o begin select ing event sources from our various com ponent s int o securit y m onit oring syst em s. Now t hat we've select ed our com ponent s t o m onit or, we have one final st ep t o com plet e before we can begin t o collect t hese event s for analysis. Recall t hat in Chapt er 3 we det ailed t he im port ance of docum ent ed net work t opology t o provide cont ext for securit y event s. We'll put som e of t hat knowledge int o pract ice here, docum ent ing t he specific host and net work inform at ion of t he com ponent s in our t arget ed syst em . 4 .4 .2 .1 . Se r ve r I P a ddr e sse s a n d h ost n a m e s We m ust enum erat e t he I P addresses and nam es of each server in our solut ion. This will allow us t o det erm ine t he valid t raffic pat t erns bet ween servers and priorit ize alert s cont aining t he host nam es and I P addresses of t arget syst em s. 4 .4 .2 .2 . " Ge n e r ic" u se r I D s I f we were t o list t he processes running on t he servers, we would find t he applicat ion com ponent s running wit h a generic user I D. This generic account should be " regist ered" t o t he applicat ion, and should run wit h very lim it ed privileges. To m onit or com ponent s effect ively, we need t o enum erat e t hese user I Ds. Once we build t his list , we will be able t o at t ribut e behavior t o regist ered user I Ds, m aking it easy t o see t hat t he behavior is benign. For exam ple, it 's com m on t o observe a large num ber of sim ult aneous connect ions in t he dat abase originat ing from t he app server. We can safely ignore t his event once it 's clear t hat it 's t he " WebSphere" user connect ing t o Oracle via WebSphere's built - in connect ion pooling. Generic account s are used for running services, such as t he " nobody" or " ht t p" user t hat runs t he Apache web server, or t he " oracle" user which execut es t he dat abase processes on t he server. Generic account s are also used t o connect t o ot her resources, such as t he " connect as" user I D specified when connect ing t o t he dat abase. 4 .4 .2 .3 . Adm in ist r a t or u se r I D s Legit im at e syst em adm inist rat ion account s m ust also be enum erat ed for sim ilar reasons. This allows you t o at t ribut e adm inist rat ive act ivit y t hat m ay show up as securit y event s. For exam ple, if correlat ed securit y event s show a user I D connect ing t o t he dat abase server and rest art ing t he dat abase list ener, we can m at ch t he user I D wit h t he " regist ered" dat abase adm inist rat or for t hat dat abase, docum ent ing t hat it 's in t he scope of his responsibilit ies. 4 .4 .2 .4 . D a t a ba se de t a ils To effect ively m onit or t he dat abase, we m ust det ail t he dat abase set up, enum erat ing t he following inform at ion:

Schem as used Account s used ( oft en t he user I D for applicat ion access is different from and m ore lim it ed t han t he schem a

owner account ) , along wit h account det ails such as t he roles grant ed t o t he account s Sensit ive obj ect s ( t ables, views, procedures) How act ivit y is logged ( norm ally logged t o AUD$ ) Access cont rols in place ( privileges grant ed t o enum erat ed account s against dat abase obj ect s) 4 .4 .2 .5 . Acce ss con t r ols Last ly, we m ust enum erat e t he access cont rols act ive in t he environm ent . Access cont rol syst em s such as firewalls and host I PS logs t hat can be useful for t arget ed m onit oring. For exam ple, if a firewall is logging DENY m essages for users t rying t o direct ly access t he dat abase from out side t he dat a cent er, analysis of t hose logs can prove useful in m onit oring and defending t he dat abase. Sim ilarly, securit y soft ware, such as Tripwire and Port Sent ry, log m essages t hat we can analyze t o beef up our t arget ed m onit oring capabilit y.

4 .5 . Bla n co W ir e le ss: Se le ct in g Ta r ge t s for M on it or in g Like m ost wireless phone carriers, Blanco collect s Social Securit y num bers from cust om ers when set t ing up t heir account s, illust rat ed in Figure 4- 8. Blanco uses t his inform at ion t o request and report credit inform at ion from one of t he large credit report ing services. As described in Chapt er 2 , Blanco has form ed policies designed t o prot ect such inform at ion and com ply wit h governm ent regulat ion. Figu r e 4 - 8 . Bla n co's a ccou n t e n t r y for m

Blanco's account m anagem ent syst em , shown in Figure 4- 9, is com posed of a com m on, t hree- t ier archit ect ure:

Apache web servers running on t hree load- balanced Red Hat Ent erprise Linux ( RHEL) servers An I BM WebSphere Applicat ion Server running on a VMware ESX server farm An Oracle dat abase ( 11 g) , clust ered and dist ribut ed across t wo RHEL servers An OpenLDAP server running on one RHEL server Dat a cent er gat eways ( t wo Cisco I OS 7200 rout ers) A NI DS ( one Sourcefire Snort server) Figu r e 4 - 9 . Bla n co W ir e le ss a ccou n t m a n a ge m e n t syst e m

Alt hough not pict ured in Figure 4- 9, load- balancing int elligence is built int o t he net work, reducing t he need for separat e, specialized equipm ent .

4 .5 .1 . Com pon e n t s t o M on it or I n Chapt er 5 , we'll discuss t he best event feeds t o use for m onit oring our t arget syst em . Our t ask for now, however, is t o ident ify each com ponent in t he solut ion. Considering t he t wo policies art iculat ed in Sect ion 1.9 , we m ust configure m onit oring for t he Dat a Prot ect ion Policy and t he Server Securit y Policy. 4 .5 .1 .1 . D a t a Pr ot e ct ion Policy To effect ively m onit or com pliance wit h t he Dat a Prot ect ion Policy, we m ust m onit or t he dat abase for plain- t ext PI I in st orage, t he net work gear providing access t o t he dat abase, and t he dat abase configurat ion, t o m ake sure t he dat abase com plies wit h our hardening specs. Our Dat a Prot ect ion Policy requires us t o do t he following:

Audit dat a st ored in t he dat abase via scheduled queries, or by m onit oring t he audit log of t he dat abase it self t o ensure t hat t he dat a is encrypt ed properly. Monit or net work t raffic, which is sat isfied by NI DSs deployed at t he dat a cent er gat eway. Ensure t hat t he dat abase configurat ion is hardened. This will require a rout ine audit against t he dat abase, using a program m at ic, scheduled vulnerabilit y t est ing t ool such as Nessus. 4 .5 .1 .2 . Se r ve r Se cu r it y Policy The Server Securit y Policy will require m onit oring of every deployed server in our solut ion, including web servers, applicat ion servers, and dat abase servers. We m ust also m onit or t he LDAP server t o t rack account access, and m ust access net work feeds ( using Net Flow) t o m onit or t raffic accessing t he I nt ernet from t hese servers.

4 .6 . Con clu sion Deep, proact ive securit y m onit oring is overwhelm ing and unproduct ive if it isn't t arget ed t o specific syst em s. By select ing m onit oring t arget s, you can narrow your focus t o t he m ost crit ical syst em s, m aking t he m ost of your securit y m onit oring equipm ent and st aff. By spending t he t im e t o ident ify good m onit oring t arget s, you will avoid wast ing t im e on unproduct ive sources and, m ore im port ant ly, im prove your chances of finding t he m ore serious securit y t hreat s facing your ent erprise. I n t he next chapt er, we'll select event feeds for m onit oring t hese select ed t arget syst em s.

Ch a pt e r 5 . Ch oose Eve n t Sou r ce s I n his book The Paradox of Choice ( Harper Perennial) , aut hor Barry Schwart z st at es: Scanning t he shelves of m y local superm arket recent ly I found 85 different variet ies and brands of crackers…285 variet ies of cookies….13 sport s drinks, 65 box drinks…and 75 t eas…80 different pain relievers…29 different chicken soups…120 past a sauces…275 variet ies of cereal.... Schwart z exam ines durable goods such as elect ronics, t hen surveys life- im pact ing decisions regarding invest m ent s, insurance, ret irem ent , and healt hcare. Schwart z's t hesis is t hat having t oo m any choices can lead t o anxiet y, dissat isfact ion, and regret . I n cont rast , t he choices for event sources, t hough varied, are not nearly as com plex as t hose lam ent ed by Schwart z, and should by no m eans cause you m ent al dist ress! Now t hat you've worked t hrough t he st eps of defining securit y policies, you know your net work, and you've select ed your t arget s, you can build on t hat foundat ion by choosing your event sources. For t he net work, syst em s, and device t ypes you are m onit oring, t here are several corresponding event dat a t ypes from which t o choose. For exam ple, net work rout ers can yield syst em st at us m essages and access- list deny logs via syslog, int erface st at ist ics via Sim ple Net work Managem ent Prot ocol ( SNMP) , net work t elem et ry via Net Flow, as well as deep packet inspect ion result s. Alt hough it m ay be t em pt ing t o use all of t hese sources for securit y event m onit oring, not all of t hem are appropriat e. This chapt er provides an overview of t he various device t ypes and t heir event sources, how you can collect t hem , and how you can inspect t hem for securit y policy violat ions. We've gat hered t he m yriad choices int o a subset of t he best event sources t o help you choose t he appropriat e sources quickly, wit hout becom ing overwhelm ed in t he sea of possible event feeds.

5 .1 . Eve n t Sou r ce Pu r pose To best det erm ine which event sources t o use for m onit oring, you m ust det erm ine how t he event source will be used. Your purpose in collect ing any event source will im pact st orage requirem ent s, dat a ret ent ion policies, and collect ion int ervals. An event source m ay be used for one or m ore of t he following reasons:

Monit oring Event collect ion consist s of cont inuous st ream s of alert s recorded and analyzed in real t im e or near- real t im e. Correspondingly, syst em s select ed for m onit oring have a lower st orage capacit y and a short er dat a ret ent ion policy because t heir window of focus is very near- t erm .

I ncident response and invest igat ion I ncident response and invest igat ion t ypically requires a higher st orage capacit y and longer dat a ret ent ion policy. Event s are oft en st ored in a dat abase for fast querying of recent event dat a ( t ypically wit hin one m ont h) . This t im e fram e depends heavily on event rat e and volum e, wit h high volum e and high event rat es decreasing ret ent ion t im e and increasing st orage requirem ent s.

Regulat ory com pliance, legal request s, and forensics Syst em s t hat fall under t he purview of governm ent regulat ion oft en have special com pliance requirem ent s. These include long- t erm , m ult iyear st orage of event dat a, support ing invest igat ions of legal m at t ers such as lawsuit s. Forensic invest igat ions oft en ext end t o dat a from t he dist ant past , requiring access t o dat a locat ed on t ape backup, t ypically st ored off- sit e. Access t o t his dat a is far from real t im e, as a rest ore from t ape backup can require hours or days t o com plet e. Let 's look at an exam ple legal request . Say you received t he following m essage from your legal counsel:

From: Dewey Cheatem (Legal Counsel) Sent: Thursday, May 8, 2008 10:57 AM To: [email protected] Cc: Michelle Howe (Legal Counsel) Subject: Computer Investigation Request Importance: High ATTORNEY/CLIENT PRIVILEGE: Computer Security Incident Response Team, We are currently involved in litigation from a former employee, Jermison BueSnockley, who states that his co-worker logged in to his email and sent damaging communications to friends, family, and complete strangers that resulted in a police investigation into alleged illicit activities. We need to find out who accessed Jermison's computers on the following dates: October 10 October 12 November 4 November 8

2007 2007 2007 2007

In addition, Mr. Bue-Snockley claims that he was signed up for several email lists of an adult nature from the following sites and dates: ~November 18th - lonely-adult-hookup.com

~November 20th - cannibis-exchange.com Please investigate this incident and provide an initial writeup by the end of next week. Sincerely, Dewey Cheatem, BigBucks Legal Services

How will you address t his request ? I s t his dat a logged? For how long is it ret ained? Will it require a rest ore from backup? I f you've collect ed all necessary event t ypes, your response m ight resem ble t he following:

From: [email protected] Sent: Friday, May 9, 2008 9:15 AM To: Dewey Cheatem (Legal Counsel) Cc: Michelle Howe (Legal Counsel) Subject: Re: Computer Investigation Request Importance: High ATTORNEY/CLIENT PRIVILEGE: Mr. Cheatem / Ms. Howe, I've checked with our lead incident handler and we should be able to provide most of this data by early next week. We won't be able to look directly at Mr. BueSnockley's computer data for 3-5 business days as the backups have already been taken offsite. Regards, Blanco CSIRT

The Blanco CSI RT t eam in t his exam ple has Net Flow, which recorded connect ivit y from a com put er connect ing t o t he I P addresses of t he websit es in quest ion, as well as proxy logs showing t he specific URLs used t o regist er Mr. BueSnockley's em ail address t o t he alleged websit es ( plus several ot hers t hat he m ay not yet have seen) . I n addit ion, t he t eam has logon/ logoff m essages collect ed via syslog from all syst em s, including Mr. Bue- Snockley's. As it t urns out , t he CSI RT invest igat ion found logon event s t hat corresponded wit h t he t im e fram e of t he Net Flow. I t was lat er learned t hat an em ployee on Jerm ison's t eam logged int o his com put er and signed him up for t he list s, in ret aliat ion for a personal conflict .

5 .1 .1 . Eve n t Colle ct ion M e t h ods When select ing event sources for securit y m onit oring, you m ust fact or in t he m et hod for collect ion, since it will affect perform ance and t im eliness. As depict ed in Figure 5- 1, t here are t wo general m et hods for collect ing event s: push and pull. Figu r e 5 - 1 . Eve n t colle ct ion u sin g bot h pu sh a n d pu ll m e t h ods

The push m et hod Wit h t he push m et hod, event s are sourced from t he device at specified int ervals or in real t im e, as configured on t he device it self. The event collect or m ust be prepared t o receive t he event s as t hey occur. Exam ples of t his m et hod include syslog m essages, access- list ( ACL) logs, and Net Flow.

The pull m et hod Wit h t he pull m et hod, event s are st ored locally on t he originat ing device and are ret rieved by t he collect or. That is, t he collect or init iat es collect ion of t he event m essages from t he device t hat generat es t he event s. Two com m on prot ocols for pulling event dat a are Securit y Device Event Exchange ( SDEE) and t he fam iliar SNMP. SDEE was developed by a working group at I CSA Labs, called t he I nt rusion Det ect ion Syst em s Consort ium ( I DSC) , which consist ed of Cisco, Fort inet , I NFOSEC Technologies, I SS, SecureWorks, Sourcefire, Sym ant ec, and Tripwire, which provide t he following inform at ion regarding t he prot ocol: The Securit y Device Event Exchange ( SDEE) specifies t he form at of t he m essages as well as t he prot ocol used t o com m unicat e t he event s generat ed by securit y devices. SDEE is designed t o be flexible and ext ensible so t hat vendors can ut ilize product specific ext ensions in a way t hat m aint ains m essaging com pat ibilit y. SDEE builds upon t he XML, HTTP and SSL/ TLS indust ry st andards t o facilit at e adopt ion by vendors and users by allowing t hem t o ut ilize exist ing soft ware t hat im plem ent s t hese st andard int erfaces. SDEE Dat a Form at

Current ly addresses t he dat a form at of int rusion det ect ion/ prot ect ion alert s Event s form at t ed as XML elem ent s Form at s specified using XML schem a. Support s schem a validat ion Docum ent ed wit h schem a annot at ions Designed t o support ext ensions and t o evolve while m aint aining core com pat ibilit y

Designed t o be efficient ly m anaged SDEE Com m unicat ion

Ext ensible. Support for different t ransport bindings Current ly specifies bindings t o HTTP and HTTP over SSL/ TLS Using exist ing t echnology eases adopt ion for providers and client s Support for aut horizat ion and encrypt ion wit h exist ing HTTP and SSL/ TLS Client init iat ed event ret rieval SDEE has been im plem ent ed on num erous securit y devices, such as firewalls, NI DSs, rout ers, and ot her net work devices. I t is also used on various host int rusion prevent ion syst em ( HI PS) solut ions, such as Cisco's CSA soft ware. The SDEE st andard, released in 2004, has been adopt ed prim arily by Cisco, I SS, and Sourcefire, which developed and docum ent ed t he st andard. The I DSC was renam ed t he Net work I PS Product Developers ( NI PD) Consort ium in March 2005. The pull collect ion m et hod offers som e advant ages over t he push collect ion m et hod, including event - rat e cont rol, preparsing ( allowing you t o specify t he specific event s t ypes t o pull) , and guarant eed delivery. These benefit s are illust rat ed in Figure 5- 2. I n cont rast , m ost connect ionless UDP- based push t echnologies, such as syslog, SNMP, and Net Flow, only provide " best effort " delivery. Consequent ly, pushed event s offer less cont rol, especially in cont rolling event rat es; t hese m essages are sent " fire and forget " . Figu r e 5 - 2 . Eve n t s pu lle d fr om a N I D S t o a colle ct or , a llow in g filt e r in g a s w e ll a s r a t e con t r ol

5 .1 .2 . Eve n t Colle ct ion I m pa ct Event collect ion can im pact a syst em in several ways, m ost com m only in t erm s of syst em perform ance and st orage requirem ent s. Due t o separat ion of dut ies, it 's unlikely t hat you'll have responsibilit y for every device used in event collect ion. You m ay have direct cont rol over t he event collect or, allowing you t o configure st orage space t o m eet your needs. The perform ance of t he devices generat ing t he event s, however, is m ore likely under t he cont rol of I T syst em and net work adm inist rat ors. This obviously com plicat es t he process of accessing t he event s, and requires careful planning t o prevent disrupt ion t o your event collect ion. Here are som e im port ant issues t o consider when you're configuring event dat a feeds:

I m pact on t he sending device's CPU Collect only necessary m essages ( m any devices use a t erm such as alert level) . This will m inim ize bot h t he perform ance im pact and t he disk st orage requirem ent s. As we m ent ioned in Chapt er 3 , using st andard t em plat es for each t ype of device will speed t he process for configuring devices.

Event m essage det ail You should analyze m essages generat ed by each event source t o verify t hat t he cont ent s are useful and have t he

appropriat e level of det ail. For exam ple, a login m essage t hat lacks t he usernam e and syst em nam e is not at t ribut able; it lacks sufficient det ail t o allow im m ediat e analysis and response. Such m essages are not wort h collect ing, and t hey t ake up space wit hout providing proport ionat e usefulness.

Event volum e The im pact of event volum e depends on t he syst em 's purpose, rat e of ut ilizat ion, and configured logging level. For exam ple, a firewall t hat is logging deny m essages via syslog can send m essages at a rat e t hat would quickly fill your collect or's disk, decreasing t he ret ent ion t im e for all of your event dat a. Event logging configurat ion plays an im port ant role in syst em perform ance, and can creat e a negat ive im pact on t he sending device's CPU. I n denial- of- service ( DoS) at t acks, for exam ple, a rout er configured wit h an access cont rol ent ry logging denied packet s could have it s perform ance severely degraded. The im pact can cascade t o ot her devices as crit ical rout er processes such as rout ing it self ( BGP peers dropping, et c.) are unable t o funct ion properly due t o t he lack of CPU resources. I t is im port ant t o verify wit h your net work hardware vendor what t he CPU im pact could be and whet her t here are ways t o m it igat e it , such as log rat e lim it ing. Cisco, for exam ple, gives t he following advice when logging ACLs: [ 4 0 ] [ 40]

See ht t p: / / www.cisco.com / web/ about / securit y/ int elligence/ acl- logging.ht m l.

ACL logging can be CPU int ensive and can negat ively affect ot her funct ions of t he net work device. There are t wo prim ary fact ors t hat cont ribut e t o t he CPU load increase from ACL logging: process swit ching of packet s t hat m at ch log- enabled access cont rol ent ries ( ACEs) and t he generat ion and t ransm ission of log m essages. You can m it igat e t his scenario wit h rat e lim it ing and by increasing t he logging int erval: The ip access-list logging interval 10 com m and lim it s log- induced process swit ching t o one packet per 10 m illiseconds, or 100 packet s per second. The logging rate-limit 100 except 4 com m and in t he exam ple lim it s log generat ion and t ransm ission t o 100 m essages per second except for log levels 4 ( warnings) t hrough 0 ( em ergencies) . The configurat ion for rat e lim it ing is quit e sim ple; t he following com m ands im plem ent t hese suggest ions for a Cisco rout er:

logging buffered informational logging rate-limit 100 except 4 ip access-list logging interval 10 Cisco's NI DS solut ion also feat ures a t hrot t ling m echanism t o prevent a DoS at t ack on t he sensor via it s event sum m arizat ion m odes. The sensor will " sum m arize" a large burst of individual event s int o a m et a- event t hat , alt hough not specific ( i.e., it doesn't include a source/ dest inat ion I P) , can prevent an overload from t he sam e event . This t hwart s at t acks t hat at t em pt t o overwhelm t he securit y st aff wit h alert volum e, such as t he " St ick" at t ack described by Coret ez Giov anni.[ 4 1 ] You will st ill get t he source and dest inat ion I Ps in t he first few event s before t he sum m arizat ion st art s. [ 41]

See ht t p: / / packet st orm securit y.nl/ dist ribut ed/ st ick.ht m .

Just as im port ant as t he perform ance im pact on t he sending device is t he need t o obt ain t he appropriat e am ount of det ail in t he collect ed m essages. Too lit t le det ail and t he m essages are not useful; t oo m uch det ail and t he m essages require unnecessary ext ra processing and st orage. For exam ple, a NI DS alert can cont ain dat a ranging from m inim al cont ext ual inform at ion t o full packet capt ure ( oft en called a t rigger packet) . I n som e sit uat ions, you m ay want t o know exact ly what packet ( s) t riggered t he alert . You m ay also require t he packet cont ent s from t he next few packet s aft er t he alert as well. For exam ple, t he following log m essage shows t hat host 10.0.30.209 connect ed t o t he HTTP port on 64.62.193.129. I f t hat is all you need t o know ( t he fact t hat one host connect ed t o t he ot her, sim ilar t o what you'll get wit h Net Flow) , t he m essage is sufficient wit h it s t im est am p and source/ dest inat ion I P inform at ion.

20:29:54.829259 IP 10.0.30.209.1514 > 64.62.193.129.80 However, you m ay want t o know specifically what host 10.0.30.209 has request ed from t he web server. Those det ails need t o com e from t he applicat ion log source, such as t he access_log file from your proxy server:

20:29:54.829259 10.0.30.209 TCP_MISS/200 1393 GET http://

64.62.193.129/J05532/a4/0/0/pcx.js text/x-c DEFAULT_CASE-DefaultGroup -

Furt her inspect ion of t he pcx.j s file shows t hat it cont ained a set of com m ands passed t o m alware running on t he host , inst ruct ing it t o download an updat ed vict im list for DoS at t acks. The last issue, event volum e and rat e, im pact s your abilit y t o ret ain dat a for t he appropriat e lengt h of t im e, as depict ed in Figure 5- 3. As t he event rat e received and st ored increases, t he available disk space correspondingly decreases. Since disk resources are finit e, your event collect or will likely purge t he alert s in a first - in, first - out ( FI FO) fashion. Figu r e 5 - 3 . Re la t ion sh ip be t w e e n e ve n t r a t e , m e ssa ge size , a n d disk u t iliza t ion

For exam ple, if your event collect or can hold 100 m illion alert m essages on 50 GB of disk space and your event rat e is norm ally 10 m essages per second, it would t ake approxim at ely 115 days ( 10 m illion seconds) t o fill your disk ( see Figure 5- 4) . I f your rat e spikes t o 50 m essages per second, t he t im e t o fill your disk will drop t o 23 days. At 100 event s per second, your ret ent ion t im e drops t o 11 days! Figu r e 5 - 4 . Re la t ion sh ip be t w e e n m e ssa ge r a t e a n d r e t e n t ion t im e

This im pact becom es clear when you consider an exam ple such as a Cisco t erm inal server configured for TACACS+ aut hent icat ion on TTY lines. The presence of noise on t he async lines causes EXEC processes t o be spawned and aut hent icat ion at t em pt s t o be sent t o t he Aut hent icat ion, Aut horizat ion, and Account ing ( AAA) server:

07:44:26 07:45:26 07:45:28 07:45:32 07:45:48 07:46:33 07:46:44 07:47:43 07:48:36 07:49:01

aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com aaa.blanco.com

: : : : : : : : : :

[ID [ID [ID [ID [ID [ID [ID [ID [ID [ID

259349 259349 450946 450946 259349 624588 259349 259349 259349 259349

local0.warning] local0.warning] local0.warning] local0.warning] local0.warning] local0.warning] local0.warning] local0.warning] local0.warning] local0.warning]

-

User User User User User User User User User User

*(#$''''%^@ not found *(#$''''%^@ not found !_)*$^ not found !_)*$^ not found *(#$''''%^@ not found !_)*$^ not found *(#$''''%^@ not found *(#$''''%^@ not found *(#$''''%^@ not found *(#$''''%^@ not found

These errant aut hent icat ion m essages are essent ially garbage dat a, filling up your logfiles and wast ing disk space. Such large volum es of m essages can burst int o your collect or, severely degrading your capacit y t o ret ain log m essages. As you can see, cont rolling m essage rat es from your devices is required t o ensure t hat your dat a will be t here when you need it , along wit h t he necessary det ail. One effect ive configurat ion t echnique is t o allot a specific quot a of disk space for each t ype of dat a source, cont rolling t he im pact of any single event source on t he ent ire collect ion syst em . Let 's consider som e specific exam ples of event dat a t ypes, and t he unique collect ion im pact of each.

5 .1 .2 .1 . H ost logs The bulk of host logs includes OS syst em m essages ( such as t hose logged t o / var/ log/ syst em or / var/ log/ m essages, et c.) t hat are writ t en locally as configured by t he syst em adm inist rat or. Host logs are alm ost always push ed t o t he collect or. The event rat e will vary am ong syst em s, depending on usage and logging level. A frequent ly used syst em configured t o log at a very high level ( such as debug logging) could cause t he syslog daem on t o use t oo m any syst em resources. This will im pact not only CPU perform ance, but local disk st orage as well; t he generat ed m essages could fill t he local syst em disk, causing t he syslog daem on or t he syst em it self t o crash. I n Exam ple 5- 1, you can see several m essages about syst em perform ance and applicat ion t ransact ions. These m essages cont ain crit ical det ails t hat you don't want t o m iss. Specifically, you can see user drobbins logging in as root wit h t he sudo com m and. Using t em plat es for host syslog configurat ion, as we will discuss in Chapt er 6 , can ensure t hat t his inform at ion is logged. I n t his exam ple, t hey were used t o specify appropriat e syslog set t ings t o capt ure login m essages for t his server. Ex a m ple 5 - 1 . Cr it ica l m e ssa ge s ca pt u r e d by cor r e ct loggin g con figu r a t ion

Sep 18 03:05:24 blanco-linux installdb[80516]: started (uid 96) Sep 18 03:05:24 blanco-linux installdb[80516]: Opened receipt database on '/' with schema 17. Sep 18 03:05:36 blanco-linux installdb[80516]: done. (0.007u + 0.005s) Sep 18 19:06:02 blanco-linux sshd[84532]: Could not write ident string to UNKNOWN Sep 18 20:59:19 blanco-linux sshd[26778]: Accepted keyboard-interactive/pam for drobbins from 64.102.56.104 port 56475 ssh2 Sep 18 20:59:19 blanco-linux sshd(pam_unix)[26784]: session opened for user drobbins by (uid=0) Sep 19 21:22:40 blanco-linux sudo: drobbins : TTY=pts/1 ; PWD=/home/drobbins ; USER=root ; COMMAND=/bin/su Sep 19 21:22:40 blanco-linux su[28169]: Successful su for root by root Sep 19 21:22:40 blanco-linux su[28169]: + pts/1 root:root Sep 19 21:22:40 blanco-linux su(pam_unix)[28169]: session opened for user root by (uid=0) Sep 18 22:29:13 blanco-linux syslog-ng[4791]: Log statistics; processed='center(queued)=258910', processed='center(received)=129455', processed='destination(messages)=129455', processed='destination(console_all)=129455', processed='source(src)=129455' Sep 18 22:30:07 blanco-linux ReportCrash[85189]: Formulating crash report for process aex-pluginmanager-bin[85188] Sep 18 22:30:07 blanco-linux ReportCrash[85189]: Saved crashreport to /Library/Logs/CrashReporter/aex-pluginmanager-bin_2008-09-18-211606_blancolinux.crash using uid: 0 gid: 0, euid: 0 egid: 0 Sep 18 22:30:11 blanco-linux ReportCrash[85189]: Formulating crash report for process aex-pluginmanager-bin[85190] Sep 18 22:30:11 blanco-linux ReportCrash[85189]: Saved crashreport to /Library/Logs/CrashReporter/aex-pluginmanager-bin_2008-09-18-211610_blancolinux.crash using uid: 0 gid: 0, euid: 0 egid: 0 Sep 18 22:30:15 blanco-linux ReportCrash[85189]: Formulating crash report for process aex-pluginmanager-bin[85191]

5 .1 .2 .2 . N e t w or k I D S Net work I DS ( NI DS) , as we will discuss in Chapt er 6 , holds an advant age over ot her event sources: it does not int roduce ext ra load on t he m onit ored syst em s. Each sensor, however, has t he pot ent ial t o generat e m illions of alert s per day if it is left unt uned. You m ust ensure t hat your NI DS is properly t uned t o prevent t he collect ion and st orage of m assive am ount s of false- posit ive alert s. You can process NI DS alert s via eit her a push or a pull t echnology, depending on it s configurat ion and feat ures. Use

caut ion when rolling out new signat ures on t he sensors, however. A poorly writ t en signat ure t hat m at ches expect ed t raffic in your net work can generat e so m any false- posit ive alert s t hat your ret ent ion will be im pact ed, as we previously discussed. A well- t uned NI DS wit h appropriat e cont ext added from net work variables, as shown in Exam ple 5- 2 and discussed furt her in Chapt er 6 , is one of t he best sources of reliable securit y event dat a. Ex a m ple 5 - 2 . CS- I PS a le r t ge n e r a t e d by a w e b a pplica t ion se r ve r u sin g Bit Tor r e n t

Message eventId=1217489871947867997 eventType=evIdsAlert hostId=blanco-ids-1 appName=sensorApp appInstanceId=493 tmTime=1221781927161 severity=2 Interface=ge3_3 Protocol=tcp riskRatingValue=55 sigId=11020 subSigId=1 sigDetails="BitTorrent protocol" src=68.118.195.234 srcDir=OUT srcport=6881 dst=10.10.1.68 dstDir=WEBAPP_SERVERS dstport=1797 cid:threatRatingValue="55" marsCategory="Info/UncommonTraffic/P2PFileShare/FileTransfer" type="windows-nt-2kxp" targetValueRating="medium" attackRelevanceRating="relevant" relevance="relevant" idSource="learned"

5 .1 .2 .3 . N e t Flow Net Flow is collect ed via push , and represent s a crit ical source of incident dat a wit h a variet y of uses for securit y. For inst ance, you can collect Net Flow in raw form at t o be st ored on disk, t hen relay it t o syst em s t hat int erpret t rends or ident ify anom alies. Because Net Flow is based on net work t raffic pat t erns, spikes in t raffic, such as t hose caused by a DoS at t ack, can negat ively im pact dat a ret ent ion. However, you can configure Net Flow collect ion wit h OSU flow- t ools t o drop ( not st ore) cert ain t raffic based on source/ dest inat ion address, source/ dest inat ion port s, I P net work, AS num ber, t ype of service, prefix lengt h, TCP flag, and ot her at t ribut es. To drop t raffic based on t hese crit eria, you can define your set t ings wit h t he capt ure filt er com m and. For m ore inform at ion, reference docum ent at ion via t he man flow-capture com m and. You should collect Net Flow at crit ical choke point s, such as dist ribut ion gat eways in a dat a cent er and perim et er devices in your DMZ. You will find t hese choke point s at your net work ingress/ egress dem arcat ions; t hey are oft en t he division point of securit y policies. I n our experience wit h OSU flow- t ools, we've found t hat 1.2 gigabit s per second of bandwidt h running at 200,000 packet s per second ( pps) can be st ored in 600 GB of st orage, allowing approxim at ely 90 days of hist ory. Figure 5- 5 depict s your visibilit y int o a net work wit hout Net Flow. Wit hout Net Flow, you are essent ially blind t o anyt hing happening on your net work. Did som eone download t he payroll dat abase? Who uploaded a t roj an file t o t he web server? How m any t im es was t he accident ally exposed dat a downloaded? The answer t o all of t hese quest ions is " I don't know" wit hout net work t elem et ry such as Net Flow. Figu r e 5 - 5 . A n e t w or k w it h ou t t e le m e t r y ( ph ot o cou r t e sy of Je n n ife r Bu de n z for Ph ot oj e n ics)

5 .1 .2 .4 . Applica t ion logs Applicat ion logs can be ext rem ely useful t o collect , as t hey provide in- dept h dat a about applicat ion act ivit y bet ween users and syst em s. Because applicat ion log form at s vary so widely, ident ifying specific incident s and ruling out false posit ives can be challenging, even wit h t he m ore popular applicat ions such as Apache Web Server and t he Squid proxy, wit h t heir large and well- est ablished user base. As you can see in Exam ple 5- 3, proxy server logs include varying levels of det ail regarding HTTP t ransact ions. The volum e and det ail of m essages in a proxy server t ypically require t hird- part y t ools t o analyze t he logs, such as AWSt at s,[ 4 2 ] Sawm ill, [ 4 3 ] or Splunk. [ 4 4 ] [ 42]

ht t p: / / awst at s.sourceforge.net /

[ 43]

ht t p: / / w w w .saw m ill.net /

[ 44]

ht t p: / / www.splunk.com /

Ex a m ple 5 - 3 . Pr ox y se r ve r logs

1219859423.875 56 10.0.30.209 TCP_MISS/200 1085 GET http://rad.msn.com/ADSAdClient31.dll?GetAd=&PG=IMUSVT - DIRECT/rad.msn.com text/html ALLOW_WBRS-DefaultGroup 1219859448.869 42 10.0.30.209 TCP_MISS/200 7106 GET http://edge1.catalog.video.msn.com/videoByTag.aspx?mk=us&ns=gallery&tag=im_f_35 -49&responseencoding=rss&p=im_f_35-49 - DIRECT/edge1.catalog.video.msn.com text/xml ALLOW_WBRS-DefaultGroup 1219859840.488 16 10.0.30.209 TCP_MISS/200 384 HEAD http://download.windowsupdate.com/v7/windowsupdate/redir/wuredir.cab?0808271800 DIRECT/download.windowsupdate.com application/octet-stream DEFAULT_CASEDefaultGroup 1219859846.256 105 10.0.30.209 TCP_MISS/200 368 HEAD http://www.update.microsoft.com/v7/microsoftupdate/redir/muauth.cab?0808271800 DIRECT/www.update.microsoft.com application/octet-stream ALLOW_WBRS-DefaultGroup 1219859857.528 20 10.0.30.209 TCP_MISS/200 368 HEAD http://download.windowsupdate.com/v7/microsoftupdate/redir/muredir.cab?0808271800 -

DIRECT/download.windowsupdate.com application/octet-stream DEFAULT_CASEDefaultGroup 1219860033.536 53 10.0.30.209 TCP_MISS/200 1085 GET http://rad.msn.com/ADSAdClient31.dll?GetAd=&PG=IMUSVT - DIRECT/rad.msn.com text/html ALLOW_WBRS-DefaultGroup 1219860058.807 33 10.0.30.209 TCP_MISS/200 7029 GET http://edge1.catalog.video.msn.com/videoByTag.aspx?mk=us&ns=gallery&tag=im_default& responseencoding=rss&p=im_default - DIRECT/edge1.catalog.video.msn.com text/xml ALLOW_WBRS-DefaultGroup 1219860068.892 62 10.0.30.209 TCP_CLIENT_REFRESH_MISS/200 364 GET http://watson.microsoft.com/StageOne/Generic/mptelemetry/80072efe/endsearch/sea rch/1_1_1593_0/mpsigdwn_dll/1_1_1593_0/windows%20defender.htm?LCID=1033&OS=5.1.2600 .2.00010100.2.0&DWVer=11.0.8163.0 - DIRECT/watson.microsoft.co m text/html ALLOW_WBRS-DefaultGroup 1219860592.185 463 10.0.30.209 TCP_CLIENT_REFRESH_MISS/302 752 PROPFIND http://services.msn.com/svcs/hotmail/httpmail.asp - DIRECT/services.msn.c om text/html ALLOW_WBRS-DefaultGroup -

Because applicat ion logs can be com plex and capacious, you should only collect t hem from t he m ost crit ical syst em s, including t hose under regulat ory com pliance requirem ent s. I f you're not in I T, you'll need t o m aint ain a st rong relat ionship wit h I T applicat ion owners, as you will need t heir cooperat ion and input t o int erpret t he logs properly. 5 .1 .2 .5 . D a t a ba se logs As we will discuss furt her in Chapt er 6 , dat abase logs are one of t he m ost challenging event sources t o collect . The m ost det ailed, useful m essages require corresponding audit ing configurat ion in t he dat abase. I n Oracle, for exam ple, audit ing feat ures are com plex and require careful configurat ion t o produce useful result s t hat will not harm dat abase perform ance. Dat abase m essages require aut om at ed parsing for efficient collect ion and st orage, as shown in Exam ple 5- 4. Since t hese syst em s are likely t o house your m ost precious int ellect ual propert y and crit ical business dat a, good dat abase logging can becom e one of t he m ost powerful pieces of securit y dat a. Ex a m ple 5 - 4 . Or a cle da t a ba se a u dit log

USERNAME --------------SYS VINAY SYS SYS VINAY

TERMIN -----pts/3 pts/1 pts/3 pts/2 pts/1

ACTION_N -------LOGOFF LOGON LOGOFF LOGOFF LOGON

TIMESTAMP --------------02152013:221056 02152013:221651 02152013:221659 02152013:222622 02152013:223022

LOGOFF_TIME RETURNCODE --------------- ---------02152013:221651 0 1017 02152013:223022 0 02152013:223508 0 1017

5 .1 .2 .6 . N e t w or k ACL logs ACL logs from net work firewall devices can provide visibilit y int o t raffic t hat is dropped ( deny logs) or passed ( perm it logs) , based on t he codified rules of t he ACL. You m ust be ext rem ely caut ious when collect ing ACL logs from net work devices because t he CPU ut ilizat ion oft en im pact s ot her device funct ions. This im pact will vary am ong hardware plat form s and operat ing syst em s, but devices t hat act as dedicat ed securit y hardware, such as com m ercial firewalls, are usually designed t o handle such workload. Cisco I OS, t he operat ing syst em used on m ost Cisco rout ers, allows you t o define updat e t hresholds and rat e lim it ing t o m it igat e t he im pact of ACL logging. Due t o t he pot ent ial im pact , however, be prepared wit h appropriat e j ust ificat ion before enabling ACL logging on any devices not specifically dedicat ed t o firewalling t raffic. Net work ACL logs can also be an invaluable source of t roubleshoot ing inform at ion. For inst ance, in Exam ple 5- 5, you can see why a part ner's Win32 com put ers are not get t ing updat es and pat ches from Microsoft ; t he t raffic is being blocked. Ex a m ple 5 - 5 . Cisco r ou t e r ACL logs

Sep 23 18:32:39.029 IST: %SEC-6-IPACCESSLOGP: list 10.56.68.104(3288) -> 65.55.27.221(443), 1 packet Sep 23 18:32:45.041 IST: %SEC-6-IPACCESSLOGP: list 10.56.68.104(3288) -> 65.55.27.221(443), 1 packet Sep 23 18:33:07.173 IST: %SEC-6-IPACCESSLOGP: list 10.56.78.215(4108) -> 65.55.27.221(443), 1 packet Sep 23 18:33:10.393 IST: %SEC-6-IPACCESSLOGP: list 10.56.78.215(4108) -> 65.55.27.221(443), 1 packet

BLANCO-XNET denied tcp BLANCO-XNET denied tcp BLANCO-XNET denied tcp BLANCO-XNET denied tcp

Now t hat you have an idea of how event s are collect ed, which event t ypes are available, and what pot ent ial perform ance im pact exist s, let 's look at how Blanco has chosen it s event sources.

5 .2 . Ch oosin g Eve n t Sou r ce s for Bla n co W ir e le ss Figure 5- 6 shows t he configurat ion of event source collect ion for Blanco's securit y m onit oring. Not ice t hat Blanco has configured Net Flow, NI DS, syslog, applicat ion logs, and dat abase audit logs t o det ect policy violat ions affect ing it s select ed m onit oring t arget s. Figu r e 5 - 6 . Bla n co's se le ct e d se cu r it y e ve n t sou r ce s

To m onit or t he t arget s select ed in Chapt er 4 , Blanco has enabled Net Flow export from it s dat a cent er rout ers, st oring t he flow dat a in a Net Flow collect or syst em . Blanco's securit y t eam will use t his dat a t o ident ify connect ions sourced from crit ical syst em s t oward ext ernal syst em s, and t o ident ify large volum e copies from dat abase syst em s. Blanco's Unix servers are configured t o send m essages t o a syslog collect ion server. The securit y t eam will m onit or t hese m essages for direct privileged logins and privilege escalat ion via sudo . The servers are also configured t o log and relay m essages recording st ops and st art s for t he web server, SSH, and dat abase processes. Blanco's NI DS is configured t o m onit or for known at t acks against t he Oracle suit e. The securit y t eam has creat ed cust om signat ures t o ident ify:

Unencrypt ed t ransm ission of Social Securit y num bers ( a policy violat ion)

The describe com m and issued against any product ion dat abase ( an indicat ion of dat abase enum erat ion by som eone unfam iliar wit h t he schem a) Blanco's dat abase adm inist rat ors have configured t he Oracle dat abase servers t o audit for:

Queries against t he SSN colum ns in t he cust om er account t able Queries against V$ ( anot her m et hod for enum erat ing syst em users, processes, and connect ed applicat ions) Direct privileged logins ( system or sys)

5 .3 . Con clu sion Not every log m essage or event source proves useful for securit y m onit oring, and even good event sources can overt ax t he devices you're m onit oring. The work we've done t o carefully choose event sources is our last select ion st ep t o configure policy- based m onit oring. A clear underst anding of how you int end t o use and collect event sources will priorit ize how you det erm ine proper configurat ion of logging levels, select collect ion hardware, and set dat a ret ent ion policies. I n Chapt er 6 , we will focus on st rat egies and m et hods t o collect t hese event sources, filt ering t hem int o act ionable securit y event s.

Ch a pt e r 6 . Fe e d a n d Tu n e You awaken t o find yourself adrift on a raft in t he m iddle of t he At lant ic Ocean. The sun is blazing and you are incredibly t hirst y. You look around you and see t hat you are surrounded by cool wat er, but it is salt wat er, not t he freshwat er you so desperat ely need. The abundance of t he wrong kind of wat er is akin t o t he deluge of useless m essages experienced from unt uned securit y alert sources such as NI DS, syslog, and applicat ion logs. I nst ead of useful, act ionable securit y alert s, t he lifeblood of incident response, you get a m out hful of salt wat er in t he form of 2 m illion NI DS alert s a day. An unt uned securit y event source will generat e alert s irrelevant t o your policies, quickly overwhelm your securit y m onit oring st aff, and reduce t he availabilit y of useful dat a in your collect ion syst em s. A properly t uned dat a source is core t o your successful securit y m onit oring, and in t his chapt er, we'll show you how t o accom plish t hat . We've defined our policies, docum ent ed knowledge of our net work, and select ed t arget s wit h event sources. Now we m ust convert t his m et adat a int o act ionable incident s by m ast ering det ect ion t echnology. We'll explain t his cent ral concept by first int roducing a net work int rusion det ect ion fram ework. This fram ework will guide our deploym ent and t uning, building on t he dat a we've gat hered in previous chapt ers. We will follow t hat fram ework by showing how t o use cust om Net Flow queries wit h aut om at ed report ing t o cat ch violat ion of securit y policies. We'll t hen explore t he vast t opic of syslog collect ion, t rying t o pull som e useful inform at ion from what can be an overwhelm ing st ream of dat a. Finally, we'll close t his chapt er by covering som e challenges t hat are com m only encount ered when t rying t o deploy t hese t echnologies in various environm ent s.

6 .1 . N e t w or k I n t r u sion D e t e ct ion Syst e m s Net work int rusion det ect ion is t he subj ect of dozens of books, cont ent ious debat es wit hin t he securit y indust ry as t o whet her it is a dead t echnology, and num erous m ispercept ions. To define it , int rusion det ect ion is a deep packet inspect ion t echnology t hat analyzes I nt ernet Prot ocol ( I P) packet s, alert ing when t raffic m at ches m alicious pat t erns. A net work int rusion det ect ion syst em ( NI DS) , oft en referred t o as a sensor, is a dedicat ed soft ware or hardware t echnology t hat perform s t he funct ion of int rusion det ect ion. I n our t elephone analogy, a NI DS is aim ed at list ening for key phrases wit hin t he call it self, rat her t han report ing t he call st at ist ics in a phone bill, as Net Flow does. Many NI DS solut ions are available, including t he open source Snort , Bro, and OSSEC, and com m ercial hardware and soft ware solut ions from Cisco, TippingPoint , I SS, Sourcefire ( t he com m ercial version of Snort ) , and McAfee. This chapt er will explain how t o get t he m ost out of deploying a NI DS via a com m on design approach and a m et hodical t uning st rat egy based on net work m et adat a. First , let 's cover som e NI DS basics, and t hen we'll drill deeper t o explain t uning m et hodologies.

6 .1 .1 . Pa ck e t An a lysis a n d Ale r t in g I n general, a NI DS device will perform som e predefined act ion when cert ain crit eria are m et while inspect ing I P packet s. This m ay include everyt hing from sending an alert t o deploying an access cont rol list ( ACL) t hat blocks connect ivit y for t he offending I P address. A NI DS prim arily ident ifies at t acks via pat t ern m at ching. This pat t ern m at ching can com prise eit her a single packet t hat m at ches a regular expression, such as ([Jj][Oo][Ii][Nn]), or m ore com plex pat t erns requiring a sequence of m at ches in a part icular order, as depict ed in Figure 6- 1. This is a great ly sim plified view of how signat ures work; dozens of books on int rusion det ect ion are available t hat go int o m uch m ore dept h. For our purposes, it 's im port ant t o rem em ber t hat a NI DS is j ust a t ool in your collect ion, not t he be- all and end- all securit y m echanism , despit e what NI DS vendors t ell you. Figu r e 6 - 1 . Pa t t e r n m a t ch in g m e t h odology u se d by a N I D S

More recent addit ions t o NI DS capabilit ies include anom aly det ect ion and OS fingerprint ing. Anom aly det ect ion works by analyzing " descript ors" of t he dat a rat her t han t he dat a it self, m uch like Net Flow. A baseline of " norm al" is est ablished and anyt hing falling out side t his baseline is t agged and alert ed as an anom aly. OS fingerprint ing works by analyzing specific charact erist ics of a given operat ing syst em 's I P st ack such t hat an OS can be reliably ident ified ( sim ilar t o an " - O" swit ch wit h t he fam iliar Nm ap net work scanning ut ilit y) . Cisco's CS- I PS 6.x soft ware can t ake advant age of t his feat ure and adj ust t he confidence level or severit y of t he alert based on whet her t he at t ack applies t o t he vict im OS, such as a Linux at t ack dest ined for a Windows server. A NI DS can also add furt her cont ext ual inform at ion int o it s alert s, aut om at ically changing confidence levels, which increases t he fidelit y of t he alert s.

6 .1 .2 . N e t w or k I n t r u sion Pr e ve n t ion Syst e m s

A net work int rusion prevent ion syst em ( NI PS) is a NI DS deployed in t he pat h of net work t raffic ( " inline" ) , rat her t han passively inspect ing it . Figure 6- 2 shows a logical view of t he t wo alt ernat ives. Figu r e 6 - 2 . Com pa r ison of t h e n e t w or k de ploym e n t sce n a r ios for N I PS ve r su s N I D S

Deploying t his t echnology inline allows t he NI PS t o drop m alicious t raffic, hence prevent ing t he at t ack and prot ect ing end host s. Many det ails m ust be considered before choosing bet ween a NI DS and a NI PS. The following sect ion provides analysis crit eria t o guide your decision for any given environm ent .

6 .1 .3 . I n t r u sion D e t e ct ion or I n t r u sion Pr e ve n t ion ? Securit y vendors com m only urge int rusion prevent ion solut ions. To decide whet her t o deploy such a solut ion, you m ust consider m any fact ors, including your goals for t he solut ion, your net work t opology, and your availabilit y requirem ent s. Two core crit eria dom inat e your crit eria for deciding on a NI PS versus a NI DS: availabilit y requirem ent s and span of cont rol. Let 's assum e you have decided t o deploy your solut ion inline, but your I T net working t eam want s t o know m ore about any pot ent ial im pact s t his deploym ent would have on availabilit y. To answer t his quest ion, you will need t o underst and your hardware and soft ware failure scenarios. I n addit ion, you'll need t o underst and t hose scenarios in your specific environm ent . This m ay sound like a daunt ing t ask, but it is a wort hwhile endeavor and m uch of t he work is already done for you. You can apply t hese basic concept s t o j ust about any t echnology you want t o put " in t he line" of net work t raffic. First , your vendor will need t o answer t he following quest ions, shown here in order of im port ance: 1. 2.

What is your hardware's MTBF? Does your solut ion have hardware- based " fail- open" capabilit ies?

3.

a . How long does fail- open t ake? Does your solut ion have soft ware- based " fail- open" capabilit ies? a . How long does fail- open t ake? b. I s it adm in- configurable?

These variables provide som e obj ect ive basis for your analysis. Now, let 's veer off t he beat en pat h a bit and play wit h som e fun m at hem at ics in t he realm of availabilit y t heory. The m et hod for calculat ing t he availabilit y of a given com ponent or series of com ponent s is expressed in Mean Tim e Bet ween Failures ( MTBF) and Mean Tim e To Repair ( MTTR) . The m ost com m on way t o express t his availabilit y calculat ion is by percent age. I t is com m only referenced by net work professionals using t he t erm " 9s" , illust rat ed in Table 6- 1 . To calculat e annual availabilit y, you can m ult iply your t arget ed 9s value by 525,600 m inut es per year. The second colum n of Table 6- 1 illust rat es t he downt im e t hat result s. Ta ble 6 - 1 . D ow n t im e cor r e spon din g t o va r iou s le ve l of " 9 s" a va ila bilit y Ava ila bilit y An n u a l dow n t im e 90% ( one 9)

36.5 days

99% ( t wo 9s)

3.65 days

99.9% ( t hree 9s)

8.76 hours

99.99% ( four 9s)

52 m inut es

99.999% ( five 9s) 5 m inut es 99.9999% ( six 9s) 31 seconds

Your availabilit y requirem ent s m ust be m et by your solut ion; if downt im e is very cost ly t o your business environm ent , your NI PS solut ion m ust lim it downt im e. Figure 6- 3 shows com m on availabilit y requirem ent s. Upt im e requirem ent s are oft en lowest in lab or t est ing environm ent s, but very high in product ion environm ent s such as dat a cent ers. Figu r e 6 - 3 . Ava ila bilit y's im pa ct on N I D S ve r su s N I PS de cision

6 .1 .3 .1 . Ava ila bilit y Availabilit y is a t im e value calculat ed in t erm s of MTBF and MTTR. The MTBF num ber is provided by t he m anufact urer and is expressed in hours. The MTTR num ber, variable and dependent upon t he specific net work, is also expressed in hours. The sim ple equat ion is as follows:

A device wit h an MTBF of 175,000 hours and an MTTR of 30 m inut es has a calculat ed annual availabilit y of 525,598 m inut es, which equals 1.52 m inut es of downt im e. Appendix C provides m ore det ail and com plex exam ples regarding t hese calculat ions. The real challenge in com put ing t he availabilit y of any syst em or net work is in underst anding t he MTTR. This value is unique t o your environm ent , affect ed by variables such as rout ing prot ocol convergence t im e and/ or spanning t ree convergence t im e, depending on your t opology. You need t o underst and exact ly what happens when an inline device fails t o forward t raffic properly. This can be due t o power, hardware, or ot her environm ent al failure. I deally, you would build a full t est environm ent wit h your int ended NI PS design and t hen t est t he different scenarios. You should be able t o est im at e t he im pact of failure by analyzing a few key variables:

I nt erface fail- open How long does it t ake for a set of inline int erfaces t o begin passing t raffic aft er a failure? Will t raffic be queued or dropped?

Layer 3 ( L3) environm ent failures When your NI PS is deployed on a rout ed net work segm ent , how long does your rout ing prot ocol t ake t o converge?

Layer 2 ( L2) environm ent failures When your NI PS is deployed on a physically redundant L2 net work segm ent , how quickly will t he spanning t ree converge? 6 .1 .3 .1 .1 . N on h a r dw a r e sou r ce s of dow n t im e We're doing t hese calculat ions based purely on hardware failure calculat ions, so you m ust also consider soft ware failure, environm ent al considerat ions, and hum an error. Com pared t o ot her net work devices, a NI PS has a m uch higher num ber of soft ware updat es ( signat ures, t ypically) . Applicat ion of t hese soft ware updat es always int roduces t he possibilit y of soft ware failure or hum an error t hat could result in an unexpect ed out age. 6 .1 .3 .1 .2 . N I PS a n d n e t w or k ba n dw idt h A coworker who designs net works for Cisco cust om ers sum s up t his issue wit h t he following st at em ent : " Packet inspect ion capacit y always t rails L2/ L3 forwarding capacit y." This rule applies t o all net work devices t hat add processing beyond sim ple connect ion st at e, such as firewalls and NI PSs. NI PS vendors t ypically support physical int erface m edia, but not line- rat e inspect ion for all int erfaces in t he NI PS. For exam ple, several vendors were quick t o include support for I EEE 802.3ae 10 Gbps int erfaces, but t heir syst em s clearly could not inspect 10 Gbps. Wit h t his in m ind, you m ust keep t he aggregat e t raffic you are t rying t o push t hrough an inline syst em below t he vendor's rat ed capacit y. Keep in m ind t hat t raffic can be " burst y," such as when backups or dat abase copies t raverse t he net work. Such burst s can cause your aggregat e t raffic t o exceed t he NI PS's inspect ion and forwarding capabilit y, which m ay in t urn cause it t o drop t raffic and creat e vague " net work problem s."

H igh - Ava ila bilit y N I PS Usin g Ph ysica l Re du n da n cy We don't int end t o im ply t hat you cannot deploy a highly available inline NI PS. On t he cont rary, vendors such as I SS and Cisco illust rat e several designs t o support high availabilit y. Most of t hese designs rely on L3 or L2 net work segm ent redundancy wit h t he forwarding prot ocol's convergence t im e driving MTTR. This adds addit ional com plexit y, which is oft en m ore t han net work operat ors are willing t o accept . 6 .1 .3 .2 . Spa n of con t r ol Availabilit y m ay not be t he largest driver for all of your environm ent s. Let 's consider anot her dat a point : your

organizat ion's span of cont rol over t he net work- connect ed endpoint s. For exam ple, you m ay have a part ner or vendor on- sit e t hat support s it s own servers and deskt ops. Because t hese syst em s are not owned or support ed by your I T st aff, you will have lim it ed knowledge about t he securit y policies governing t hem . On Windows syst em s: do users run as Adm inist rat or? Has ant ivirus soft ware been inst alled? Are t he syst em s prom pt ly pat ched? Since t hese syst em s don't belong t o you, your span of cont rol is lim it ed. Anot her exam ple of a lim it ed span of cont rol scenario is a developm ent lab where t he sam e best pract ices and cont rols t hat apply in your product ion I T- support ed environm ent s cannot be im plem ent ed. Figure 6- 4 represent s t his concept graphically. Figu r e 6 - 4 . Spa n of con t r ol a n d r e la t ion sh ip t o N I D S ve r su s N I PS

6 .2 . N I D S D e ploym e n t Fr a m e w or k Deploying a NI DS can be som ewhat daunt ing, but if you begin wit h a com m on fram ework t hat applies t o your environm ent s, it becom es m anageable and, dare we say, alm ost easy. This fram ework st art s by defining a finit e set of designs t hat should address your different net work environm ent s. For sim plicit y and brevit y, we'll look at t he DMZ, t he dat a cent er, and t he ext ranet . You can m odify t hese designs t o suit ot her environm ent s as well. The key is t o t ry t o apply t he fram ework t o t he environm ent based on your knowledge of it s funct ion and t opology, as we described in Chapt er 3 . I m plem ent t he fram ework via t he following st eps:

Analyze Size your solut ion and select com ponent s based on t he t raffic requirem ent s, funct ion, and user base for t he t arget environm ent .

Design Choose from your list of designs and m odify for any differences in net work t opology or funct ion.

Deploy Select and properly deploy hardware according t o t he design, m aking sure t o accom m odat e any unique requirem ent s.

Tune and m anage Adj ust sensor configurat ion set t ings t o elim inat e false posit ives, build net work intelligence int o your alert s, deploy new at t ack signat ures, and creat e cust om signat ures.

6 .2 .1 . An a lyze You m ust consider several fact ors when t aking on t he t ask of analyzing any given environm ent . As you will see in t he exam ples t hat follow, each fact or will have varying levels of im pact , wit h aggregat e bandwidt h and net work t opology carrying t he m ost weight .

Aggregat e bandwidt h To assess aggregat e bandwidt h, sum t he ut ilizat ion rat es on your net works of int erest . I t 's best t o do t his wit h a t ool t hat can record ut ilizat ion over t im e via graphs, such as t he Mult i Rout er Traffic Grapher ( MRTG) discussed in Chapt er 3 . These graphs allow you t o com pare hist ory, t o help you spot t he m axim um t raffic rat es.

Net work t raffic m ix Next , you need t o underst and t he net work t raffic m ix. By net work t raffic m ix , we're t alking about whet her t his is an I nt ernet DMZ net work, a rem ot e access net work, or a dat a cent er net work, as t his will likely define t he securit y policies expect ed for t he net work. I n addit ion, you need t o know what kind of t raffic t raverses t he net work. What charact erizes t he t raffic? I s it HTTP or HTTPS? I s it I PSEC virt ual privat e net work ( VPN) t raffic? The answers t o t hese quest ions will im pact t he locat ion of t he sensor in t he net work, part icularly when encrypt ed t raffic is present , as it is not possible t o inspect t he encrypt ed packet cont ent wit h a NI DS. ( You'll need t o posit ion it where t he t raffic is plain t ext , if t hat 's possible.)

Net work t opology Analysis of net work t opology requires good docum ent at ion, so it will likely require int eract ion wit h t hose who designed and support t he environm ent . I m port ant fact ors t o consider include pat h redundancy and rout ing behavior. You need t o underst and whet her t raffic always t raverses a preferred pat h or whet her t he t raffic is load- balanced across m ult iple links. This inform at ion is necessary t o help you avoid asym m et ry in your t raffic

view, as illust rat ed in t he sidebar, " The Trouble wit h Traffic Asym m et ry." The net work t opology diagram in Figure 6- 5 will serve as a m odel for som e basic exam ples.

Th e Tr ou ble w it h Tr a ffic Asym m e t r y When redundancy is designed int o a rout ed net work, t raffic is oft en load- balanced across m ult iple links. Packet s from a single conversat ion can be rout ed across m ore t han one link. The NI DS's abilit y t o m at ch st ored pat t erns is based on t he assum pt ion t hat it has inspect ed every packet wit hin a conversat ion. When t he NI DS sees only part of a conversat ion ( because one of t he redundant links is not being m onit ored) , it cannot m at ch t he pat t erns cont ained wit hin t he rule set . To prevent such problem s, you m ust design your solut ion so t hat t he NI DS receives a copy of t he ent ire conversat ion. Figu r e 6 - 5 . Ex a m ple n e t w or k t opology

I n Figure 6- 5, you can see t hat gat eways GW1 and GW2, which rout e bet ween net works A and B, has seven links connect ing t he t wo net works. We've m easured t he bandwidt h ut ilized per link over a period of one m ont h, and represent ed it in Table 6- 2 . Ta ble 6 - 2 . Ba n dw idt h a n a lysis for t h e lin k s in Figu r e 6 - 5 Lin k Spe e d Average Maxim um 1

1 Gbps

280 Mbps 500 Mbps

2

1 Gbps

250 Mbps 440 Mbps

3

1 Gbps

100 Mbps 130 Mbps

4

1 Gbps

100 Mbps 130 Mbps

Lin k Spe e d

Average Maxim um

5

1 Gbps

10 Kbps

10 Kbps

6

100 Mbps 70 Mbps

90 Mbps

7

100 Mbps 15 Mbps

15 Mbps

To inspect all t raffic t raversing bot h GW1 and GW2 from Net work A, you can find t he required bandwidt h by calculat ing t he sum of t he average bandwidt h used on links 1, 2, 3, and 4, which equals 730 Mbps: 280 Mbps + 250 Mbps + 100 Mbps + 100 Mbps = 730 Mbps The NI DS solut ion would pot ent ially need t o handle m axim um dat a rat es above 1 Gbps t o handle increasing ut ilizat ion over t im e or spikes in t raffic. I f we reorient t he goal for t his NI DS deploym ent t o wat ch all t raffic from Net work B t hrough t he sam e gat eways, we can calculat e t he required bandwidt h via t he sum of t he average ut ilizat ion for links 6 and 7, j ust over 100 Mbps. Key t o bot h scenarios is t he successful capt ure of bidirect ional t raffic on all pert inent int erfaces. Failure t o do so will lead t o asym m et ric t raffic t hat will confuse t he sensor, as only part of a host - t o- host conversat ion will be seen, which will ult im at ely decrease t he likelihood of t he sensor alert ing on an at t ack.

6 .2 .2 . D e sign The level of st andardizat ion in your various net works will have an enorm ous im pact on how few or how m any net work designs you will need. I ndeed, t he sim plified deploym ent of a service int o a hom ogeneous net work environm ent is one of t he m any underest im at ed ret urns on invest m ent of st andardizat ion. Based on our exam ple analysis of t he environm ent in Figure 6- 5, we know we will need an inspect ion capabilit y of approxim at ely 1 Gbps. Three prim ary fact ors will influence t he specific hardware design:

Physical locat ion I n Figure 6- 5, if gat eways GW1 and GW2 are in t he sam e physical locat ion ( such as a single rack) , we probably won't have t o deal wit h com plicat ions such as t he availabilit y of int er- building fiber cabling. I f inst ead t he gat eways are geographically separat ed, we m ay need t o acquire int er- building cabling, or use rem ot e t raffic copying feat ures, such as Cisco's RSPAN. [ 4 5 ] [ 45]

ht t p: / / www.cisco.com / en/ US/ docs/ swit ches/ lan/ cat alyst 6500/ cat os/ 5.x/ configurat ion/ guide/ span.ht m l

I nt erface opt ions I n Figure 6- 5, we need t o capt ure t he t raffic on gat eways GW1 and GW2 connect ing Net work A t o Net work B. I f we m ap int erfaces 1: 1 ( for sim plicit y, we'll assum e receive ( Rx) and t ransm it ( Tx) t ot als are less t han 1 Gbps) , we will require a sensor wit h four 1 Gbps int erfaces.

Net work device capabilit ies I n Figure 6- 5, gat eways GW1 and GW2 m ust support t he abilit y t o copy t raffic t o an int erface. I f t he gat eways do not support t his feat ure, you can use a net work t ap t o capt ure t he t raffic. ( Not e t hat a t ap is oft en a single point of failure and can negat ively im pact your availabilit y.) Alt ernat ively, you could capt ure t raffic at t he devices in Net work A.

A N ot e on Ta ps Despit e what you m ight hear from vendor sales st aff, a t ap is a piece of hardware placed direct ly in t he physical m edia pat h. You m ust consider t he MTBF and MTTR of t he t ap hardware when det erm ining t he availabilit y of your solut ion. 6 .2 .2 .1 . D M Z de sign I n Figure 6- 6, 1 Gbps int erfaces connect I SP gat eways ( I SP GW1 and I SP GW2) t o DMZ gat eways ( DMZ GW1 and DMZ GW2) . I f our goal is t o capt ure all I nt ernet t raffic t raversing t he DMZ, we m ust capt ure t he int erfaces

connect ing t o links 1, 2, 3, and 4. I f our I SP links t o t he I nt ernet are 155 Mbps ( I SP GW1) and 200 Mbps ( I SP GW2) , our inspect ion capacit y m ust be at least 355 Mbps. A NI DS sensor wit h four 1 Gbps int erfaces should accom m odat e 355 Mbps j ust fine. I f inst ead our goal was t o inspect t raffic t raversing t he DMZ from t he corporat e net work, we only need t o m irror t raffic for links 6 and 7 t o t he sensor. I n eit her case, we m ust have bidirect ional t raffic ( bot h Rx/ Tx) for successful inspect ion. Figu r e 6 - 6 . Pla ce m e n t of N I D S t o in spe ct a D M Z w it h r e du n da n t I SP con n e ct ion s

This is a fairly sim ple exam ple and assum es t hat we have no e- com m erce or ot her DMZ net works. Let 's consider a m ore com plex set up. Figure 6- 7 depict s a DMZ environm ent wit h an e- com m erce net work sim ilar t o what you'd find in m any organizat ions. This could also represent public- facing services deployed in a DMZ. Assum ing t hat you want t o inspect all ingress/ egress t raffic for t he e- com m erce and corporat e net works, you would need t o copy t raffic from links 6 and 7 t o one sensor, and copy t raffic from links 8 and 9 t o a second sensor. Rem em ber again t o collect bidirect ional t raffic on each link. Anot her approach ( in which you could use a single sensor) t o ensure t hat we have a full view of t raffic is t o capt ure Rx- only on all int erfaces of DMZ gat eways GW1 and GW2 ( t hat is, Rx for int erfaces 1 t hrough 9) . This Rx- only m et hod will work well in com plex environm ent s where t here are a large num ber of net work links t o inspect . Figu r e 6 - 7 . Con figu r a t ion of N I D S t o a ccom m oda t e a m or e com ple x D M Z n e t w or k

6 .2 .2 .2 . D a t a ce n t e r de sign Figure 6- 8 depict s a dat a cent er net work connect ed t o t he corporat e backbone by 10 Gbps uplinks via redundant dat a cent er gat eways. Wit hin t he dat a cent er, each server net work has 10 Gbps uplinks t o t he dat a cent er gat eways. This high- bandwidt h environm ent pushes an aggregat e of nearly 1 Tbps ( t erabit s per second) wit hin t he access layer ( int ra- segm ent t raffic) . I nspect ing j ust t he uplinks t o t he dat a cent er gat eways could require 80 Gbps of inspect ion capacit y, which is well beyond t he rat ed capacit y of any vendor's NI DS hardware t oday. You can address t his com m on net work scenario by focusing your solut ion on t he dat a cent er gat eways, leveraging t heir placem ent as a net work choke point and only inspect ing t he ingress and egress t raffic t o t he dat a cent er environm ent . Even wit h t his choke point - focused approach, speeds can pot ent ially reach t he 40 Gbps level. Such high- bandwidt h requirem ent s are norm ally m et by load- balancing t raffic across a farm of NI DS sensors. Several vendors—am ong t hem , Top Layer Net works—offer load- balancing solut ions. Again, t he m ost im port ant point here is t o be sure t o get bidirect ional t raffic and ensure t hat t he load- balancing algorit hm used is det erm inist ic so t hat all host –host conversat ions go t o a single sensor. Figu r e 6 - 8 . A da t a ce n t e r n e t w or k de sign —de m on st r a t in g ch ok e poin t colle ct ion a n d se n sor loa d ba la ncing

The dat a cent er design in Figure 6- 8 allows inspect ion of all t raffic ent ering and leaving t he dat a cent er from t he corporat e net work and elim inat es any inspect ion asym m et ry int roduced by t he redundant links. The backbone uplinks from t he dat a cent er gat eways ( links 1 t hrough 4) are copied t o a NI DS load balancer and dist ribut ed t o t he sensor farm . We wouldn't dare im plem ent t his inline at t he dat a cent er gat eways ( or dist ribut ion layer) , due t o t he high bandwidt h requirem ent s and asym m et ric pat hs. 6 .2 .2 .3 . Ex t r a n e t de sign Ext ranet m onit oring is especially challenging, as we have less cont rol over t he end syst em s, users, and policies. Figure 6- 9 depict s ext ranet connect ivit y wit h four different part ners, all connect ed via redundant WAN links. For t his scenario, we'll assum e t hat physical and net work access policies are poorly enforced, m aking it a good candidat e for a NI PS deploym ent . Figu r e 6 - 9 . Se n sor pla ce m e n t a n d con n e ct ivit y for e x t r a n e t

Net work links 1 t hrough 4 have an average aggregat e bandwidt h ut ilizat ion of 90 Mbps. To m aint ain t he redundancy of t he net work links, you can deploy a single sensor wit h four pairs of 100 Mbps int erfaces ( eight t ot al int erfaces) . This inline sensor—a NI PS—is configured via specific rules t o prot ect t he corporat e net work from at t acks sourced from t hese part ner net works. The downside? The NI PS sensor is now a single point of failure, adding net work provisioning and priorit y support t o t he responsibilit ies of your t eam .

6 .2 .3 . D e ploy I f you've analyzed your needs and designed your solut ion according t o our advice, you should now be well prepared t o deploy your NI DS. I n t his sect ion, we'll share our experience wit h som e oft en overlooked " got chas" t hat can delay your inst allat ion.

Asym m et ric rout ing As discussed t hroughout t his chapt er, net work pat h redundancy, a com m on feat ure of high- availabilit y design, int roduces m ult iple pat hs and can confuse your NI DS wit h t raffic asym m et ry. You m ust address t his in your design t o prevent asym m et ry from affect ing your NI DS.

Jum bo fram es I f j um bo fram es are enabled on t he net work segm ent s you're m onit oring, m ake sure your select ed NI DS hardware support s t hem .

Applicat ion accelerat ion Net work opt im izat ion solut ions such as Cisco's Wide Area Applicat ion Services ( WAAS)[ 4 6 ] are used t o accelerat e applicat ion perform ance over t he WAN. To provide such accelerat ion, t hey m odify TCP set t ings and com press dat a, which causes packet s t o appear " garbled" t o t he NI DS. I f such opt im izat ion has been deployed on your t arget net works, you m ust ensure t hat t he NI DS' t raffic collect ion is out side t he opt im ized pat h. [ 46]

ht t p: / / www.cisco.com / en/ US/ product s/ ps5680/ Product s_Sub_Cat egory_Hom e.ht m l

Physical m edia Verify t hat appropriat e int erfaces are available for your select ed plat form and t hat t he correct m edia t ypes are present . For exam ple, if your NI DS has LC- t ype connect ors for t he fiber 1 Gbps int erface, m ake sure t he inst all locat ion has LC connect ions available, or has LC- SC, LC- MTRJ, or ot her appropriat e pat ch j um pers. You will also need t o m aint ain agreem ent s wit h t he net work support organizat ions t o ensure t hat your dat a capt ure sources are reliably m aint ained t hrough t roubleshoot ing and hardware/ soft ware upgrades. Having a NI DS as a st andard part of your net work archit ect ure can be ext rem ely beneficial, because when new t echnologies, t opologies, or designs are considered, t he NI DS will be a part of t hat considerat ion. Chapt er 7 int roduces t echniques for m aint aining t hese im port ant event feeds.

6 .2 .4 . Tu n e a n d M a n a ge Once you've com plet ed t he analysis, design, and deploym ent of your NI DS solut ion, you m ust begin t he ongoing process of t uning and m anaging t he solut ion. Wit hout t uning, your sensors will generat e vast num bers of false posit ives, overwhelm ing m onit oring st aff, and event ually causing t he NI DS t o be ignored. Many vendors claim t heir NI DS solut ions t une t hem selves aut om at ically, rem inding m e of lat e- night t elevision com m ercials where engines are run wit hout oil or red wine st ains disappear before your eyes. I t 's sim ply not t rue: t uning is necessary for t he lifet im e of t he NI DS deploym ent . 6 .2 .4 .1 . Tu n e a t t h e se n sor To begin t uning your NI DS, evaluat e t he m ost frequent alert s direct ly on t he sensor. Cisco's I PS soft ware versions 5 and 6 allow you t o look at t he sensor st at ist ics wit h t he com m and show statistics virtual:

blanco-dc-ids-1# show statistics virtual Virtual Sensor Statistics Statistics for Virtual Sensor vs0 Per-Signature SigEvent count since reset Sig Sig Sig Sig Sig Sig Sig Sig

3159.0 5829.0 5837.0 5847.1 6005.0 6054.0 6055.0 3002.0

= = = = = = = =

12 17 22 2 281 49 7 2045681

Not ice t hat signat ure num ber 3002.0 has fired 2 m illion t im es since t he sensor was reset ( which was probably at t he last power cycle) . Signat ure 3002.0 is a TCP SYN Port Sweep. Based on t he descript ion ( which we looked up on Cisco's websit e) , it could easily be at t ribut ed t o a " benign t rigger" ( a false posit ive) , such as t he net work act ivit y generat ed by net work m anagem ent t ools. We'll need t o look furt her t o det erm ine what 's causing t his signat ure t o fire. On t he sensor, we can t rack down which I P addresses have been causing t his alert t o fire in t he past m inut e via the show event alert com m and:

blanco-dc-nms-1# show event alert past 00:01:00 | include 3002

evIdsAlert: eventId=1173255092619515843 severity=high vendor=Cisco originator: hostId: blanco-dc-nms-1 appName: sensorApp time: 2007/04/23 18:50:47 2007/04/23 18:50:47 UTC signature: description=TCP SYN Port Sweep id=3002 version=S2 subsigId: 0 marsCategory: Info/Misc/Scanner marsCategory: Probe/FromScanner interfaceGroup: vs0 vlan: 0 participants: attacker: addr: locality=IN 10.6.30.5 target: addr: locality=IN 10.9.4.4 port: 80 The com m and out put point s t o 10.6.30.5 as t he source of t his t raffic. Because you know your net work, you're aware t hat t his is a net work m anagem ent device. You can now t une t he sensor t o suppress alert s for t his signat ure when t he source m at ches t he address of t he net work m anagem ent device:

blanco-dc-nms-1# conf t blanco-dc-nms-1(config)# service event-action-rules rules0 filters insert drop_mgt_system_alerts signature-id-range 3002 attacker-address-range 10.6.30.5 victim-address-range $IN actions-to-remove produce-alert The preceding configurat ion st anza suppresses alert s from signat ure I D 3002, but only when t he source I P address m at ches 10.6.30.5. Lat er in t his chapt er, we will explain how t o use net work and host variables, enabling us t o apply powerful t uning across our NI DS deploym ent . 6 .2 .4 .2 . Tu n e a t t h e SI M I f you have a Securit y I nform at ion Manager ( SI M) , you can leverage it t o furt her t une your NI DS. Many vendors sell SI M product s, including ArcSight , net Forensics, and Cisco. Query your SI M for t he m ost frequent event s over a 24hour period, t hen invest igat e t he cause, beginning wit h t he m ost frequent event s ( t he really noisy st uff) . Wash, rinse, and repeat . Once you're sat isfied t hat you've suppressed as m any false posit ives as possible ( and you're sat isfied wit h your dat a ret ent ion) , you're halfway t here. You can use t he SI M and t he sensor t o verify t he im pact of new signat ure rollout s. For exam ple, if you pushed out a cust om signat ure t hat had poorly writ t en regex, you could m at ch on t raffic t hat is benign and cause t housands of false posit ives. You can m ake t his verificat ion at t he sensor or at t he SI M, wit h verificat ion at t he SI M becom ing increasingly easier as t he num ber of sensors you have grows. 6 .2 .4 .3 . N e t w or k va r ia ble s I n Chapt er 3 , we dem onst rat ed t he power of net work t axonom y and t elem et ry t o add cont ext ual underst anding t o your I P address space. You can t urn t his enum erat ion of net work t ypes int o " net work variables" t hat you can use in your NI DS configurat ion t o provide cont ext t o your alert s. Several NI DS product s support t his capabilit y, including t he open source Snort soft ware. To configure your NI DS properly, begin by creat ing net work variables from t he net work m et adat a you've gat hered. To add m ore granular cont ext , creat e host variables using t he sam e crit eria. Net work and host variables will provide t he cont ext needed t o creat e act ionable NI DS alert s. Figure 6- 10 depict s a sim ple I P address allocat ion: Cam pus, Dat a Cent er, Part ner, DMZ, and Rem ot e Access net works. Using t he subnet s referenced in Figure 6- 10, let 's creat e our net work variables in Snort , placing t hem in t he

snort .conf configurat ion file:

var var var var var

DC_NETWORK 172.16.16.0/20 PARTNER_NETWORK 172.16.8.0/23 RA_NETWORK 172.16.32.0/20 CAMPUS_NETWORK 172.16.0.0/22 DMZ_NETWORK 172.16.4.0/23

Let 's t ake a st ep backward—wit hout t hese variables defined, our NI DS alert s will only display raw I P inform at ion:

attacker: addr: locality=IN 172.16.9.25 port: 4888 target: addr: locality=IN 172.16.17.220 port: 80 See t he need for cont ext ? All we can see from t his alert is t hat I P address 172.16.9.25 t ried t o at t ack 172.16.17.220. How can we priorit ize t his alert from t he ot her 50 alert s rolling in? Once net work variables are configured in t he NI DS, our alert s get friendly:

attacker: addr: locality=PARTNER_NETWORK 172.16.9.25 port: 4888 target: addr: locality=DC_NETWORK 172.16.17.220 port: 80 Figu r e 6 - 1 0 . Sim ple I P a ddr e ss a lloca t ion w it h de scr ipt ion of fu n ct ion

This alert shows clearly t hat a host in t he part ner net work ( PARTNER_NETWORK) at t acked a host in t he dat a cent er ( DC_NETWORK) ! Think of all t he nslookups and traceroutes you j ust saved yourself. As an added bonus, t his inform at ion can also be used direct ly as query crit eria in your SI M. For exam ple: " show m e all alert s from PARTNER_NETWORK." The power of t hese variables is j ust as obvious for t uning your NI DS. You can use t hem t o reference ent ire subnet s, list s of subnet s, list s of host s, or a com binat ion of bot h wit h a single variable. 6 .2 .4 .4 . Tu n in g w it h h ost va r ia ble s Host variables are slight ly different from net work variables: t hey reference only a single I P address at a t im e. Net work variables t hat define ranges of I P addresses are m ost useful for cont ext ual inform at ion and for som e " m et at uning" such as applying broad t uning policies t o ent ire subnet s and blocks of addresses. We st ill need som et hing m ore granular t o properly t une t he NI DS, as m ost false posit ives will be relat ed t o specific applicat ion t raffic running on a finit e set of host s. Host variables provide t his level of granularit y for t uning false posit ives. Typically, list s of I P addresses for servers of t he sam e kind, such as DNS servers or em ail servers, are t he m ost useful. For exam ple, t he last t im e I checked, we had m ore t han 70 DNS servers in our environm ent . Rat her t han creat ing 70 t uning st at em ent s or even a single st at em ent wit h a list of 70 servers, it is m uch easier t o creat e a variable wit h t he 70 server I P addresses and reference t hat variable in t he t uning st at em ent . Wit h a com m on service such as DNS, it is likely t hat you'll have m ore t han one t uning st at em ent referencing DNS servers, so a variable saves you even m ore t im e, as all you need t o do when a new DNS server is brought online is t o add t he new I P address t o t he variable. All t uning referencing t hat variable will now be applied t o t he new I P address as well! Let 's t ake a look at an exam ple. Previously, we saw t hat a net work m anagem ent server was causing signat ure 3002 t o fire. We know t here are several ot her m anagem ent servers as well, so we need t o creat e a single variable for t hem and reference it in our NI DS t uning. By doing so, we can be assured t hat all t he m anagem ent servers have t he

proper t uning st at em ent s applied:

blanco-dc-nms-1# conf t blanco-dc-nms-1(config)# service event-action-rules rules0 variables MGT_SYSTEMS address 10.6.30.5,10.6.30.6,10.30.6.7,10.50.1.5,10.50.1.6,10.50.1.7 filters insert drop_mgt_system_alerts signature-id-range 4003,3002,2100,2152 attacker-address-range $MGT_SYSTEMS victim-address-range $IN actions-to-remove produce-alert|produce-verbose-alert Let 's assum e t hat we found t hree ot her signat ures t hat were being t riggered by t he net work m anagem ent servers ( 4003, 2100, and 2152) . The preceding t uning st at em ent will drop alert s for all I P addresses defined in t he MGT_SYSTEMS variable. Should I T deploy a new m anagem ent syst em , you need only add it s I P address t o t he variable. New signat ures t hat fire false posit ives sourced from m anagem ent syst em s can be t uned wit h t his variable as well.

A N ot e on D ocu m e n t a t ion One very easily overlooked aspect of NI DS t uning is docum ent at ion of your t uning effort s. Will you rem em ber six m ont hs from now why you t uned I P address 10.44.3.56 for signat ure 4488? Probably not , so you need t o discipline yourself t o docum ent every t uning st at em ent and variable. I deally, you will reference cases t hat led you t o apply such t uning in t he docum ent at ion it self. This will m ake t hings m uch easier should you swit ch vendors or deploy new sensors in a sim ilar net work environm ent . I nit ially, our t eam used flat files wit h RCS revision cont rol, but as t he num ber of sensors and NI DS environm ent s increased t he scalabilit y of flat files quickly eroded. We now use Bugzilla t o t rack our NI DS t uning and it has worked very well ( ht t p: / / www.bugzilla.org/ ) . 6 .2 .4 .5 . Cu st om sign a t u r e s The last NI DS t opic we'll cover involves cust om signat ures. These are rules t hat you creat e and add t o your NI DS, which allow you t o react quickly t o new t hreat s in your environm ent . For exam ple, let 's say you find a piece of m alware on your net work t hat t ries t o resolve a dom ain host ing a com m and and cont rol I RC server: t k.t hirt yfiv.org. I f you want t o ident ify ot her syst em s infect ed wit h t his t roj an, you can deploy a signat ure t hat will fire when t he regex for t his very specific st ring of t ext is seen in a DNS query. On a Cisco I PS sensor, t he configurat ion for t his signat ure is as follows ( not ice t he regex st ring for t k.t hirt yfiv.org) :

sig-description sig-name IRC BOT DOMAINS sig-string-info IRC BOT DOMAINS sig-comment IRC BOT DOMAINS exit engine atomic-ip specify-l4-protocol yes specify-payload-inspection yes regex-string (\x03[Tt][Kk]\x0b[Tt][Hh][Ii][Rr][Tt][Yy][Ff][Ii][Vv]\x03[Oo][Rr][Gg]} exit l4-protocol udp specify-dst-port yes dst-port 53 You'll want t o t une t his cust om signat ure t o drop your DNS servers as a source of at t ack; t hey m ay query t he root server t o resolve t he I P and place it in it s cache.

A properly t uned NI DS deploym ent is one of t he m ost powerful securit y t ools. I t can provide high- fidelit y, act ionable securit y event s t o your incident response t eam . Just rem em ber, you'll get out of it what you put int o it .

6 .3 . Syst e m Loggin g One of t he m ost challenging sources of event dat a t hat you will want t o collect com es from t he wonderful world of syst em logging, or syslog . Syslog inform at ion is part icularly useful in environm ent s where m uch of t he applicat ion t raffic is encrypt ed and cannot be analyzed by t radit ional packet inspect ion. I n fact , syslog m ay be t he only visibilit y you have in m any cases. I n addit ion t o being an operat ional and securit y best pract ice, core t o several securit y st andards and governm ent regulat ions including t he Federal I nform at ion Securit y Managem ent Act of 2002 ( FI SMA) , t he Sarbanes- Oxley Act of 2002 ( SOX) , t he Gram m - Leach- Bliley Act ( GLBA) , t he Paym ent Card I ndust ry ( PCI ) , t he Healt h I nsurance Port abilit y and Account abilit y Act of 1996 ( HI PAA) , and ot hers is t he collect ion of syst em log event s. I n t he Unix world, syslog is a st andard for forwarding log m essages in an I P net work. Syslog m essages are records of event s occurring on a syst em . These event s can t ake several form s, such as a record of a user logging in, a service st art ing, and even cust om m essages sent from applicat ions running on t he syst em . According t o t he lat est version of t he I ETF working group discussion, t o be: [ 47]

[ 47]

t he form at for a syslog m essage is suggest ed

ht t p: / / www.syslog.cc/ iet f/ draft s/ draft - iet f- syslog- prot ocol- 23.t xt

{ PRI } { VERSI ON} { TI MESTAMP} { HOSTNAME} { PROCI D} { MSGI D} { STRUCTURED DATA} That is, t he priorit y of t he m essage, version of t he syslog prot ocol, t im est am p ( as specified by RFC 3339) , host nam e, process ident ifier, m essage ident ifier ( used only for filt ering purposes) , and st ruct ured dat a where t he cont ent s of t he m essage are cont ained. Syslog m essages can and should be filt ered t o include only specific priorit ies. To do t his effect ively, we need t o underst and how t he priorit y num ber is calculat ed. According t o RFC 3164: The Facilit ies and Severit ies of t he m essages are num erically coded wit h decim al values. Som e of t he operat ing syst em daem ons and processes have been assigned Facilit y values. Processes and daem ons t hat have not been explicit ly assigned a Facilit y m ay use any of t he " local use" facilit ies or t hey m ay use t he " user- level" Facilit y. Those Facilit ies t hat have been designat ed are shown in t he following t able along wit h t heir num erical code values. 0 kernel m essages 1 user- level m essages 2 m ail syst em 3 syst em daem ons 4 securit y/ aut horizat ion m essages ( not e 1) 5 m essages generat ed int ernally by syslogd 6 line print er subsyst em 7 net work news subsyst em 8 UUCP subsyst em 9 clock daem on ( not e 2) 10 securit y/ aut horizat ion m essages ( not e 1) 11 FTP daem on 12 NTP subsyst em 13 log audit ( not e 1) 14 log alert ( not e 1) 15 clock daem on ( not e 2) 16 local use 0 ( local0)

17 local 18 local 19 local 20 local 21 local 22 local 23 local

use 1 ( local1) use 2 ( local2) use 3 ( local3) use 4 ( local4) use 5 ( local5) use 6 ( local6) use 7 ( local7)

N u m e r ica l Code

Fa cilit y

Table 1. syslog Message Facilit ies Not e 1 - Various operat ing syst em s have been found t o ut ilize Facilit ies 4, 10, 13 and 14 for securit y/ aut horizat ion, audit , and alert m essages which seem t o be sim ilar. Not e 2 - Various operat ing syst em s have been found t o ut ilize bot h Facilit ies 9 and 15 for clock ( cron/ at ) m essages. Each m essage Priorit y also has a decim al Severit y level indicat or. These are described in t he following t able along wit h t heir num erical values. 0 Em ergency: syst em is unusable 1 Alert : act ion m ust be t aken im m ediat ely 2 Crit ical: crit ical condit ions 3 Error: error condit ions 4 Warning: warning condit ions 5 Not ice: norm al but significant condit ion 6 I nform at ional: inform at ional m essages 7 Debug: debug- level m essages N u m e r ica l Code

Se ve r it y

Table 2. syslog Message Severit ies The Priorit y value is calculat ed by first m ult iplying t he Facilit y num ber by 8 and t hen adding t he num erical value of t he Severit y. For exam ple, a kernel m essage ( Facilit y= 0) wit h a Severit y of Em ergency ( Severit y= 0) would have a Priorit y value of 0. Also, a " local use 4" m essage ( Facilit y= 20) wit h a Severit y of Not ice ( Severit y= 5) would have a Priorit y value of 165. I n t he PRI part of a syslog m essage, t hese values would be placed bet ween t he angle bracket s as < 0> and < 165> respect ively. The only t im e a value of " 0" will follow t he " < " is for t he Priorit y value of " 0" . Ot herwise, leading " 0" s MUST NOT be used. Here is an exam ple of a syslog m essage from a Linux- based syst em , which illust rat es t he use of t his form at :

Apr 20 04:16:50 bbx-vip-1.blanco.com ntpd [4024]: synchronized to 10.13.136.1

Not ice t hat t he Priorit y field is not included, as it is not required or configured on t his part icular syst em . When you break down t he preceding code based on t he I ETF form at , you get t he following: Tim est am p: Apr 20 04:16:50 Host nam e: bbx-vip-1.blanco.com ProcI D: ntpd [4024] St ruct ured dat a: synchronized to 10.13.136.1 This m essage t ells us t hat t he ntpd process on server bbx-vip-1.blanco.com synchronized it s t im e t o 10.13.136.1 on April 20 at 04: 16: 50. I s t his m essage useful in t he cont ext of securit y m onit oring? The answer is no, which is t rue for t he vast m aj orit y of m essages you're likely t o receive from any given host . Depending on how syslog is configured, you m ay find t he volum e of m essages overwhelm ing. Therein lies t he challenge t o collect ing syslog dat a for securit y m onit oring. As wit h NI DSs, t he separat ion of " t he wheat from t he chaff" det erm ines whet her syslog will be a useful event source or a noisy dist ract ion.

6 .3 .1 . Ke y Syslog Eve n t s To m ake t he collect ion of syslog dat a m anageable, first we m ust focus on som e key syslog event s. For securit y m onit oring, we are int erest ed in four t ypes of event s:

Aut hent icat ion event s Aut horizat ion event s Daem on st at us event s Securit y applicat ion event s I t should be not ed t hat different flavors of Unix/ Linux st ore t heir logfiles in different locat ions and use different filenam es. Be aware of which files you expect t o cont ain aut hent icat ion logs, aut horizat ion logs, syst em m essages, and ot her dat a. As an exam ple of t his diversit y, here are t he locat ions of t he su log m essages for each of t he following Unix variant s: FreeBSD: / var/ log/ m essages HP- UX: / var/ adm / sulog Linux: / var/ log/ m essages Solaris: As defined in / et c/ default / su This diversit y in log locat ion is where you should address t wo im port ant issues upfront . First , applying st andardizat ion t o a set of servers running t he sam e OS will m ake it m uch easier t o collect syslog dat a from t hem . Take advant age of st andardizat ion by im plem ent ing a syslog configurat ion t em plat e on all servers of a given * nix dist ribut ion. Second, you can't be an expert in everyt hing. You will likely need help from t he I T sys adm ins. Buy t hem a beer, t hank t hem for st andardizing as m uch as t hey can, and be up front about what you're t rying t o accom plish. 6 .3 .1 .1 . Au t h e n t ica t ion e ve n t s Aut hent icat ion event s record access t o a syst em or applicat ion, generally referred t o as logon and logoff m essages. These are a prim ary source of securit y m onit oring dat a because t hey provide t im est am ped at t ribut ion of account usage by userid. A set of logon and logoff m essages for a single userid can show when t he user logged in ( and subsequent ly logged out ) , giving a t im e fram e for when t he account was in use. These m essages m ay help you uncover som e unexpect ed event s, such as privileged account s accessing syst em s at unusual t im es or from unusual syst em s:

Apr 25 20:22:16 xsrv.blanco.com sshd(pam_unix)[15585]: session opened for user root by root(uid=0)

From t his syslog m essage, we can see t hat a session was opened for t he root user account on xsrv.blanco.com at 20: 22: 16 on April 25. Perhaps t his is out side what is considered norm al because it is aft er business hours, or perhaps xsrv.blanco.com is a server t hat cont ains very sensit ive inform at ion. You m ay want t o invest igat e unusual login m essages. Access by service account s ( such as www or apache ) is norm al, but access from ot her account s m ight be unusual and require invest igat ion.

6 .3 .1 .2 . Au t h or iza t ion e ve n t s Aut horizat ion event s record changes of privilege level for a given userid, such as changing t o t he root user via t he su –root com m and, or via t he superuser- do facilit y as in sudo su – root . The following m essage is an exam ple of an aut horizat ion event :

Apr 4 21:25:49 xsrv.blanco.com sudo: USER=root ; COMMAND=/bin/su -

rajesh : TTY=pts/4 ; PWD=/users/rajesh ;

The userid rajesh at t em pt ed t o becom e t he " root " user by issuing t he com m and sudo su – . Such m essages would be com plet ely norm al if rajesh was a syst em adm inist rat or whose j ob was t o m anage t he server. However, if user rajesh is an int ern or an adm inist rat ive assist ant , furt her invest igat ion is likely warrant ed: we could query our logs for userid rajesh t o see whet her t here is any ot her suspicious act ivit y. 6 .3 .1 .3 . D a e m on st a t u s e ve n t s Daem on st at us event s record a change in st at e for a given daem on, t ypically st art and st op m essages. These m essages alone do not necessarily indicat e suspicious behavior. You m ust int erpret t he m essages wit hin t he cont ext s of t im e of day and userid. For exam ple, consider t he following:

Apr 27 13:11:03 (uid=0) Apr 27 13:11:10 cfry(uid=44642) Apr 27 13:11:30 Apr 27 13:11:30

blanco-nnc sshd(pam_unix)[2862]: session opened for user cfry by blanco-nnc su(pam_unix)[2901]: session opened for user root by blanco-nnc sshd: sshd -TERM succeeded blanco-nnc sshd: succeeded

We see t hat user cfry connect ed t o server blanco-nnc j ust aft er 1: 00 p.m . He t hen proceeded t o open a root session, and soon aft er, t he SSH daem on, sshd , was st opped and st art ed. I s it norm al for services t o be st opped and st art ed on t his server at 1: 00 p.m .? I s cfry a sys adm in? Querying our logs furt her, our invest igat ion finds t hat userid cfry had recent ly downloaded and inst alled a t roj aned SSH daem on t hat capt ures all keyst rokes and logs t hem t o a file. Monit oring daem on st at us event s, in t his case, allowed us t o discover t he suspicious act ivit y t hat drove deeper invest igat ion. 6 .3 .1 .4 . Se cu r it y a pplica t ion e ve n t s Securit y applicat ion event s record act ivit ies of applicat ions inst alled specifically t o perform a securit y funct ion, such as Tripwire, Sam hain, Port Sent ry, and Cisco Securit y Agent ( CSA) . These file int egrit y and host - based NI DSs can be a source of valuable incident response dat a.

6 .3 .2 . Syslog Te m pla t e s Let 's assum e you know t he kinds of m essages you want t o receive. How do you ensure t hat you receive t hem from each syst em in your infrast ruct ure? I f you are a large ent erprise t hat runs m any different operat ing syst em s, you will need t o have a st andard syslog configurat ion t em plat e if you hope t o collect log m essages consist ent ly and reliably. The t em plat e will likely vary by server funct ion and locat ion, such as a DMZ web server versus an int ernal web server. Also, you m ay choose t o log m ore det ailed levels on your m ost crit ical servers. The following is an exam ple of a host configurat ion file for a DMZ server running Solaris:

# Copyright (c) 1991-1998 by Sun Microsystems, Inc. # All rights reserved.

# # syslog configuration file. # # This file is processed by m4 so be careful to quote ('') names # that match m4 reserved words. Also, within ifdef's, arguments # containing commas must be quoted. # *.err;kern.notice;auth.notice /dev/sysmsg *.err;kern.debug;daemon.notice;mail.crit /var/adm/messages *.alert;kern.err;daemon.err *.alert

root

operator

*.emerg

*

# if a non-loghost machine chooses to have authentication messages # sent to the loghost machine, un-comment out the following line: #auth.notice ifdef('LOGHOST', /var/log/authlog, @loghost) mail.debug

ifdef('LOGHOST', /var/log/syslog, @loghost)

# # non-loghost machines will use the following lines to cause "user" # log messages to be logged locally. # ifdef('LOGHOST', , user.err /dev/sysmsg user.err /var/adm/messages user.alert 'root, operator' user.emerg * ) # # $Id: syslog.conf template,v 1.5 2005/02/30 rajavin Exp $ # local2.notice /var/log/sudolog local2.notice @dmzsyslog.blanco.com local3.info /var/adm/ssh.log local3.info @dmzsyslog.blanco.com local4.info /var/adm/tcpwrap.log local4.info @dmzsyslog.blanco.com local7.info /var/adm/portsentry.log local7.info @dmzsyslog.blanco.com *.err;kern.debug;daemon.notice;mail.crit @dmzsyslog.blanco.com auth.notice /var/log/authlog auth.notice @dmzsyslog.blanco.com

6 .3 .3 . Ke y W in dow s Log Eve n t s Microsoft operat ing syst em s can writ e t o different Event logfiles t hat are divided int o applicat ion, syst em , and securit y logs. Unfort unat ely, it 's not easy t o ret rieve t hese logs from t he syst em wit hout t he use of t hird- part y soft ware such as I nt erSect Alliance's Snare. [ 4 8 ] I n addit ion, if you are t rying t o view t he logs on anyt hing ot her t han an end syst em , it is nearly im possible t o do so wit h t he Event Viewer provided in Windows. Get t ing t he log m essages out of t he Event Viewer and int o a st andard syslog form at will allow us t o use t hem for securit y m onit oring. The event s list ed in t he following sect ions are t he kinds of m essages you will want t o record for securit y m onit oring and for incident response.

[ 48]

ht t p: / / www.int ersect alliance.com / proj ect s/ SnareWindows/

N OTE

Much of t he inform at ion in t he following sect ions is based on t he excellent inform at ion put t oget her by Randy Franklin Sm it h, which you can access at t he Ult im at e Windows Securit y sit e.[ 4 9 ] [ 49]

ht t p: / / www.ult im at ewindowssecurit y.com / encyclopedia.aspx

6 .3 .3 .1 . W in dow s a u t h e n t ica t ion Num erous Windows event I Ds are devot ed t o aut hent icat ion m essages—referred t o as " logon" and " logoff" m essages. Table 6- 3 list s t he different Windows aut hent icat ion event I Ds, t heir applicable OS versions, and a descript ion of t he event. 528 All versions Successful logon 529 All versions Logon failure: unknown usernam e or bad password 530 All versions Logon failure: account logon t im e rest rict ion violat ion 531 All versions Logon failure: account current ly disabled 532 All versions Logon failure: t he specified user account has expired 533 All versions Logon failure: user not allowed t o log on at t his com put er 534 All versions Logon failure: user has not been grant ed t he request ed logon t ype at t his m achine 535 All versions Logon failure: t he specified account 's password has expired 536 All versions Logon failure: t he Net Logon com ponent is not act ive 537 All versions Logon failure: t he logon at t em pt failed for ot her reasons 538 All versions User logoff 539 All versions Logon failure: account locked out 540 Windows XP, Windows 2000, Windows Server 2003 Successful net work logon 552 Windows Server 2003 Logon at t em pt using explicit credent ials Ta ble 6 - 3 . W in dow s logon a n d logoff e ve n t s Eve n t I D OS ve r sion Descript ion 6 .3 .3 .2 . W in dow s a u t h or iza t ion Several Windows event I Ds are devot ed t o aut horizat ion m essages, used t o record access t o privileged services or privileged act ions. Table 6- 4 list s t he different Windows aut horizat ion event I Ds, t heir applicable OS versions, and a descript ion of t he event .

577 All versions Privileged service called 578 All versions Privileged obj ect operat ion 601 Windows Server 2003 At t em pt t o inst all service 602 Windows Server 2003 Scheduled t ask creat ed 608 Windows Server 2003 User right assigned 609 All versions User right rem oved 612 All versions Audit policy change Ta ble 6 - 4 . W in dow s a u t h or iza t ion e ve n t s Eve n t I D OS ve r sion Descript ion

Of course, all of t he event I Ds change wit h Windows Vist a and Windows Server 2008 ( see Table 6- 5 ) . 4624 An account was successfully logged on 4625 An account failed t o log on 4648 A logon was at t em pt ed using explicit credent ials Ta ble 6 - 5 . W in dow s Vist a a n d W in dow s Se r ve r 2 0 0 8 logon e ve n t s Eve n t I D Descript ion 6 .3 .3 .3 . W in dow s pr oce ss st a t u s e ve n t s Windows process st at us event s record t he act ion of st art ing and st opping processes on a Windows syst em . Table 6- 6 provides a list of such event s. Not e t hat recorded event s will likely be volum inous. You will need t o filt er t hem t o save only t hose t hat are crit ical t o server availabilit y, int egrit y, or confident ialit y. 592 All versions A new process has been creat ed 593 All versions A process has exit ed 594 All versions A handle t o an obj ect has been duplicat ed 595 All versions I ndirect access t o an obj ect has been obt ained Ta ble 6 - 6 . W in dow s pr oce ss st a t u s e ve n t s Eve n t I D OS ve r sion Descript ion 6 .3 .3 .4 . W in dow s dom a in con t r olle r e ve n t s Windows dom ain cont roller event s record event s unique t o dom ain cont rollers, usually on a syst em funct ioning solely for t hat purpose. This includes event s recording policy and group changes, securit y soft ware ( Kerberos) , and

aut hent icat ion m essages. Table 6- 7 provides a list of relevant dom ain cont roller event s, and Table 6- 8 provides a list of relevant Kerberos event s. Table 6- 9 provides a cross- reference for convert ing error codes ( which are recorded in decim al form at ) t o hexadecim al. I t also serves as a reference for com m on error m essages you m ay receive. 617 All versions Kerberos policy changed 632 All versions Global group m em ber changed 636 All versions Local group m em ber changed 643 Windows 2000 Dom ain policy changed Windows Server 2003 Dom ain policy changed 660 All versions Universal group m em ber changed 672 [ 5 0 ] Windows Server 2003 Kerberos success and failure 675 [ 5 1 ] Windows 2000 Preaut hent icat ion failed ( Kerberos) 680 [ 5 2 ] Windows 2000 Account used for logon by % 1 Windows Server 2003 Logon at t em pt 681 [ 5 3 ] Windows 2000 The logon t o account : % 1 by: % 2 from workst at ion: % 3 failed Windows Server 2003 The logon t o account : % 1 by: % 2 from workst at ion: % 3 failed Ta ble 6 - 7 . W in dow s dom a in con t r olle r e ve n t s Eve n t I D OS ve r sion Descript ion

[ 50]

From page 83 of RFC 1510, ht t p: / / www.iet f.org/ rfc/ rfc1510.t xt .

[ 51]

Also from page 83 of RFC 1510, ht t p: / / www.iet f.org/ rfc/ rfc1510.t xt .

[ 52]

Windows Server 2003 uses event I D 680 inst ead of 681.

[ 5 3 ] From ht t p: / / support .m icrosoft .com / kb/ 273499/ en- us/ : " Not e t hat t he error codes in t he Event Log m essage are in decim al form , but t hey are act ually hexadecim al values." You can t ranslat e t he values from decim al t o hexadecim al by using t he Calculat or t ool in Windows 2000, or by viewing Table 6- 7 . The m ost com m on error codes you m ay receive are defined in Table 6- 7 .

6 Client not found in Kerberos dat abase ( no such usernam e) 18 Client 's credent ials have been revoked ( account disabled) 23 Password has expired 24 Preaut hent icat ion inform at ion was invalid ( incorrect password) Ta ble 6 - 8 . Ke r be r os e r r or codes Er r or code Ca u se 3221225572 C0000064

User logon wit h m isspelled or bad user account 3221225578 C000006A User logon wit h m isspelled or bad password 3221225581 C000006D User logon has incorrect usernam e 3221225583 C000006F User logon out side aut horized hours 3221225584 C0000070 User logon from unaut horized workst at ion 3221225585 C0000071 User logon wit h expired password 3221225586 C0000072 User logon t o account disabled by adm inist rat or 3221225875 C0000193 User logon wit h expired account 3221226020 C0000224 User logon wit h " Change Password at Next Logon" flagged 3221226036 C0000234 User logon wit h account locked Ta ble 6 - 9 . N TLM e r r or code s Er r or code H e x a de cim a l va lu e Ca u se

Colle ct in g W in dow s Se r ve r Eve n t s in Syslog For m a t One of t he prim ary challenges t o collect ing Windows event s from a server is get t ing t he event s out of t he propriet ary Event Viewer and int o a st andard form at such as syslog. A com m on way t o do t his is via t hird- part y soft ware such as Snare. Snare convert s t he event s int o a syslog- com pat ible form at and has feat ures t o allow an adm inist rat or t o specify which event I Ds will be forwarded t o a syslog collect or. The Snare form at for a Windows Adm inist rat or logon event looks like t his:

Dec 12 18:27:22 dc.blanco.com MSWinEventLog 1 Security 39226 Dec 12 18:27:22 2008 538 Security Adminstrator User Success Audit DC Logon/Logoff User Logoff: User Name: Administrator Domain: BWIR Logon ID: (0x0,0x404843F) Logon Type: 3 39211

6 .3 .3 .5 . W in dow s se cu r it y a pplica t ion e ve n t s Ant ivirus logs can be an essent ial t ool in det ect ing and m it igat ing out breaks in a t im ely and precise fashion. I m agine t his scenario: a m iscreant creat es a virus variant and int ends t o t arget execut ives at your com pany ( a pract ice t hat is oft en called spearphishing ) by sending t hem an infect ed file via em ail wit h som e specially craft ed social engineering cont ent . Unfort unat ely, som e of your execut ives clicked t he file and a few of t hem ran t he virus file. One of t he m ore t echnically savvy execut ives sends an em ail t o report t he issue:

Security Team, I received an email yesterday that had a suspicious attachment. The email contained some references to an upcoming product and appeared to be from one of our vendor partners. I didn't open the attachment and my antivirus didn't alert on it initially. But this morning the AV program says that the file has a virus but it couldn't clean it. Please have a look at my computer when I return next week. I spoke with a few other executives and they received the email as well.

Thanks, Mikael Shekotzi Vice President, CFO Blanco Wireless, Inc.

Luckily, you are collect ing your ant ivirus logs int o your syslog collect or and can query for act ivit y on t his execut ive's com put er, called m shekot zi- wxp , im m ediat ely. You find t hat your ant ivirus product det ect ed t he virus but couldn't clean it :

Tue_Apr_29_12:10:03_EDT_2008|mshekotzi-wxp|172.16.8.12|Administrator|2008-04-29 16:06:27|On-Demand Scan|W32/Sdbot.gen.an|trojan|c:\WINDOWS\system32\Explorer\xA0.exe|5.2.0|4.0.5283|Cl ean-Delete Error|Unresolved

A quick query of your syslog dat a for file xA0.exe reveals t hat at least seven ot her execut ive officers at Blanco received t he em ail, including t he CEO and CTO. I n addit ion, one of t he execut ive adm inist rat ive assist ant s opened a second em ail from t he sam e sender t hat included a Word docum ent . Her com put er showed a CSA m essage t hat let s us know t hat " som et hing bad" has probably happened:

Tue_Apr_29_13:40:05_EDT_2008|ddmilas-wxp|10.6.20.245|186|833|2008-04-30 01:36:20|The process 'C:\Program Files\Microsoft Office\OFFICE11\WINWORD.EXE' (as user BLANCO\ddmilas) attempted to access a resource which resulted in the user being asked the following question. 'The process C:\Program Files\Microsoft Office\OFFICE11\WINWORD.EXE is attempting to invoke a system function from a buffer. Do you wish to allow this?' The user was queried and a 'Yes' response was received.

We now know which execut ives received t he em ail, which execut ives' com put ers were able t o det ect and delet e t he virus, and which were not . By analyzing t he CSA log m essages we can see which em ployees' syst em s were likely t roj aned, because t hey answered " yes" t o t he pop- up CSA prom pt . These logs allow us t o priorit ize rem ediat ion effort s appropriat e t o t he scope of t he spearphishing at t ack. Syslog dat a can be ext rem ely beneficial, but collect ion and analysis appear quit e daunt ing due t o t he num ber and variet y of possible m essages recorded by syst em s. Windows event s can be ext rem ely difficult t o collect in a useful way given t he com plexit y of t he operat ing syst em 's im plem ent at ion ( Hex codes, Kerberos codes, different codes for different OS versions) , t he propriet ary form at ( export t o CSV doesn't adequat ely address t his) , and t he sheer volum e of event s t hat a single server can generat e. However, if we follow t he guidance given in Chapt er 4 ( t o choose our t arget syst em s) , and rest rict t he event s we are collect ing and analyzing t o include only securit y event s, t he collect ion process becom es m uch m ore m anageable. St art wit h crit ical Windows servers, creat e a workable logging policy, and t hen apply t hat sam e policy t o ot her crit ical Windows servers of t he sam e t ype.

6 .3 .4 . Applica t ion Loggin g Though t hey are m ore com plex, applicat ion logs are an excellent source of securit y event dat a. Depending on t he level of cust om izat ion allowed, applicat ion logs can be correlat ed wit h ot her logs t o provide insight int o specific event t ypes, and can be a valuable source of forensic dat a. Several applicat ions now support logging direct ly int o syslog and provide a range of logging levels. Apache Web Server is a good exam ple of t his capabilit y. URL access logs ( Apache access_log ) are an incredibly powerful source of forensic dat a for securit y m onit oring. I m agine a scenario where your web applicat ion t eam found t hat a m isconfigurat ion m ay have exposed som e confident ial dat a:

Security Team, We received an email from a customer claiming that we exposed his contact data via our webserver hosted on www.blancowireless.com. We checked into his claim and it appears to be valid. I'm not sure how to respond to this situation, there may be more information exposed as well. Can you help us look at the logs? Thanks, Jayson Alodnom Web Administrator Blanco Wireless, Inc.

A quick phone call wit h Jayson finds t hat t he web adm inist rat ors are logging t he access_log file locally on t he server and t here should be a couple of weeks' wort h of hist ory in t here. Looking t hrough t he logs we find t hat several downloads of a spreadsheet occurred because som eone post ed it t o a cust om er collaborat ion blog host ed on blancowireless.com . The first ent ry shows a 2 MB file being uploaded t o t he blog:

37.33.7.57 - - [28/Apr/2008:08:31:11 -0700] "POST /blancoblog/submit.php HTTP/1.1" 200 2045589 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"

The following t wo access_log file ent ries show downloads of t he Excel spreadsheet by t wo different I P addresses:

33.80.7.20 - - [29/Apr/2008:01:08:55 -0700] "GET /blancoblog/php/uploads/G8-exechome-phone-numbers.xls HTTP/1.0" 200 2045589 17.12.44.88 - - [29/Apr/2008:01:48:11 -0700] "GET /blancoblog/php/uploads/G8-exechome-phone-numbers.xls HTTP/1.0" 200 2045589

A full query of t he logs shows t hat at least 20 ot her downloads occurred. I n addit ion, t he I P address 37.33.7.57 t hat uploaded t he file appears t o have uploaded a few ot her files, but queries don't show download act ivit y. Querying ot her syst em logs, we see t hat 37.33.7.57 was t rying t o exploit various vulnerabilit ies, for which Blanco's syst em adm inist rat ors had already pat ched. I t was decided t o block t he I P address from reaching Blanco's net work, which should m it igat e furt her at t acks.

6 .3 .5 . D a t a ba se Loggin g One oft en- overlooked source of incident dat a com es from t he audit logs in a dat abase. The SYS.AUD$ t able in Oracle can be a source of ext rem ely useful securit y incident inform at ion . N OTE When discussing dat abase logging wit h your dat abase adm inist rat ors, you m ay find t hat m uch of t his inform at ion is already being logged as part of regulat ory com pliance wit h SOX, HI PAA, or t he European Union Direct ive on Privacy and Elect ronic Com m unicat ions. Get t ing access t o t hese logs should be considerably easier t han convincing t he DBAs t o t urn on logging j ust for you.

According t o t he docum ent at ion for Oracle 10g , [ 5 4 ] an Oracle dat abase has four general t ypes of audit ing: [ 54]

ht t p: / / download- east .oracle.com / docs/ cd/ B14117_01/ net work.101/ b10773/ audit ing.ht m # DBSEG125

St at em ent audit ing Wat ching SQL st at em ent s, privileges, obj ect s, and so on. The audit inform at ion is writ t en t o eit her SYS.AUD$ or t he operat ing syst em as syslog, as configured in t he AUDIT_TRAIL init ializat ion param et er.

Fine- grained audit ing Wat ching for specific act ivit ies, such as t im e of day, user- based rules, specific t ables being updat ed or select ed, or a Boolean condit ion check for a com binat ion of rules. These are writ t en t o SYS.FGA_LOG$ or t o an XML file residing on t he operat ing syst em . This is configured in t he DBMS_FGA.ADD_POLICY param et er.

Privilege audit ing Wat ching for user SYS and anyone connect ing wit h SYSDBA or SYSOPER privileges by logging t hese act ivit ies t o syslog on Unix or t he Event log on Windows. This is configured in t he AUDIT_TRAIL param et er.

Schem a obj ect audit ing Focused audit ing based on a specified schem a obj ect , such as SELECT on a sensit ive obj ect . This t ype of audit ing applies t o all users of t he dat abase. The audit inform at ion is writ t en t o eit her SYS.AUD$ or t he OS syslog, as set in t he AUDIT_TRAIL init ializat ion param et er.

Be careful when at t em pt ing t o audit act ions on a dat abase. Query t im e and disk ut ilizat ion can be negat ively im pact ed by an overzealous audit ing configurat ion, such as audit ing SELECT st at em ent s on a com m only used dat abase. I n addit ion, t he AUD$ t able is in t he SYSTEM t ablespace, which will cause t he dat abase t o hang when filled t o capacit y. Therefore, it is crit ical t o have a purge st rat egy in place for t he AUD$ t able.

The default audit ing configurat ion for Oracle records dat abase st art up and shut down, connect ions from privileged account s, and changes m ade t o t he dat abase st ruct ure. Oracle's docum ent at ion [ 5 5 ] provides configurat ion guidance wit h exam ples. Here are a few exam ples of act ivit ies you m ay want t o record: [ 55]

ht t p: / / download.oracle.com / docs/ cd/ B28359_01/ net work.111/ b28531/ t oc.ht m

DESCRIBE st at em ent s on product ion dat abases, which indicat e reconnaissance by som eone unfam iliar wit h t he dat abase st ruct ure SELECT

st at em ent s for sensit ive fields in a dat abase, such as em ployee com pensat ion or m edical records

Dat abase connect ions for privileged account s out side of norm al m aint enance hours Audit ing m essages can be recorded in a t able, t o an XML file on t he local syst em , or t o syslog on t he local syst em . The recording locat ion is set as an init ializat ion param et er for t he dat abase. Here is an exam ple of bot h t he XML and syslog form at s for an AUDIT m essage. I t shows user JEBEDIAH has issued a SQL query on sensit ive dat abase fields ( select * from medical.provider where provider = :h ) . Here's t he m essage in XML form at :

10.2

1 224655 10 1 2008-04-08T08:46:10.77845 JEBEDIAH oracle blancodb1 32389 pts/2 0 MEDICAL PROVIDER/Object_Name> 103 0 6447496045 ---------S------ select * from medical.provider where provider = :h

Here's t he m essage in syslog form at :

Apr 10 08:46:10 oradba Oracle Audit[28955]: SESSIONID: "224655" ENTRYID: "1" STATEMENT: "10" USERID: "JEBEDIAH" USERHOST: "blancodb1" TERMINAL: "pts/2" ACTION: "103" RETURNCODE: "0" OBJ$CREATOR: "JJRM" OBJ$NAME: "PROVIDER" SES$ACTIONS: "---------S------" SES$TID: "6447474" OS$USERID: "oracle"

To pull or push t hese m essages from t he dat abase host , you m ust use eit her XML or syslog. You can export syslog from t he dat abase host by configuring t he host 's syslog daem on, but XML will require addit ional script ing. We recom m end t he use of syslog for sim plicit y. You can accom plish t his wit h t he following configurat ion st eps: 1.

Set init ializat ion param et ers ( in init .ora ) :

AUDIT_TRAIL=OS AUDIT_SYSLOG_LEVEL=USER.ALERT 2.

Configure / et c/ syslog.conf t o writ e t o a separat e logfile for audit logs:

user.alert /var/log/audit.log This configurat ion will separat e t he m essages generat ed by Oracle audit ing from t he m essages generat ed by t he syst em ; ot herwise, t he audit t rail m essages will be logged t o / var/ log/ m essages by default .

6 .3 .6 . Colle ct in g Syslog

Syslog collect ion solut ions can be as sim ple as a server running a syslog daem on, or as com plex as an infrast ruct ure wit h m ult iple layers of forwarding and filt ering bet ween servers. Many open source and com m ercial t ools are available t o build a solut ion. Syslog collect ion for large ent erprises t ypically m akes use of a well- indexed dat abase t o provide fast query t im es. As m ent ioned previously, you can configure each host t o forward syslog m essages t o a cent ral collect ion server, as depict ed in Figure 6- 11 . Figu r e 6 - 1 1 . Syslog colle ct ion

Using t he st andard syslog prot ocol, you can collect from a variet y of sources, including ant ivirus and host int rusion det ect ion syst em ( HI DS) logs on your deskt op net work, event s from your net work devices, and server and applicat ion logs. Windows Event logs, as m ent ioned previously in t his chapt er, require a forwarding m echanism such as Snare t o send in syslog form at . I f your environm ent is larger t han what a single syslog collect or can handle, you m ay need t o deploy regional syslog collect ors t hat relay t he m essages t o a cent ralized collect or, as depict ed in Figure 6- 12 . syslog-ng is a popular syslog collect or and relay. I t provides cont ent - based filt ering and t he abilit y t o use connect ionorient ed TCP as t he t ransport in place of t he default UDP, which is connect ionless. The cent ral collect or, depict ed at t he bot t om of Figure 6- 12 , is oft en a soft ware solut ion t hat provides fast queries for large volum es of dat a, such as t hose offered by LogLogic [ 5 6 ] and Splunk. [ 5 7 ] These packages t ypically load t he m essages int o a well- indexed

dat abase t hat is opt im ized for fast query perform ance. I n addit ion t o securit y m onit oring report s, t hey also offer specialized " canned" report s for regulat ory com pliance, operat ional issues, and business report ing. For exam ple, our t eam queries ant ivirus logs every four hours t o find syst em s report ing a virus t hat is det ect ed but unresolved: [ 56]

ht t p: / / www.loglogic.com /

[ 57]

ht t p: / / www.splunk.com /

Thu_May__8_10:30:09_EDT_2008|SAL-W2K|172.16.64.95|NT AUTHORITY\SYSTEM|2008-06-08 13:58:09|OAS|Vundo|trojan|C:\WINDOWS\system32\sstts.dll|5.2.0|4.0.5290|Clean-Delete Error|Unresolved

The preceding m essage t ells us t hat t here is a t roj an on SAL- W2K t hat our ant ivirus soft ware cannot clean; t herefore, a deskt op PC support t echnician should rem ediat e or rebuild t he syst em im m ediat ely. We can now query t he rest of our logs for act ivit y sourced from t he m achine's Net BI OS nam e ( SAL- W2K) or I P address ( 172.16.64.95) . The following are exam ple CSA logs from t his m achine:

Mon_May__21_06:10:01_EDT_2008|SAL-W2K|172.16.16.64|1691|833|2008-06-08 15:37:16|The process 'C:\Program Files\Internet Explorer\iexplore.exe' (as user BLANCO\SAL) attempted to access a resource which resulted in the user being asked the following question. 'A process is attempting to invoke C:\Documents and Settings\SAL\Local Settings\Temporary Internet Files\Content.IE5\7GF6TOJ3\registry-cleanexpert[1].exe, which has not previously been seen on this system. Do you wish to allow this?' The user was queried and a 'Yes' response was received.

Blanco em ployee Salm on Fydle's syst em logged a CSA m essage, showing t hat an execut able t ried t o launch from his browser's cache. Unfort unat ely, he answered " yes" when CSA asked him whet her he want ed t o allow t his act ivit y. From t he following m essage, we can see t hat t he m achine is likely root kit t ed ( ent irely com prom ised wit h m alware) :

Tue_May__22_01:50:03_EDT_2008|SAL-W2K|172.16.16.64|186|547|2008-06-07 00:00:51|The process 'C:\Program Files\HaX\Hax\Hax.exe' (as user BLANCO\SAL) attempted to call the function VirtualHax from a buffer (the return address was 0x8700ced). The code at this address is 'fc506a04 68001000 0051ff93 891c0010 8a07a880 74042c80 8807598d 45fc50ff' This either happens when a process uses self-modifying code or when a process has been subverted by a buffer overflow attack. The operation was allowed.

Figu r e 6 - 1 2 . Syslog colle ct ion w it h r e la y a n d da t a ba se in de x in g

CSA is doing it s j ob, knocking down m odules as t hey t ry t o load, as shown in t he following m essage:

Tue_May__22_06:30:08_EDT_2008|SAL-W2K|72.163.165.229|46|762|2008-06-06 15:58:42|Kernel functionality has been modified by the module sprs.sys. The module 'sprs.sys' is used by entries in the System syscall table. The specified action was taken to set detected rootkit as Untrusted.

Finally, CSA st ops a connect ion from being m ade t o a likely com m and and cont rol server or a sit e host ing addit ional m alware. You also can use t his I P address in a Net Flow query t o ident ify any ot her syst em s t hat m ay have successfully connect ed t o I P address 207.46.248.249. At t he very least , we need t o get SAL- W2K off t he net work!

Tue_May_22_07:00:08_EDT_2008|SAL-W2K|172.16.16.64|54|452|2008-04-22 16:27:12|The process 'C:\WINDOWS\explorer.exe' (as user BLANCO\SAL) attempted to initiate a connection as a client on TCP port 80 to 207.46.248.249 using interface Wired\Intel(R) PRO/1000 MT Mobile Connection. The operation was denied.

The following syslog ent ries from a Linux server show a user, cfry , t rying t o escalat e privileges t o t he root user using the su – com m and, wit h several failures followed by a successful su m essage:

Apr 28 21:17:25 uid=1000 euid=0 Apr 28 21:17:27 Apr 28 21:17:27

blanco-lnx su(pam_unix)[9710]: authentication failure; logname= tty=pts/0 ruser=cfry rhost= user=root blanco-lnx su[9710]: pam_authenticate: Authentication failure blanco-lnx su[9710]: FAILED su for root by cfry

Apr 28 21:17:27 blanco-lnx su[9710]: - pts/0 cfry:root Apr 28 21:17:30 blanco-lnx su(pam_unix)[9711]: authentication failure; logname= uid=1000 euid=0 tty=pts/0 ruser=cfry rhost= user=root Apr 28 21:17:31 blanco-lnx su[9711]: pam_authenticate: Authentication failure Apr 28 21:17:31 blanco-lnx su[9711]: FAILED su for root by cfry Apr 28 21:17:31 blanco-lnx su[9711]: - pts/0 cfry:root Apr 28 21:17:36 blanco-lnx sudo(pam_unix)[9712]: authentication failure; logname= uid=0 euid=0 tty=pts/0 ruser= rhost= user=cfry Apr 28 21:17:45 blanco-lnx sudo: cfry : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/home/cfry ; USER=root ; COMMAND=/bin/su Apr 28 21:17:52 blanco-lnx su[9713]: Successful su for root by cfry Apr 28 21:17:52 blanco-lnx su[9713]: + pts/0 cfry:root Apr 28 21:17:52 blanco-lnx su(pam_unix)[9713]: session opened for user root by (uid=1000)

Perhaps user cfry is a sys adm in who had t he Caps Lock key act ivat ed t he first few t im es he t ried t o becom e root . Maybe cfry is a sys adm in who is violat ing process by not using t he sudo com m and t o becom e root . Or m aybe cfry is a t rainer at t he on- sit e gym and shouldn't even be logged in t o a shell session on a Linux server! The m ore cont ext you can add t o your m essages, t he m ore useful t he event source. I n t his next sect ion, we'll highlight t he power of Net Flow t o provide cont ext t o your event logs.

6 .4 . N e t Flow As we've assert ed t hroughout t his book, Net Flow is a free and powerful t ool, and should be one of t he m ost powerful t ools in your securit y t ool belt . Net Flow deploym ent is fairly sim ple. I f you have a large ent erprise, aggregat ing your dat a and perform ing queries across dat a set s can add com plexit y, but it can be script ed wit h som e sim ple Perl. Net Flow is an invaluable forensic invest igat ion t ool t hat you can query on dem and. The exam ples t hat follow are built on OSU flow- t ools, which we discussed in Chapt er 3 . Figure 6- 13 depict s a sim ple Net Flow collect ion in a sm all net work environm ent . Not ice t hat for ease of operat ion and separat ion of dat a, we're running t wo flow-cap ture inst ances on a single Net Flow collect or. The I SP gat eway rout er is configured t o export flows t o t he collect or on UDP port 2090 and t he dat a cent er gat eways are configured t o export on UDP 2095. The collect or has one process list ening on UDP 2090 for I SP gat eway flows, and t he ot her is list ening for t he DC gat eway flows. Figu r e 6 - 1 3 . Sim ple N e t Flow colle ct ion

This flow-capture com m and:

/usr/local/netflow/bin/flow-capture -w /var/local/flows/isp-data -E90G -V5 -A1134 0/0/2090 -S5 -p /var/run/netflow/isp-flow-capture.pid -N-1 -n288

cont ains several com m and- line flags, which will do t he following:

Capt ure I SP gat eway flows t o t he / var/ local/ flows/ isp- dat a direct ory ( m ake sure t he Net Flow user can writ e here! ) Begin rolling over once t here is 90 GB of flow dat a ( i.e., if you had a 95 GB drive) Collect and st ore dat a in Net Flow version 5 form at Tag flows wit h your AS num ber ( e.g., AS1134) List en and capt ure on UDP 2090 ( not e: you m ay have t o poke a hole in your firewall because flow collect ion is in your DMZ and your collect or is probably inside t he firewall) Log a stats m essage every five m inut es Specify t he process I D isp- flow- capt ure.pid in locat ion / var/ run/ net flow Set t he nest ing level t o YYYY- MM- DD/ flow- file ( see man flow-capture for opt ions) Creat e a flow file 288 t im es per day ( every five m inut es) This provides visibilit y int o ingress and egress t raffic at t he I SP int erconnect layer, so any net work t raffic com ing int o or leaving t he corporat e net work is logged. To cont inue wit h our exam ple in Figure 6- 13, we are collect ing dat a cent er gat eway flows on port 2095 t o a different dat a direct ory:

/usr/local/netflow/bin/flow-capture -w /var/local/flows/dc-data -E90G -V5 -A1134 0/0/2095 -S5 -p /var/run/netflow/dc-flow-capture.pid -N-1 -n288

This provides visibilit y int o ingress and egress t raffic at t he dat a cent er gat eway layer, so any net work t raffic com ing int o or leaving t he dat a cent er net works is logged. You can run queries against t hese t wo set ups by sim ply specifying t he appropriat e dat a direct ory.

6 .4 .1 . OSU flow - t ools N e t Flow Ca pt u r e Filt e r in g You m ay have a shared backbone rout er t hat com bines dat a cent er net works wit h ot her net works, inst ead of a dedicat ed dat a cent er gat eway. This doesn't prevent you from collect ing only dat a cent er Net Flow, should you not care t o log t he ot her t raffic. You can accom plish t his wit h t he OSU flow- t ools flow-capture com m and com bined wit h an opt ion t o use a filt er configurat ion file, filt er.cfg, locat ed in / usr/ local/ net flow/ var/ cfg. The following will allow for capt ure of t he 10.14.19.0/ 24 net work:

filter-primitive dc-address-mask type ip-address-mask permit 10.14.19.0 255.255.255.0 filter-definition dc-only match dc-address-mask Finally, t he com m and t o m ake use of t his capt ure filt er wit h flow-capture looks like t his:

/usr/local/netflow/bin/flow-capture -F dc-only -w /var/local/flows/dc-data -E90G -V5 -A1134 0/0/2095 -S5 -p /var/run/netflow/dc-flow-capture.pid -N-1 -n288

The preceding com m and will log all flows t hat m at ch t he filt er definit ion and will drop all of t he ot her unwant ed flow dat a.

6 .4 .2 . OSU flow - t ools flow - fa n ou t We oft en recom m end sending Net Flow t o any device t hat can use it . To accom plish t his, we can use t he flow- fanout t ool from t he flow- t ools suit e. Figure 6- 14 shows t he export of Net Flow from DMZ and dat a cent er rout ers t o separat e Net Flow collect ors. The flows are t hen copied t o t hree solut ions t hat can perform anom aly det ect ion and perform ance analysis on t he dat a: Arbor's Peakflow X, Cisco's CS- MARS, and Lancope's St ealt hWat ch. Figu r e 6 - 1 4 . Copyin g N e t Flow t o ot h e r de vice s w it h flow - fa n ou t

The flow-capture flags will be t he sam e as t he ones we used for t he exam ple depict ed in Figure 6- 13. To use flowfanout , we m ust " insert " it bet ween t he rout ers sending dat a and t he flow- capt ure process. This sends a copy t o localhost for logging and a copy t o t he ext ernal devices. The configurat ion changes a bit as follows:

/usr/local/netflow/bin/flow-fanout 0/0/2090 0/0/2089 10.1.1.4/10.1.1.7/2091 10.1.1.4/10.1.1.8/2092 You can int erpret t he preceding DMZ Net Flow collect or config as " t ake flows seen on UDP port 2090 and copy t hem t o localhost : 2089, CS- MARS: 2091, and Lancope: 2092" . Next , we need t o t ell t he collect or t o list en on 2089 and capt ure as appropriat e:

/usr/local/netflow/bin/flow-capture -w /var/local/flows/data -E90G -V5 -A1134 0/0/2089 -S5 -p /var/run/netflow/dflow-capture.pid -N-1 -n288

You can int erpret t he following DC Net Flow collect or config as " t ake flows seen on UDP port 2095 and copy t hem t o localhost : 2094, Arbor: 2096, and CS- MARS: 2097" .

/usr/local/netflow/bin/flow-fanout 0/0/2095 0/0/2094 10.1.1.5/10.1.1.6/2096 10.1.1.5/10.1.1.7/2097 Next , we m ust t ell t he collect or t o list en on 2094 and capt ure:

/usr/local/netflow/bin/flow-capture -w /var/local/flows/data -E90G -V5 -A1134 0/0/2094 -S5 -p /var/run/netflow/dflow-capture.pid -N-1 -n288

Now, from a single source of Net Flow, you can share Net Flow wit h as m any ot her devices as you want ! This is our preferred m et hod, as we have bot h t he raw flows kept for forensics and t he live feeds t o m ult iple t ools, each using t he dat a in different ways.

6 .5 . Bla n co's Se cu r it y Ale r t Sou r ce s Blanco Wireless has several sources for securit y alert s, including NI DS, syslog, applicat ion logs, dat abase logs, host ant ivirus logs, host NI DS logs, rout er and firewall logs, and Net Flow ( see Figure 6- 15) .

6 .5 .1 . N I D S Blanco Wireless has NI DS inst alled, and is inspect ing t he backbone uplinks on t he dat a cent er gat eway rout ers ( connect ions 1, 2, 3, and 4) . Analysis of t he net work t raffic shows an average of 12 Mbps wit h spikes of up t o 60 Mbps. Blanco will inst all a single Cisco I PS 4255 sensor wit h four int erfaces running I PS soft ware version 6.0. The rout ers are configured t o provide a copy of each int erface's bidirect ional t raffic. This set up provides a view of ingress/ egress dat a cent er t raffic, but does not capt ure int ra- dat a cent er t raffic, as is desired. Blanco's securit y t eam has t aken advant age of t he inform at ion gat hered in it s I nt ernet Prot ocol Address Managem ent ( I PAM) soft ware and m akes use of net work locale variables in t uning t he sensors ( not e t hat t he " slash not at ion" is broken down int o ranges of addresses t o represent subnet s) :

variables variables variables variables variables

DESKTOP_NETWORKS 10.10.32.0-10.10.63.255 WINDOWS_SERVERS 10.10.0.0-10.10.0.127 ORACLE_SERVERS 10.10.0.128-10.10.0.255 VMWARE_SERVERS 10.10.1.0-10.10.1.63 WEBAPP_SERVERS 10.10.1.64-10.10.1.127 Figu r e 6 - 1 5 . Bla n co W ir e le ss's se cu r it y a le r t sou r ce s

Because Blanco handles sensit ive cust om er inform at ion, it has writ t en a cust om NI DS signat ure t o wat ch for unencrypt ed Social Securit y num bers on t he wire:

signatures 60001 0 ! sig-description sig-name SSN_POLICY sig-string-info SSN HANDLING POLICY VIOLATION sig-comment CLEARTEXT SSN exit engine string-tcp event-action produce-verbose-alert specify-min-match-length no regex-string ([0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]) service-ports 1-65535 exit This signat ure will m at ch on regex for t he U.S. Social Securit y num ber form at # # # - # # - # # # # if it 's seen on any TCP port s.

6 .5 .2 . Syslog Blanco's Linux servers are configured t o send all aut hent icat ion event s, aut horizat ion event s, and daem on st at us event s t o syslog servers. Blanco's Windows 2000 servers have a Snare agent inst alled and configured t o forward event s in syslog form at , including aut hent icat ion and aut horizat ion.

6 .5 .3 . Apa ch e Logs Blanco's web applicat ions run on Apache Web Server. The syst em adm inist rat ors have configured Apache ( according t o guidance in O'Reilly's Apache Cookbook by Ken Coar and Rich Bowen) t o log errors and aut hent icat ion event s t o syslog. This was accom plished by configuring ht t pd.conf wit h t he following param et er:

ErrorLog syslog:user and wit h t his Perl script t o grab aut hent icat ion event s and send t hem t o syslog: Logging your access log t o syslog t akes a lit t le work. Add t he following t o your configurat ion file:

CustomLog |/usr/local/apache/bin/apache_syslog combined where apache_syslog is a program t hat looks like t he following:

#!/usr/bin/perl use Sys::Syslog qw( :DEFAULT setlogsock ); setlogsock('unix'); openlog('apache', 'cons', 'pid', 'user'); while ($log = ) { syslog('notice', $log); } closelog; This allows t he securit y t eam t o see when t he applicat ion st ops and rest art s as well as when users log in t o t he applicat ion.

6 .5 .4 . D a t a ba se Logs Blanco's dat abase adm inist rat ors have configured t heir Oracle 10g environm ent t o send syst em m essages and audit event s direct ly t o syslog. This includes dat abase st art and st op event s, as well as audit ing of all act ivit ies occurring wit h SYS and OPS$ORACLE account s. On Blanco's product ion dat abases, connect ions t o t hese account s should have an associat ed Change Request or not e from t he dat abase adm inist rat or m aking t he connect ion.

6 .5 .5 . An t ivir u s a n d H I D S Logs Blanco's em ployee deskt op and lapt op com put ers are configured t o send t heir ant ivirus logs and t heir host int rusion det ect ion logs t o a cent ralized collect or where t hey are st ored. Blanco has creat ed aut om at ed queries t o ident ify ant ivirus m essages wit h t he word Unresolved , as well as m essages from t he HI DS showing t hat a user has disabled t he ant ivirus or HI DS soft ware.

6 .5 .6 . N e t w or k D e vice Logs All of Blanco's net work devices are logging t o t he syslog server so t hat changes t o t heir configurat ions are recorded.

6 .5 .7 . N e t Flow As shown in Figure 3- 13 in Chapt er 3 as well as in Figure 6- 15, Blanco collect s Net Flow from it s DMZ and it s dat a cent er environm ent s. I n addit ion, Blanco runs aut om at ed report s t o search for policy violat ions, including indicat ions t hat dat a is being copied out of t he environm ent .

6 .6 . Con clu sion The heart of securit y m onit oring—configuring syst em s t o record, forward, and collect securit y event s—culm inat es t he preparat ion of all t he previous chapt ers. This chapt er provided guidance on how you can carefully configure syst em s t hat fit your infrast ruct ure, and t hen t une t hem so you can det ect t he real securit y event s. I n t he next chapt er, we'll explain how t o keep t hings hum m ing. Once you've gone t o all t his t rouble t o configure your event s, you don't want t hem t o go m issing, now do you?

Ch a pt e r 7 . M a in t a in D e pe n da ble Eve n t Sou r ce s I n " St alking t he Wily Hacker," Cliff St oll describes his invest igat ion of a m aj or securit y breach at Lawrence Berkeley Laborat ories. According t o t he account , t he hacker t ook pains t o avoid leaving t racks of his int rusion: " Whenever possible, he disabled account ing and audit t rails, so t here would be no t race of his presence...." [ 5 8 ] [ 58]

St oll, C. 1988. " St alking t he Wily Hacker." Com m unicat ions of t he ACM 31( 5) : p. 484.

Securit y event s such as user act ivit y logs, net work int rusion det ect ion syst em ( NI DS) alert s, server logs, and net work device records are indispensable foot print s, allowing securit y invest igat ors t o t race act ivit y and m onit or for problem s. Wit hout reliable event sources, m onit oring is a fut ile exercise—t here's no way t o discern lack of act ivit y from unrecorded act ivit y. Figure 7- 1 displays a rout er t raffic graph, generat ed by Mult i Rout er Traffic Grapher ( MRTG) t o show t raffic t hroughput for a series of net work int erfaces. I t illust rat es t raffic received by eight NI DSs in a product ion environm ent . There's clearly a problem wit h one of t he sensors—it 's no longer receiving any t raffic. Figu r e 7 - 1 . Rou t e r gr a ph ge n e r a t e d via M RTG, illu st r a t in g a dr a m a t ic dr op in t r a ffic for on e of t h e m on it or e d r ou t e r s

This m ay be a planned out age. Perhaps t he sensor was j ust placed int o service, or it s SPAN was disconnect ed from t he rout er for scheduled m aint enance. I t 's possible t here's a m alicious explanat ion for t his out age, but as is t rue wit h m uch of securit y m onit oring, t he explanat ion is likely m undane. Regardless of t he source for t his out age, you'll be blind t o securit y alert s while t he sensor is offline. What can you do t o cat ch t hese problem s, keeping downt im e t o a m inim um ?

Consider ot her gaps—syst em s or net work segm ent s of which you're not even aware. I f you have a relat ively large net work, it 's probably not const ant ; new net works and syst em s are being added m ont hly or even weekly. This illust rat es anot her blind spot : new areas of t he net work and new syst em s where securit y int rusions could go unnot iced; you probably aren't yet collect ing event s from t hese syst em s. Wit hout t hese event s, t here's not hing t o t rigger alert s in your m onit oring syst em . How can you prevent such gaps while wat ching for subst ant ive changes t o your environm ent ? To m aint ain reliable securit y m onit oring, you m ust not only deploy t he proper event sources, but also m onit or t he event sources t o cat ch int errupt ions. I n addit ion, you m ust keep wat ch for new syst em s deployed int o your environm ent t o ensure t hat event s are properly recorded. This chapt er will int roduce you t o t ools and processes t o keep securit y event s cont inually recorded. We'll describe how t o m aint ain proper device configurat ions, how t o keep an eye on your event feeds, and how t o m aint ain agreem ent s wit h t he support st aff.

7 .1 . M a in t a in D e vice Con figu r a t ion s One of t he m ost effect ive m eans for keeping event sources working reliably is t o m anage configurat ions on t he devices t hem selves. Recall t hat Chapt er 5 provided guidance for configuring securit y event feeds. For exam ple, a NI DS deployed passively requires a copy of t he net work t raffic via a rout er swit ch port analyzer ( SPAN) port , and a Unix server host ing an Apache web server should be configured t o log aut hent icat ion event s t o an event collect or via it s syslog.conf file. Consist ent event collect ion depends on specific configurat ion direct ives on t he source syst em s. I f a net work engineer rem oves a SPAN or a syst em adm inist rat or overwrit es t he syslog.conf set t ings, event feeds will suddenly st op. To m inim ize such errors, t his sect ion int roduces t he following t echniques:

Maint ain docum ent ed agreem ent s wit h device adm inist rat ors. Where possible, leverage an aut om at ed configurat ion syst em t o ensure t hat all syst em s m aint ain proper configurat ion for sending event s.

7 .1 .1 . Cr e a t e Se r vice Le ve l Agr e e m e n t s Device adm inist rat ors ( t hose who m aint ain servers, net work equipm ent , and dat abases) hold responsibilit y for keeping t hings running sm oot hly and securely on t he devices t hey support . They also hold superuser privileges on t hese devices, allowing t hem t o m ake changes t o configurat ions as t hey see fit . Consequent ly, t he event feeds upon which you depend will require t heir cooperat ion t o configure t hese devices and keep t hem funct ioning properly. You m ay have a st rong rapport wit h t hese adm inist rat ors, and believe you can depend on t hem t o keep your event feeds running sm oot hly. However, organizat ions change—a boss is prom ot ed, a colleague reassigned—and can disrupt your inform al agreem ent s. To ensure com m it m ent t o your goal, we recom m end you docum ent your configurat ion requirem ent s in form al agreem ent s. Such agreem ent s are best expressed in a cont ract called a service level agreem ent ( SLA) , form ed bet ween t wo part ies: you ( t he client part y) , and t he adm inist rat or's organizat ion ( t he service provider part y) . When building an SLA, be sure t o clearly represent t he following:

Rat ionale Why is t he agreem ent necessary? What value does it prot ect for t he business?

Specific, required act ions for each part y in t he agreem ent This is t he heart of t he SLA, t o docum ent what 's expect ed in a way t hat can be quant ified. I t 's not reasonable t o docum ent what 's required of only one part y. Each part y m ust bear som e responsibilit y.

Tim e period in which t he agreem ent will be in effect ( Alm ost ) not hing last s forever.

Signat ures of account able m anagers for each part y t o t he agreem ent These can be ink or elect ronic signat ures. The point is t o ensure t hat you've reached a final agreem ent so t hat you can put it in effect and hold each part y account able. You'll need a separat e SLA for each organizat ion you engage. I f your com pany is organized like m ost are, t his m eans you'll have one SLA per t ype of event source. Table 7- 1 out lines t he t ypes of configurat ion direct ives you'll want t o m aint ain via t he SLA and t he com m only used nam es for t he organizat ions you'll probably need t o engage. Ta ble 7 - 1 . Con figu r a t ion dir e ct ive s t o m a in t a in de pe n da ble e ve n t sou r ce s, a n d t h e t e a m s t o e n ga ge for SLAs

Eve n t sou r ce

I T or ga n iza t ion

NI DS alert s

Net work engineering/ adm inist rat ion

Con figu r a t ion su ppor t SPAN port s on rout ers and swit ches t hat feed NI DSs Net work flow logging from rout ers t o flow collect ors Select ed securit y event t ypes t o log Net work device logging t o event collect ors

Server event s ( syslog, et c.)

Syst em adm inist rat ion/ host ing Select ed securit y event t ypes t o log Log dest inat ion ( t o your event collect ors) OS support for net work flow collect ors OS support for event collect ors

Dat abase event s

Dat abase adm inist rat ion ( DBA) Select ed securit y event t ypes t o log Account s and privileges t o read securit y event s from audit t able or logfile Log dest inat ion ( t o your event collect ors)

Applicat ion event s ( web servers, app servers, et c.)

Applicat ion adm inist rat ion ( webm ast ers, et c.)

Select ed securit y event t ypes t o log Log dest inat ion ( t o your event collect ors) Add- on script s t o push event s t o your collect ors ( if necessary)

7 .1 .2 . Ba ck I t Up w it h Policy Just as Chapt er 2 described, policy is t he foundat ion for effect ive securit y m onit oring. Before you build an SLA, est ablish a device logging policy t hat docum ent s expect at ions for t he following:

Which devices m ust log event s ( which devices are " in scope" for t he policy) Which event t ypes m ust be logged Where event s m ust be logged ( a " collect ion server" for securit y event s) Who m ust have access t o such logs ( be sure t o include your securit y t eam m em bers) This policy provides t he foundat ion for building an SLA and m aint aining a configurat ion t em plat e, which we'll illust rat e in t he next sect ion.

7 .1 .3 . SLA Se ct ion s To build your SLA, include t he following sect ions: [ 5 9 ] [ 59]

UC Sant a Cruz I nform at ion Technology Services Services Agreem ent , 2008.

Service descript ion

This is where you should docum ent t he scope of t he service being offered. By scope, we m ean t o define t he bounds of t he service, indicat ing t hat you are com m it t ing t o only cert ain devices, cert ain part s of t he globe, and so fort h. For t he purpose of securit y event m onit oring, you should describe exact ly which devices are included in t he scope of support and what event feeds you are expect ing t o be m aint ained.

Roles and responsibilit ies This is where each part icipant is nam ed, bot h t he service provider and t he service client . For t he purpose of event m onit oring, t his should nam e t he group t hat m aint ains t he device and exact ly what is expect ed, including m aint enance of event feeds and configurat ion direct ives. You are t he client of such an agreem ent .

Request ing service This is where t he process for m aking request s is docum ent ed. For securit y event feeds, t his should docum ent how t he client ( you) is expect ed t o m ake request s of t he device owner, and how t he device owner is t o m ake you aware of service out ages or configurat ion changes.

Hours of operat ion This is where you should docum ent expect at ions regarding expect ed up t im e of event feeds and support st aff.

Response t im es This is where you should docum ent t he expect ed t urnaround t im es for report ed issues wit h event feeds.

Escalat ion Docum ent how cases will be escalat ed, including t o whom t hey will be escalat ed, when, and under what condit ions. This sect ion should also include inform at ion about how request s are priorit ized.

Maint enance and service changes Docum ent t he expect ed m aint enance schedule ( e.g., " every Sunday from m idnight t o 3: 00 a.m ." ) , as well as a process for com m unicat ing changes t o t he event feeds.

Pricing I f your com pany uses a chargeback syst em , or if an out side organizat ion m aint ains your event feeds, you m ay need t o docum ent t he pricing for various services in t his sect ion. This should include a process for how rat es are set , as well as a cat alog of charges t hat t he client should expect .

Report ing, reviewing, and audit ing Docum ent t he st art / end dat es for your agreem ent , and include a regular review process and owner. Not e t he dat es in t his sect ion for t he next review, t he previous review, and so on. This sect ion should also not e t he official elect ronic " hom e" of t his docum ent .

Associat ed policies and processes I t 's im port ant t o link t his agreem ent t o specific policies about event recording, configurat ion m anagem ent , and securit y. You'll find a sam ple SLA for support ing NI DS feeds in Appendix B.

7 .1 .4 . Au t om a t e d Con figu r a t ion M a n a ge m e n t SLAs are an effect ive m eans of m aint aining agreem ent s, but t hey're only as reliable as t he hum an beings who

m aint ain t hem . An im port ant supplem ent t o m aint ain sound configurat ion for collect ing securit y event s is t o m aint ain your device configurat ions wit h aut om at ed t ools. For exam ple, if you run Red Hat Ent erprise Linux ( RHEL) servers, you're probably already using t he Red Hat Net work ( RHN) t o download and apply package updat es. You also can use RHN t o push configurat ion files t o m anaged servers. Such a configurat ion service allows adm inist rat ors t o deploy packages based on a " t em plat e" for syst em configurat ions, ensuring t hat all m anaged syst em s have correct set t ings applied for firewall set t ings, logging, user aut hent icat ion, and so on. For Windows servers, Act ive Direct ory ( AD) dom ains allow server configurat ions t o be m aint ained and m anaged wit h Group Policy Obj ect s ( GPOs) . You can even develop your own t ools wit h a bit of shell script ing and som e SSH t rust s. For exam ple, t o m aint ain reliable securit y event s from your Unix servers, use a t em plat e like t he following one for syslog.conf, t o ensure t hat t he correct event s are logged and sent t o your log collect ion server ( in our exam ple, t he collect ion server is at 10.83.4.102) :

# Sample section of syslog.conf to enable remote logging local2.notice /var/log/sudolog local2.notice @10.83.4.102 local3.info /var/adm/ssh.log local3.info @10.83.4.102 local4.info /var/adm/tcpwrap.log local4.info @10.83.4.102 local7.info /var/adm/portsentry.log local7.info @10.83.4.102 *.err;kern.debug;daemon.notice;mail.crit @10.83.4.102 auth.notice /var/log/authlog auth.notice @@10.83.4.102 At Cisco, for exam ple, we use som et hing called aut oconfig t em plat es. This allows us t o deploy a set of approved t em plat es t o our net work devices. To ensure t hat we have t he securit y feeds we need, we could build a " securit y event t em plat e" t o m ake sure Net Flow and event logs are sent t o t he appropriat e event collect ors. A t em plat e for Net Flow feeds m ight look like t his:

# Setup NetFlow export to our NetFlow collector at 10.83.4.99 ip flow-export source Loopback0 ip flow-export version 5 ip flow-export destination 10.83.4.99 2055 A t em plat e for syslog event s would look like t his:

# Setup logging logging logging logging

logging for certain events to our event log collector at 10.83.4.102 facility auth facility daemon trap informational host 10.83.4.102

I f you wish t o use off- t he- shelf product s t o aut om at e t he configurat ion of your net work devices you could use Cisco Net work Com pliance Manager, or Cisco Securit y Manager which m anages configurat ion of securit y devices.

7 .2 . M on it or t h e M on it or s I f you've worked in inform at ion securit y for long, you've not iced a phrase em erge over t he past few years: " wat ching t he wat chers" . The idea is t o keep an eye on your privileged users—t he ones m aint aining t he syst em s for com m on users. We m ust apply t he sam e concept here: t o m ake sure t he syst em s used for conduct ing securit y m onit oring are reliably m aint ained. Wit hout t hat assurance, crit ical securit y event s m ay be lost or deliberat ely suppressed. To effect ively m onit or t he healt h of your m onit oring syst em , you m ust first set a benchm ark for " norm al" act ivit y. As Æleen Frisch writ es in Essent ial Syst em Adm inist rat ion: " As wit h m ost of life, perform ance t uning is m uch harder when you have t o guess what norm al is. I f you don't know what t he various syst em perform ance m et rics usually show when perform ance is accept able, it will be very hard t o figure out what is wrong when perform ance degrades. Accordingly, it is essent ial t o do rout ine syst em m onit oring and t o m aint ain records of perform ance- relat ed st at ist ics over t im e."[ 6 0 ] [ 60]

Frisch, Æleen. 2002. Essent ial Syst em Adm inist rat ion, Third Edit ion. Sebast opol, CA: O'Reilly Media, I nc.

You'll use t his benchm ark t o com pare recent act ivit y and det erm ine whet her anyt hing has changed, and whet her t hat change requires at t ent ion. For exam ple, if you know t hat your NI DS norm ally sust ains 400 Mbps of t raffic, a sudden, sust ained drop t o 5 Mbps should cause you t o t ake not e and look for problem s on t he net work or t he devices. I f you norm ally receive 150 m essages per hour from your Windows dom ain cont roller, you'll want t o be not ified if t he rat e has dropped t o 0 in t he past 60 m inut es. I n t his sect ion, we'll discuss how t o keep your m onit oring syst em s running sm oot hly by t racking t he syst em healt h ( CPU, m em ory, disk, et c.) , as well as each event feed. We'll discuss each t ype of syst em ( NI DS, server, and dat abase) , along wit h specific checks for each t ype of syst em , t o keep t hem report ing reliably.

7 .2 .1 . M on it or Syst e m H e a lt h The syst em s t hat generat e and feed event s necessary for securit y m onit oring m ust be m onit ored t hem selves; if t hey're not funct ioning properly, t hey can't cont inue sending event s. Syst em load, disk space, and built - in hardware sensors should be observed t o provide early warning of present or im m inent failure. To m onit or overall syst em healt h, observe t he following syst em indicat ors, m easuring t hem against your " norm al" benchm ark . 7 .2 .1 .1 . M on it or syst e m loa d Take snapshot s of syst em load at regular int ervals ( spaced a few seconds apart ) t o t rend and wat ch for spikes. This includes snapshot s of CPU perform ance such as what you can see wit h uptime, procinfo, or w:

[mnystrom@blanco-rich-1 ~]$ uptime 07:25:11 up 32 days, 17:04, 1 user,

load average: 0.01, 0.02, 0.00

I t t urns out t hat different t ypes of syst em s com put e t his num ber different ly. Linux does not t ake int o account t he num ber of processes wait ing for t he CPU, whereas HP does. Therefore, you'll want t o underst and how t he num ber is calculat ed before you set alert ing t hresholds. However, it 's fair t o say t hat if t he syst em is running norm ally when t he load is less t han 1, seeing it spike t o 4 or 5 indicat es t hat som et hing has drast ically changed, and you'll want t o look int o what 's causing t he spike. 7 .2 .1 .2 . M on it or m e m or y Wat ch m em ory t o ensure t hat you aren't exhaust ing all of your free or swap m em ory. I t 's good t o check m em ory ut ilizat ion against what 's " norm al," but som e processes grow over t im e. At t he very least , check t o m ake sure you don't run out of physical or swap m em ory. On a Unix m achine, you can check t his wit h com m ands such as top, free , and meminfo. 7 .2 .1 .3 . M on it or disk spa ce Wat ch available disk space t o ensure t hat t here's adequat e room t o st ore collect ed event s. Keep an eye on space ut ilizat ion in t he part it ion where you're collect ing dat a, and where t he syst em m ust keep disk space available t o funct ion properly. You can check available disk space on a Unix server wit h t he com m and df. Beyond j ust having disk space available, you'll also want t o ensure t hat t he space is writ able. 7 .2 .1 .4 . M on it or n e t w or k pe r for m a n ce Check net work perform ance and t hroughput t o ensure t hat securit y event s aren't overwhelm ing your net work

int erface. You can check t his on Unix ( as root ) , wit h t he ifconfig com m and:

bash-3.00$ sudo ifconfig Password: eth0 Link encap:Ethernet HWaddr 00:1E:0B:34:9F:D7 inet addr:10.83.4.102 Bcast:10.83.4.255 Mask:255.255.255.0 inet6 addr: fe80::21e:bff:fe34:9ee7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:20892745 errors:0 dropped:0 overruns:0 frame:0 TX packets:12713946 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2754879057 (2.5 GiB) TX bytes:1151366228 (1.0 GiB) Interrupt:209 Verify t hat you're not seeing errors on t he int erface by inspect ing t he errors sect ion of t he out put . To cont inually m onit or your syst em s, you should program t hese com m ands t o run at frequent int ervals, benchm arking t hem against norm al values. We'll describe an approach for accom plishing t his lat er in Sect ion 7.4 . Now t hat we've discussed how t o m onit or t he healt h of your syst em s, we'll address how you should m onit or each t ype of device t o ensure a cont inual st ream of event s.

7 .2 .2 . M on it or t h e N I D S When funct ioning properly, a NI DS is one of t he best event sources for securit y m onit oring. To funct ion properly, a NI DS m ust have bidirect ional visibilit y t o t he net work segm ent s it m onit ors; it m ust be able t o fully inspect every single packet on t he wire. To m aint ain your NI DS so t hat you have an unint errupt ed st ream of event s, you m ust be sure t hat a num ber of areas are working properly: m onit or t he t raffic feeds, sensor processes, and recent alert s generat ed t o ensure t hat t he NI DS is perform ing as expect ed. 7 .2 .2 .1 . M on it or t r a ffic fe e ds ( u plin k s) To ensure t hat you're get t ing t raffic t o your NI DS, wat ch t he port s feeding int o it . I f your NI DS is configured t o receive t raffic via a SPAN port , wat ch t he SPANs t o ensure t hat t hey're st ill point ed t o your NI DS. ( Occasionally a net work engineer will " st eal" t he SPAN port t o t roubleshoot ot her problem s. She m ay forget t o reset t he configurat ion for your NI DS.) Alt ernat ively, if you've deployed your NI DS in t he line of t raffic ( inline) , wat ch t he t raffic rat es t o ensure t hat not hing has been int errupt ed. I t will be obvious if your NI DS is no longer receiving t raffic, as all downst ream t raffic will also be int errupt ed. N OTE

MRTG is an excellent open source t ool for wat ching t raffic volum es. I t uses Sim ple Net work Managem ent Prot ocol ( SNMP) for st at us updat es on t he m onit ored device. I f MRTG shows a sudden drop in net work t raffic for your NI DS, it 's likely t hat you've lost a t raffic feed. To ensure t hat your rout ers are st ill configured t o send t raffic t o your NI DS, you m ust wat ch t he SPAN configurat ion on t he rout ers t hem selves. I f you analyze t he configurat ion of t he m onit or sessions on t he rout er, you should see t hat it rem ains configured t o m irror t raffic t o your NI DS. The best way t o ensure t hat it 's working properly is t o docum ent t he out put of t he working configurat ion, and com pare it t o t he running configurat ion at regular int ervals. For exam ple, here's t he out put of show monitor on an I OS 12.2 rout er:

blanco-dc-gw1>show monitor Session 1 - --------Type : Local Session Source Ports : Both : Gi1/1 Destination Ports : Gi3/11

Session 2 - --------Type Source Ports Both Destination Ports

: Local Session : : Gi1/2 : Gi3/12

I n t his exam ple, a NI DS is connect ed t o port Gi3/ 11 on Session 1 and Gi3/ 12 on Session 2. 7 .2 .2 .2 . M on it or se n sor pr oce sse s Even if t raffic is reaching your NI DS, you m ust be sure it can process t he event s. The NI DS will have som e processes responsible for receiving and analyzing event s, which m ust be kept running t o cont inue analyzing t raffic and raising alert s. You should wat ch t he sensor processes t o be sure t hey're running norm ally. For exam ple, on a Cisco NI DS, t he Analysis Engine m ust be running for t he sensor t o process event s. I n a Snort deploym ent , it 's t he snort process t hat 's crit ical. You m ay wish t o add furt her checks t o ensure t hat t he required processes are not in a " zom bie" st at e, and t hat t hey're get t ing sufficient CPU t im e. 7 .2 .2 .3 . M on it or a le r t s I f your rout er is sending t raffic t o t he NI DS, and t he processes are running norm ally, everyt hing is probably working fine. One furt her check t hat you can add is t o ensure t hat alert s are being generat ed. To do so, wat ch t he alert s com ing from t he NI DS t o ensure t hat you've received at least one alert wit hin t he past few m inut es. On m any net works, a properly t uned device will st ill generat e event s very frequent ly, so if you've not seen a single event wit hin t he past 30 m inut es, som et hing is definit ely wrong.

7 .2 .3 . M on it or N e t w or k Flow Colle ct ion As we described in previous chapt ers, Cisco Net Flow is a popular open st andard for recording net work flows. We recom m end capt uring such flows for analysis, for m onit oring, and as a secondary source t o support invest igat ions. To ensure t hat your net work flows are flowing, m onit or t he collect ion servers t o m ake sure you're get t ing t he am ount of dat a you expect from t he places you expect . You should m onit or t he syst em used t o collect t he flows, t he processes t hat collect and st ore t he flows, t he net work t raffic along t he segm ent s in your net work, and t he rout ers sending Net Flow t o your collect ion syst em . N OTE We recom m end collect ing Net Flow from rout ers t hat act as choke point s on your net work. Typically, such choke point s are deployed where you're already doing t raffic filt ering, such as t he corporat e I nt ernet connect ion, dat a cent er gat eways, and gat eways t o specialized net works for part ners or labs. 7 .2 .3 .1 . M on it or syst e m h e a lt h Most im port ant ly, t he net work flow collect or m ust it self be wat ched t o ensure t hat it can receive, st ore, and relay flows properly. This requires " healt h m onit oring," as we expounded earlier in t his sect ion. Healt h m onit oring checks t hat t he collect or can keep up wit h incom ing dat a, and t hat t he disk space and perm issions are properly configured so t hat capt ured flows can be properly st ored. 7 .2 .3 .2 . M on it or t r a ffic fe e ds fr om r ou t e r s There's m ore t han one way t o m onit or for flows com ing in from each rout er you're expect ing. I t 's m ost im port ant t o wat ch t hat t he sending rout ers are configured t o send flows t o your collect or. On each rout er, verify t hat t he ip flow export com m and is properly configured. For exam ple, here's t he Net Flow configurat ion on a Cisco I OS 12.2 rout er. I t 's configured t o export Net Flow v5 t o our collect or at 10.83.4.99 on port 2055.

blanco-dc-gw1>show ip flow export Flow export v5 is enabled for main cache Exporting flows to 10.83.4.99 (2055) Exporting using source interface Loopback0 Version 5 flow records, peer-as 327844689 flows exported in 10928453 udp datagrams -- output clipped for brevity --

You should also confirm t hat Net Flow is being export ed from t he proper int erface ( in t he preceding exam ple, it 's export ing via t he Loopback0 int erface) . I f you check t his again in a few seconds, you should see t he num ber of export ed flows increasing, indicat ing t hat it 's st ill export ing properly. 7 .2 .3 .3 . M on it or colle ct or n e t w or k con figu r a t ion Now t hat you've checked t he configurat ion from t he rout ers, you should also check t he collect or t o be sure it 's list ening t o receive flows, and t hat t hey're being received. Assum ing t he collect or is running Unix, sim ply use netstat t o check t hat t he collect or is list ening for Net Flow t raffic on t he proper port by " grepping" for t he correct list ening port :

[mnystrom@blanco-dc-nfc-1 ~]$ netstat -l | grep 2055 udp 0 0 *:2055 *:* You'll also want t o see whet her you're receiving t raffic from each rout er. One t echnique is t o creat e iptables rules on t he collect or it self, adding an ent ry for each rout er from which you expect t o receive Net Flow. For exam ple, if you want t o m onit or t he am ount of t raffic you've received from t wo gat eway rout ers, add ACCEPT rules for each rout er t o the iptables INPUT chain:

[root@blanco-dc-nfc-1 ~]# iptables -vL INPUT Chain INPUT (policy ACCEPT 9875 packets, 14M bytes) pkts bytes target prot opt in out source destination -A INPUT -s 10.83.4.1 -p udp -m udp --dport 2055 -j ACCEPT -A INPUT -s 10.83.4.2 -p udp -m udp --dport 2055 -j ACCEPT ACCEPT COMMIT N OTE A t ypical use for iptables is t o filt er t raffic by including a DROP st at em ent som ewhere in t he chain, t o dict at e t hat any t raffic not explicit ly allowed should be blocked. Because you're using iptables only t o keep count of t raffic from each source, you don't need t o include a DROP rule in t he configurat ion file unless you want t o filt er t raffic. The iptables chain in t he preceding code is " accept ing" connect ions from gat eways 10.83.4.1 and .2 ( corresponding t o blanco- dc- gw1 and gw2, respect ively) . You can periodically run iptables -VL INPUT, t o see how m any byt es and packet s have arrived from each accept ed host in t he chain. Sim ply com pare t hose num bers wit h t he ones from a few m inut es ago t o prove t hat you're get t ing t raffic from each source:

[root@blanco-dc-nfc-1 ~]# iptables -vL INPUT Chain INPUT (policy ACCEPT 9875 packets, 14M bytes) pkts bytes target prot opt in out source destination 159 230K ACCEPT udp -- any any blanco-dc-gw1 anywhere udp dpt:2055 4692 6776K ACCEPT udp -- any any blanco-dc-gw2 anywhere udp dpt:2055 The preceding out put proves t hat you've received 159 packet s and 230,000 byt es since t he last flush of t he count ers. Once you've checked t hese num bers, flush t he count ers so t hat you can see whet her you're accum ulat ing flows at t he next read. You can do t his easily wit h t he " zero count ers" opt ion on t he iptables com m and ( indicat ed in t he following code by t he –Z opt ion) :

[root@blanco-dc-nfc-1 ~]# iptables -vxL -Z INPUT Chain INPUT (policy ACCEPT 273 packets, 130896 bytes) pkts bytes target prot opt in out source destination 43 64156 ACCEPT udp -- any any blanco-dc-gw1 anywhere udp dpt:2055 11 16412 ACCEPT udp -- any any blanco-dc-gw2 anywhere udp dpt:2055 Zeroing chain 'INPUT' 7 .2 .3 .4 . M on it or colle ct ion dir e ct or ie s To prevent a problem where rout ers are sending Net Flow but you're not able t o st ore it , check t he direct ories where Net Flow is st ored t o be sure perm issions are set properly. For exam ple, t o check t he / apps/ net flow direct ory on a Unix box, check t hat t he write bit is properly set for t he user writ ing t he flows:

[mnystrom@blanco-dc-nfc-1 data]$ ls -l /apps/netflow total 12 drwxr-xr-x 243 netflow infosec 12288 Jul 28 09:41 data I n t he preceding exam ple, you can see t hat t he netflow user ( t he owner of t he file) has full perm issions on t he data file, so t he file syst em is properly configured. This is a good place t o verify t hat dat afiles have been writ t en t o t he collect ion direct ory wit hin t he past few m inut es. 7 .2 .3 .5 . M on it or colle ct ion pr oce sse s Once you've verified t hat t he syst em can receive flows and t hat dat a is com ing from t he rout ers, you should ensure t hat t he processes used for collect ing and st oring ( and, if applicable, relaying) Net Flow are running, and t hat t hey haven't failed or gone int o a zom bie st at e. To check processes, run a com m and on t he operat ing syst em t o see whet her all t he correct processes are running. Of course, you'll first need t o know which processes should be running, and how m any of t hem .

[mnystrom@blanco-dc-nfc-1 ~]$ ps -ef | grep netflow netflow 29982 1 0 2007 ? 1-09:34:16 /usr/local/netflow/bin/flowcapture -w /apps/netflow/data -E54G -V5 0/0/2056 -S5 -p /var/run/netflow/flowcapture.pid -N-1 -n288 netflow 29999 1 0 2007 ? 05:32:35 /usr/local/netflow/bin/flow-fanout 0/0/2055 0/0/2056 10.71.18.5/10.71.18.6/2055

I n t he preceding exam ple, one flow- capt ure and one flow- fanout process is running. Wit h OSU flow- t ools, t hese processes are used t o receive and st ore dat a t o t he local syst em , and relay t hem out t o ot her syst em s. 7 .2 .3 .6 . M a in t a in flow r e t e n t ion Many collect ion t ools provide a m eans for keeping st orage wit hin t he bounds of available disk space by rot at ing out older dat a. Wit h OSU flow- t ools, t he flow- capt ure process allows you t o set t he expire size or expire count for flows. These set t ings allow you t o set a lim it on how m any flows you want t o keep, or how m uch space you want t o use for st oring flows. Assum ing, for exam ple, t hat you want t o keep as m uch dat a as possible wit hin 1 TB of disk space, you could use t he opt ion -E 1024G on flow- capt ure. This opt ion t ells flow- capt ure t o t hrow away t he oldest flows in order t o keep wit hin t he 1 TB ( 1,024 GB) disk space lim it . The knowledge t hat you have a full t ool chest is useless wit hout t he accom panying knowledge of which t ools are cont ained wit hin. Likewise, you'll want t o know j ust how far back your Net Flow records go, so you can support necessary invest igat ions and m onit oring. I f your collect ion direct ories are nam ed by dat e, your j ob is an easy query against direct ory nam es. I f not , sim ply finding t he dat e of t he oldest direct ory in your collect ion should show you how far back you can query Net Flow records.

W a t ch for N e w Syst e m s I n a large ent erprise, it 's difficult t o keep abreast of t he new syst em s deployed. I f rout ing changes, syst em s are m oved, or new syst em s are added t o provide load balancing, your m onit oring will be blind t o t hose event s unt il you add t heir event s t o your syst em s. The net work engineering t eam , for exam ple, m ay deploy an addit ional rout er t o handle load on t he net work, bring up a new dat a cent er, or deploy a new I nt ernet Point of Presence ( PoP) . The syst em adm inist rat ion t eam m ay replace a Linux server wit h a virt ual m achine, or t he DBAs m ay load- balance t he dat abase inst ance across m ult iple servers. Discovering t hese infrast ruct ure changes is crit ical and com plex. Here are a few ideas for uncovering t hese changes. Change m anagem ent . Most ent erprises have a m eans of docum ent ing im port ant changes t o t he environm ent . This is oft en in t he form of change request s t hat are t hen com m unicat ed, reviewed, and approved by appropriat e t eam s and represent at ives. More sophist icat ed syst em s even allow users t o regist er int erest in part icular host s or devices, queuing not ificat ion when changes are scheduled. This can be a useful m eans of not ing sim ple changes t o t he environm ent , such as configuring a rout er so t hat it peers wit h a new rout er. Such changes can serve as t riggers for you t o reach out t o t he responsible t eam so t hat you can get your securit y event feeds configured. Net work scans. Net work scanning t ools discover t he devices present on net work segm ent s, oft en including addit ional probes t o det erm ine how t he devices are configured. These scans can t urn up new devices t hat m ight ot herwise be m issed, allowing you t o cat ch new syst em s and det erm ine whet her t hey should be in t he scope of your securit y m onit oring. Use net work scans t o wat ch for new rout ers and syst em s appearing, especially wit hin subnet ranges t hat correspond t o t he ranges t arget ed for m onit oring. Your com pany will probably use cent rally m anaged scanning syst em s, but here's an exam ple of what you can discover wit h a sim ple ping scan from Nm ap:

thor$ nmap -sP 10.3.1.0/24 Starting Nmap 4.50 ( http://insecure.org ) at 2008-08-09 17:07 EDT Host corp-dc-gw1.blanco.com (10.3.1.1) appears to be up. Host webserver (10.3.1.18) appears to be up. Host webserver-bkup (10.3.1.19) appears to be up. Host new-webserver (10.3.1.20) appears to be up. Nmap done: 256 IP addresses (7 hosts up) scanned in 1.589 seconds This scan looked for host s on t he 10.3.1.0/ 24 net work segm ent . Assum ing t his net work segm ent is in range of your t arget ed m onit oring, you can keep a list of known host s on t his segm ent , wat ching for new ones appearing. Once you see new host s, det erm ine what inform at ion you m ight need t o collect from t hem t o m aint ain full coverage for securit y m onit oring. Relat ionships. I n " Parent s: The Ant i- Drug," a series of public service announcem ent s, parent s are encouraged t o spend t im e t alking t o t heir t eenagers t o det ect signs of drug abuse. The point is t hat personal conversat ions are clearly t he m ost reliable way t o uncover inform at ion. As painful as t his analogy sounds ( t alking t o your net work adm inist rat ors about capacit y planning is not t he sam e as t alking t o your kids about drugs) , if you hope t o det ect im port ant changes t o an environm ent , you should regularly engage wit h I T st aff. This will help you uncover what 's changing and what t hey're planning for t he environm ent . At Cisco, a st rong relat ionship wit h I T Net working has helped I nfoSec discover new PoPs under deploym ent , new dat a cent ers, and im port ant organizat ional changes.

7 .2 .4 . M on it or Eve n t Log Colle ct or s Like net work flow collect ors, event log collect ors run soft ware for gat hering and forwarding event logs. To keep event s flowing t o securit y m onit oring syst em s, and t o ensure t hat downst ream syst em s are not kept wait ing for event s rout ed t hrough event log collect ors, m onit or syst em healt h as well as t he healt h of collect ion and relay processes. Most im port ant ly, you'll need t o wat ch t he event flow com ing from originat ing devices t o prevent event gaps in your m onit oring. 7 .2 .4 .1 . M on it or syst e m h e a lt h

Monit or syst em healt h using t he sam e approach used for Net Flow collect ors. Verify t hat t he collect or is funct ioning properly and has enough m em ory, t hat load is sufficient for processing, and t hat t he disk can handle m ore incom ing dat a. 7 .2 .4 .2 . M on it or colle ct ion pr oce sse s For event collect ion t o work properly, t he syst em processes used for receiving and st oring event s m ust run const ant ly, list ening for relayed event s. For exam ple, when using syslog-ng for collect ing event logs, t he syslog-ng process m ust be list ening for relayed syslog, norm ally via t he st andard syslog port of UDP/ 514. To check whet her t he process is running and list ening on t he correct port , use netstat t o check for running UDP list eners and what processes are bound t o t hose port s. You can do t his wit h t he -a opt ion t o show all connect ions, along wit h t he -p opt ion t o show program s bound t o port s:

[mnystrom@syslog-blanco-1 ~]$ sudo netstat -ap | grep syslog tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 1813/syslog-ng udp 0 0 0.0.0.0:514 0.0.0.0:* 1813/syslog-ng udp 0 0 10.83.4.102:32851 10.71.180.220:514 ESTABLISHED 1813/syslog-ng udp 0 0 10.83.4.102:32873 10.70.149.28:516 ESTABLISHED 1813/syslog-ng unix 10 [ ] DGRAM 4663 1800/syslogd /dev/log N OTE You m ust run t his com m and as a privileged user ( root on Unix) , t o see t he st at us of port s < 1024. Based on t he out put from t he preceding netstat -ap com m and, we can see t hat syslog-ng is list ening for syslog, as we expect . syslog-ng can handle list ening for incom ing syslog as well as st oring it and relaying it t o ot her servers. This is accom plished wit hin one process, so t here's no need t o m onit or ot her processes on t his host . Alt ernat ive event collect ion syst em s m ay use m ore processes in collect ing and st oring event s. For exam ple, Splunk, a com m ercial flexible analysis t ool for event s, requires processes for receiving event s, helper processes, a wat chdog process, and a web server.

7 .2 .4 .3 . M on it or colle ct ion dir e ct or ie s ( logs) You m ust wat ch t he syst em s t hat are originat ing event s t o t he log collect or t o ensure t hat t hey're st ill sending dat a. There are a num ber of ways t o oversee t hese event st ream s t o ensure t hat t hey're st ill flowing, but t he m ost painless m et hod is t o wat ch t he growing logfiles, checking for recent event s from each syst em . Begin wit h a list of host nam es from which you're expect ing event s, and t hen check t he event logs for recent event s from such syst em s. Count any recent event as a " heart beat " from t he source syst em , indicat ing t hat it 's st ill sending event s. There's no need t o evaluat e t he t ype of m essage t hat arrived from t he source syst em ; it s recent t im est am p is enough t o indicat e t hat t he event feed is funct ioning. To check for a recent m essage from a source, search t hrough t he direct ory of collect ed logs. This m ay be a sim ple exercise if t he direct ory is organized by event source, but if you're using syslog-ng , you'll have t o slog t hrough t he m ass of dat a t o find t he recent m essages. Let 's say it 's August 9 at 2: 59 p.m ., and you want t o check for a m essage from t he webserver- 1 server wit hin t he past hour. Using egrep ( ext ended grep) on a Unix syst em , you can const ruct t his search t o find som et hing wit hin t he hour of 14: 00, across all logfiles in t he current direct ory:

egrep "Aug

9 14:.*webserver-1" /apps/logs/all-*

The egrep com m and t ells t he Unix server t o run a regular expression m at ch against t he files. The .* bet ween t he dat e and t he server nam e is a regular expression t elling egrep t o ignore everyt hing bet ween t he dat e form at and t he server nam e. Even a single m essage wit hin t he past hour is an indicat or t hat t he server is running and sending event s t o t he collect or, as we expect . N OTE This works only if t he sending server and t he event collect or are t im e- synchronized. I f t hey're not , none of t he m onit oring in t his chapt er or book will work for you, as you can't correlat e event s. To synchronize t im e on your servers, use Net work Tim e Prot ocol and a com m on t im e offset ( we suggest GMT) . 7 .2 .4 .4 . M on it or n e t w or k t r a ffic

Wat ch t he net work int erface card ( NI C) for m essages arriving from t he source syst em . The set up works as we described in Sect ion 7.2.3 . Creat e an iptables configurat ion wit h an ACCEPT rule for each source server. Monit or t he accum ulat ed packet s for each rule in iptables, flushing each t im e you check t he t able. 7 .2 .4 .5 . Au dit con figu r a t ion s Technically, audit ing isn't act ive m onit oring, but t he servers originat ing event s t o your collect or require a configurat ion t hat will capt ure t he correct event s and relay t hem t o t he proper collect or. To m aint ain t he correct configurat ion on syst em s, deploy an audit ing solut ion t hat can check it s logging configurat ion against a reference t em plat e. Check for t he following t wo configurat ion direct ives in t he syst em 's log configurat ion:

I s t he relay set t o t he I P address of your event collect or? Are proper event t ypes being logged? A sim ple script t o access t he source server and grep t he syslog.conf m ight include a check such as t his:

$ ssh webserver-1 grep 10.83.4.102 /etc/syslog.conf *.* @10.83.4.102 This checks t he webserver- 1 server for t he presence of t he 10.83.4.102 collect ion server in t he syslog.conf file, and finds it in t he response. Script s t hat rem ot ely check configurat ion values require set up of public key aut hent icat ion t o t he t arget server prior t o execut ion. Take precaut ions t o prot ect t he privat e keys, and never perm it connect ions t o product ion servers from nonproduct ion host s, as t hose keys are less prot ect ed . 7 .2 .4 .6 . M a in t a in log r e t e n t ion Just as we described in Sect ion 7.2.3 , it 's vit al t o m ake sure your event collect ion doesn't overrun t he available disk space. You can m onit or t his by wat ching available disk space ( covered in " Monit or syst em healt h" ) , but you'll need t o proact ively rot at e your collect ed logs, expiring t he oldest cont ent t o m ake room for new event s. This feat ure is not nat ive t o syslog-ng , so you'll need com plem ent ary t ools t o accom plish it . The m ost com m on pairing is logrot at e, a freely available Unix ut ilit y aut hored by Erik Troan of Red Hat . This ut ilit y is scheduled t o run as a daily cron, archiving logfiles by rot at ing, com pressing, and delet ing t hem . [ 6 1 ] [ 61]

See ht t p: / / linuxcom m and.org/ m an_pages/ logrot at e8.ht m l for m ore inform at ion.

7 .3 . M on it or D a t a ba se s Dat abases can be crit ical sources of securit y event s, especially on syst em s t hat st ore sensit ive dat a. To ensure t hat t hey keep a cont inuous flow of m essages, m onit or t he processes t hat record event s in t he dat abase, and m onit or t he st orage locat ions for t he event s, t o m ake sure new m essages are showing up. This sect ion explores t echniques for m onit oring dat abase set t ings for audit ing and logging.

7 .3 .1 . M on it or Or a cle You can configure Oracle t o record a wide variet y of event s. To m ake cert ain Oracle is properly capt uring an audit t rail, you m ust ensure t hat event s are being logged properly and t hat audit set t ings are properly set . As an addit ional precaut ion, configure t he dat abase t o capt ure adm inist rat ive event s t o audit DBA act ivit y. 7 .3 .1 .1 . M a in t a in Or a cle syst e m w ide a u dit se t t in gs To enable event logging in Oracle ( Oracle calls it " audit ing" ) , you m ust set t he audit_trail param et er in t he init .ora file. The dat abase references t his core configurat ion file at st art up, and specifies whet her t o log event s t o t he Oracle dat abase ( audit_trail = db or audit_trail = true) or t o a logfile ( audit_trail = os) on t he host file syst em . As an added st ep, rout inely check t he configurat ion files t o be sure t he Oracle dat abase rem ains properly configured for logging. From t he dat abase, you can ensure t hat syst em wide audit ing is t urned on by checking t he v$parameter t able:

select NAME, VALUE from V$PARAMETER where NAME like 'audit%'; NAME VALUE -----------------------------audit_trail DB audit_file_dest ?/rdbms/audit Here you can see t hat t he adm inist rat or has configured init .ora wit h audit_trail = db, and t hat audit dat a will be writ t en t o t he dat abase's audit area ( SYS.AUD$) . 7 .3 .1 .2 . M on it or Or a cle a u dit e ve n t s Should you decide t o collect event s t o t he file syst em , it 's easy t o relay t hose logs int o your event log collect ion server. However, if you want Oracle t o record event s t o t he SYS.AUD$ t able, you m ust keep wat ch over t hat t able t o confirm t hat new event s are arriving regularly. For exam ple, here's a sim ple query t o check for new event s arriving every five m inut es:

select count(*) from SYS.AUD$ where (current_timestamp - aud$.timestamp#) > 5*1/24/60 I f t he query ret urns a count great er t han zero, you've confirm ed t hat som e event s have been received wit hin t he past five m inut es. To be sure you've received audit event s of a cert ain t ype ( for a specific t able, colum n, user, et c.) , sim ply add crit eria t o t he where clause in t he preceding query. 7 .3 .1 .3 . M a in t a in Or a cle a u dit se t t in gs on obj e ct s I f event s are capt ured in t he SYS.AUD$ t able ( or t he file syst em , depending on how Oracle is configured) , you can have confidence t hat t he syst em is configured t o properly log event s. However, because configurat ion changes can disrupt t he flow of event s, it 's useful t o m onit or Oracle's audit set t ings t o be sure you're capt uring audit event s. When you configure audit ing on a t able, Oracle st ores t he set t ing in m et adat a. The DBA_OBJ_AUDIT_OPTS view list s t hese set t ings, cross- referencing all obj ect s wit h audit set t ings for every possible st at em ent . You should regularly query t his view t o verify t hat audit set t ings are correct . For exam ple, t o ensure t hat audit ing is st ill enabled for successful and unsuccessful queries against t he CUSTOMERS t able ( where Social Securit y num bers are st ored) , issue t he query:

select SEL from dba_obj_audit_opts where object_name = 'CUSTOMERS';

I f it 's properly configured for audit ing, it should ret urn:

SEL --A/A This shows t hat audit ing is enabled for bot h successful and unsuccessful SELECT st at em ent s against t hat t able. 7 .3 .1 .4 . M on it or a dm in ist r a t ive pr ivile ge s I t 's considered a best pract ice t o ensure t hat event logs are im m ut able so t hat a m alicious user cannot cover her t racks by m odifying t hem . Because a DBA has superuser privileges, she will always be able t o m odify t he SYS.AUD$ t able. Two sim ple configurat ion direct ives will record DBA act ivit y and provide a record of any at t em pt t o rem ove dat a from t he audit ing t able. First , set a direct ive t o capt ure all changes t o t he SYS.AUD$ t able. Connect as SYSDBA and m odify t he SYS.AUD$ t able wit h t his st at em ent : [ 6 2 ] [ 62]

Loney, K., and B. Bryla. 2005. Oracle Dat abase 10g DBA Handbook. Oracle Press.

audit ALL on SYS.AUD$ by access; This will ensure t hat every operat ion against t he SYS.AUD$ t able—including delet e and t runcat e—are insert ed int o t he SYS.AUD$ t able as audit ent ries. I f a DBA t ries t o cover her t racks by delet ing ent ries from t he t able, t hose act ions will be logged int o t he t able it self! You should also provide an off- dat abase log of DBA act ivit y. To do so, t urn on file- based audit ing of DBA act ions by adding t his configurat ion direct ive t o init .ora: [ 6 3 ] [ 63]

Oracle Corporat ion. 2002. Oracle9i Dat abase Adm inist rat or's Guide.

AUDIT_SYS_OPERATIONS = TRUE This will configure Oracle t o log all SYS operat ions ( including SYSDBA and SYSOPER) t o t he operat ing syst em logfile, which, if t he syst em is configured correct ly, is off- lim it s t o t he DBAs.

7 .3 .2 . M on it or M ySQL Se r ve r s The current version of MySQL ( 5.x) has lim it ed opt ions t o enable audit ing. SQL query st at em ent s are logged in t he " general query" logfile. This is enabled if you specify t he --log opt ion when st art ing t he dat abase. To ensure t hat m essages are st ill being writ t en t o t he general query log, sim ply wat ch recent ent ries t o t he m ysqld.log query logfile, which shows recent event s:

080228 15:27:50

1170 1170 1170 1170

Connect user@host on database_name Query SET NAMES "utf8" Query SELECT something FROM sometable WHERE some=thing Quit

Wat ching t his file allows you t o verify t hat queries are st ill being writ t en t o t he logfile.

Use Ca n a r y Eve n t s t o M on it or Sou r ce s Here's an addit ional t ool t o m onit or event sources: generat e " canary" event s and wat ch for t hem downst ream . You can use a canary event like a heart beat t o verify t hat event s are being capt ured, relayed, and collect ed properly. For each event source described in t his chapt er, we've out lined an exam ple of how t o generat e and find canary event s. Not e t hat for t hese illust rat ions, coalm ine is a server used for execut ing script s t o generat e canary event s. Net Flow. Pick a rout er sending Net Flow t o your collect or. Now pick a server wit h an I nt ernet Prot ocol ( I P) address on t he ot her side of t hat rout er ( bet ween coalm ine and t he I P you're t est ing) . Run a script on coalm ine t o ping t hat I P address at a regular int erval. On t he Net Flow collect or, regularly query t he st ored Net Flow records t o look for t he canary event . The query will be very specific and efficient , as it 's searching for an exact m at ch on source and dest inat ion I P address, along wit h t he exact prot ocol ( I CMP) . Here's an exam ple using flow- cat and flow- filt er, which are part of OSU flow- t ools. [ 6 4 ] I n t his exam ple, coalm ine uses I P 10.10.0.12. We're building a query in t he flow.acl file, searching t he Net Flow direct ory using flow- filt er t o find only t he canary event s:

[blanco-nfc]$ cat flow.acl ip access-list standard canary-flow permit host 10.10.0.128 host 172.17.54.31 [blanco-nfc]$ flow-cat /apps/netflow/data/2008-09-18/ft* | flow-filter -r1 -Scanary-flow NI DS. Find a host t hat requires t raversal by a rout er m onit ored by each NI DS, and generat e an alert by script ing a ping from coalm ine t o t hat host 's I P. On t he NI DS, build and deploy a cust om signat ure t o det ect your canary connect ion. Now, your m onit oring t ools can wat ch for t he canary alert downst ream ( in t he Securit y I nform at ion Manager [ SI M] or wherever your NI DS alert s are collect ed) . Host Logs. On every m onit ored host , deploy a script t hat generat es a canary event every few m inut es. On t he event collect or, query for t he canary event from each host at a regular int erval. Dat abase Audit Event s. Pick a t able t o m onit or. Deploy a script t o query t he t arget t able every few m inut es, using a well- prot ect ed t est account . Monit or t he audit log for queries by t he t est user against t he specific t able.

[ 6 4 ] Nat ional Chi Nan Universit y. OSU flow- t ools docum ent s; ht t p: / / 163.22.3.90/ flow- doc/ ( accessed Sept em ber 18, 2008) .

7 .4 . Au t om a t e d Syst e m M on it or in g So far, t his chapt er has illust rat ed m anual t echniques t o m onit or and m aint ain event feeds for NI DS, event collect ors, and dat abases. However, aut om at ion is t he key t o sust aining a cont inuous st ream of event s for securit y m onit oring. I n t his sect ion, we will exam ine a few com m ercial and open source t ools available for net work and syst em m onit oring, applying t hem t o keep dependable event st ream s for securit y m onit oring.

7 .4 .1 . Tr a dit ion a l N e t w or k M on it or in g a n d M a n a ge m e n t Syst e m s Net work m onit oring syst em s are t ypes of net work m anagem ent syst em s. Monit oring is conduct ed by agent s running on " end st at ions" ( t he syst em s being m onit ored) . These end st at ions are observed by m anagem ent ent it ies t hat are program m ed t o respond t o t he agent s. [ 6 5 ] These t ools are designed t o m onit or syst em healt h, t hough new funct ions are const ant ly being int roduced int o such soft ware. [ 65]

Cast elli, M.J. 2001. Net work Consult ant s Handbook . I ndianapolis: Cisco Press.

Net work m onit oring has been around for decades, not ably in t he form of HP OpenView Net work Node Manager ( NNM) , which uses SNMP ( along wit h ot her t echnologies) t o m aint ain visibilit y int o syst em healt h. OpenView present s a dashboard overview of all m anaged devices, using varying colors t o illust rat e t heir healt h and event st at us. Com pet it ors include I BM Tivoli Monit oring, CA Unicent er TNG, and Microsoft SMS. I t 's hard t o pin a nam e on such product s because m any of t hem handle provisioning, healt h m onit oring, invent ory, and configurat ion. Such t ools have been used t o m anage host operat ing syst em s, net work devices, applicat ion soft ware, and dat abases. I n recent years, a num ber of free open source product s have m at ured t o t he point t hat t hey are now viable alt ernat ives t o com m ercial net work m onit oring syst em s, including such t ools as Nagios, OpenNMS, Pandora, and Zenoss. N OTE

As t he syst em s we've described here offer m uch m ore t han net work device m onit oring, it 's hard t o call t hem " net work m onit oring" solut ions. We couldn't find an accept ed t erm t hat describes m ore t han t hat , so we'll j ust call t hem " syst em m onit oring" solut ions for t he durat ion of t his chapt er. 7 .4 .1 .1 . H ow syst e m m on it or in g w or k s Syst em m onit oring solut ions m aint ain a dashboard t hat allows an " at - a- glance" view of device st at us. They t rack t he healt h and event st at us of each syst em using a variet y of input s, and cont ain t he following elem ent s:

Agent s t hat assess t he healt h of m anaged devices ( via SNMP, script s, local agent s, et c.) A st at e ret ent ion dat abase t o t rack t he healt h inform at ion recorded for each syst em A m onit oring console, which provides an overview of up- t o- t he- m inut e device status Agent s can run on t he m onit ored device, such as an SNMP agent report ing t o a syst em m onit or or an inst alled agent . Alt ernat ively, t he cent ral m onit oring server can probe t he device t o evaluat e out put and record t he healt h of t he com ponent . Figure 7- 2 depict s a t radit ional net work m onit oring syst em . Figu r e 7 - 2 . A t r a dit ion a l n e t w or k m on it or in g syst e m sh ow in g a da sh boa r d r e pr e se n t in g t h e a va ila bilit y of e a ch de vice

The agent s t hat run in a syst em m onit oring solut ion check t he healt h of a com ponent or end st at ion. They t ypically run a script cont aining com m ands m uch like t he ones we've described in t his chapt er. For exam ple, an agent could SSH int o a server and run a com m and, evaluat ing t he out put t o det erm ine whet her t he syst em is perform ing wit hin prespecified boundaries. Alt ernat ively, t he agent m ay run on t he m onit ored device it self, t riggering an SNMP t rap when CPU ut ilizat ion crosses an upper t hreshold. To leverage such a syst em for your t arget ed m onit oring event sources, you should configure it t o m onit or t he follow ing:

Syst em healt h of every device report ing event s St at us of accum ulat ing event s from each sending device St at us of accum ulat ing event s on event receivers Healt h of each event collect or Healt h of each com ponent in det ect ion syst em s The st at us of each it em should be represent ed on t he net work m onit or dashboard, allowing you t o observe " at a glance" t hose areas t hat require adm inist rat ive at t ent ion.

7 .4 .2 . H ow t o M on it or t h e M on it or s You've likely seen m onit oring syst em s displaying a dashboard on large screens in offices and dat a cent ers. What value, however, is such a display wit hout act ion? Som eone m ust be assigned responsibilit y t o not only view t he alert s, but also—and m ore im port ant ly—t ake act ion t o address t he problem s highlight ed on t he dashboard. Monit oring t he st at us of securit y event feeds is a t ask best assigned t o t he t eam already conduct ing securit y m onit oring. These st aff m em bers depend on event feeds from t he m onit ored syst em s, and are in t he best posit ion t o not ice when t hey go offline. Many of t he syst em m onit oring solut ions provide built - in case handling and not at ion capabilit ies, allowing st aff t o

t rack t he st at us of alarm s. To use t hese effect ively, est ablish m onit oring procedures t hat include at t ent ion t o t he syst em m onit oring solut ion, wit h docum ent ed, assigned responsibilit ies. Good case handling procedures will allow st aff t o see t he st at us of such alarm s wit hout m ist akenly t roubleshoot ing t he sam e problem s, duplicat ing t heir effort .

7 .4 .3 . M on it or in g w it h N a gios Nagios, form erly known as Net Saint , is a feat ure- rich syst em m onit oring package. I t displays current inform at ion about syst em or resource st at us across an ent ire net work. I n addit ion, you can configure it t o send alert s and perform rem edial act ions when problem s are det ect ed. [ 6 6 ] [ 6 6 ] Frisch, Æleen. " Top Five Open Source Packages for Syst em Adm inist rat ors, Part 4." O'Reilly ONLam p.com , Decem ber 5, 2002; ht t p: / / www.onlam p.com / pub/ a/ onlam p/ 2002/ 12/ 05/ essent ialsysadm in.ht m l ( accessed August 25, 2008) .

Figure 7- 3 is a screenshot of Nagios's dashboard, giving an overview of t he syst em s being m onit ored, along wit h an indicat ion of t he st at us for each device. Figu r e 7 - 3 . Th e N a gios da sh boa r d sh ow in g t h e st a t u s of gr ou ps of de vice s, lin k in g t o a de t a ile d st a t u s for e a ch

Nagios list s an overall st at us for host s and services, along wit h a sum m ary for each group of servers. When you click on a specific host , Nagios displays a det ailed st at us page for t he host ( shown in Figure 7- 4 ) . This det ailed st at us page specifies m onit ored aspect s, provides a set of available com m ands for m onit oring t he host , and provides an int erface t o m ake not e of issues. Figu r e 7 - 4 . D e t a ile d de vice st a t u s displa ye d in N a gios

Nagios norm ally inst alls script s on t he syst em t o be m onit ored, allowing t he cent ral server t o execut e script s via a " plug- in" fram ework t hat report s st at us back t o t he Nagios server. These script s check t he st at us of each m onit ored elem ent wit hin a set of " warning" and " crit ical" boundaries set for t he elem ent . For exam ple, t o m onit or syst em m em ory, a Nagios plug- in allows t he adm inist rat or t o set bot h a warning and a crit ical alert , passed as param et ers t o t he script when it st art s. Here's synt ax t o st art a plug- in t hat checks m em ory, ret urning a warning if free physical m em ory is below 10% of available m em ory and raising a crit ical alert if it 's below 5% :

check_mem -w 10 -c 5 Most plug- ins are execut ed from t he Nagios server via NRPE ( rem ot e procedure execut ion) or SSH. For cases where a syst em cannot be act ively polled for it s st at us, Nagios can accom m odat e a passive check, which allows a script on t he m onit ored syst em t o send it s st at us direct ly t o t he Nagios server. I n addit ion, on som e of t he syst em s you'll need t o m onit or, you will be unable t o inst all any plug- in. For t hose cases, Nagios can run Expect script s on t he m onit ored syst em , connect ing via SSH from t he Nagios server it self. The next sect ion will illust rat e how you can use Nagios t o aut om at e syst em and net work m onit oring for Blanco Wireless.

7 .5 . Syst e m M on it or in g for Bla n co W ir e le ss Blanco Wireless has configured t arget ed m onit oring for it s account m anagem ent syst em . To m onit or t he syst em , Blanco is leveraging t he following event sources:

Net Flow collect ion at t he dat a cent er and DMZ gat eways Syslog collect ion from servers NI DS Dat abase event logs from t he Oracle 10g dat abases Using Nagios, Blanco can aut om at e syst em m onit oring t o ensure high availabilit y for securit y m onit oring ( see Figur e 7- 5) . N OTE

Nagios set up and configurat ion is det ailed on t he Nagios websit e at ht t p: / / www.nagios.org/ . Plug- ins and script s are list ed at ht t p: / / www.NagiosExchange.org/ and ht t p: / / www.NagiosPlugins.org/ . Figu r e 7 - 5 . Au t om a t in g syst e m m on it or in g u sin g N a gios t o scr ipt ch e ck s for syst e m h e a lt h a n d con t in u ou s e ve n t flow

7 .5 .1 . M on it or N e t Flow Colle ct ion The Net Flow collect ors for Blanco are running OSU flow- t ools for collect ing and querying Net Flow records. To ensure cont inuous flow collect ion from crit ical sources ( t he DC and DMZ gat eway rout ers sending flows t o t he collect ors) , Nagios plug- ins are configured t o observe t he following:

Collect ors are cont inuously receiving flows from bot h dat a cent er and DMZ rout ers. Collect ors are in good healt h, so t hey're able t o receive and st ore flows properly. Collect ion processes are running and funct ioning properly. For Nagios t o m onit or t he collect ors, Blanco has deployed t he NRPE daem on t o run direct ly on t he collect ors. Blanco has configured t he plug- ins t o m onit or syst em healt h and collect or processes, and t o check t hat dat a is st ill com ing from t he rout ers configured t o send flows for collect ion.

7 .5 .2 . M on it or Colle ct or H e a lt h To effect ively m onit or collect or healt h, Blanco has configured t he following series of Nagios plug- ins. 7 .5 .2 .1 . D isk spa ce

Check t o be sure t here's adequat e disk space by running t he check_disk plug- in t o wat ch t he collect ion direct ory ( / apps/ net flow/ dat a) , warning if less t han 10% is available and sending a crit ical alert if less t han 5% is available:

check_disk -w 10 -c 5 -p /apps/netflow/data 7 .5 .2 .2 . Pe r m ission s Ensure t hat t he direct ory st oring flows rem ains writ able for incom ing flows:

check_diskrw /apps/netflow/data 7 .5 .2 .3 . Loa d Evaluat e whet her t he syst em load is excessive. Warn if t he load exceeds 5, 10, or 15, and raise a crit ical alert if it exceeds 20, 25, or 30:

check_load -w 15,10,5 -c 30,25,20 7 .5 .2 .4 . M e m or y Verify t hat a reasonable am ount of physical m em ory is st ill available on t he syst em . Warn if t here's less t han 5% free m em ory, and raise a crit ical alert if t here's less t han 1% :

check_mem -w 5 -c 1 7 .5 .2 .5 . Sw a p spa ce Ensure t hat t here's a reasonable am ount of swap space on t he box, warning if t here's less t han 90% , and raising a crit ical alert when it dips below 80% :

check_swap -w 90%% -c 80%% 7 .5 .3 . M on it or Colle ct ion Pr oce sse s To m ake sure t he soft ware is cont inuously collect ing and st oring flows, wat ch t he file syst em and processes t o m ake sure t hey're doing t heir j ob. To accom plish t his, wat ch t he following. 7 .5 .3 .1 . Con t in u ou s flow s Configure t he Nagios check_lat est file plug- in t o wat ch t he / apps/ net flow/ dat a collect ion direct ory. Set t he script t o warn if t he m ost recent file is m ore t han 30 m inut es old. This script does not accept param et ers via t he com m and line, so you m ust edit t he script t o set t he required param et ers. 7 .5 .3 .2 . Pr oce sse s Ensure t hat t he correct num ber of collect ion and relay processes is running. First , m ake sure t he flow-capture process is running, and t hat it raises a crit ical alert if t here isn't at least one, or if t here is m ore t han five running:

check_procs -c 1:5 -C flow-capture The param et ers described here specify only an upper lim it because t he synt ax requires us t o do so. I f you're running flow-fanout, add anot her check_procs t o wat ch t hat process.

7 .5 .4 . M on it or Flow s fr om Ga t e w a y Rou t e r s To ensure cont inuous flow collect ion, verify t hat each rout er is sending Net Flow t o t he collect or as expect ed. Blanco could have script ed a configurat ion check on each rout er from which Net Flow is expect ed. I nst ead, it chose t o wat ch t he net work t raffic com ing int o t he collect or t o ensure t hat t he t raffic is received from each expect ed rout er. There are no Nagios plug- ins t o accom plish t his, so Blanco has developed a cust om script t o wat ch net work t raffic via iptables. The script will read accum ulat ed t raffic received in an int erval, drop t he result s int o a file, and periodically push t he file t o t he Nagios server via it s NSCA fram ework. Because Blanco is sending DMZ flows t o one collect or and DC flows t o anot her, t he configurat ions for m onit oring will be ident ical. Here, Blanco has built an iptables rule script , using an " accept " rule for each rout er expect ed t o send

Net Flow. I t 's easy t o see t he t wo rout ers and t he port where t hey're sending flows—UDP/ 2055, in t he iptables configur at ion:

-A INPUT -s dc-gw-1 -p udp -m udp --dport 2055 -j ACCEPT -A INPUT -s dc-gw-2 -p udp -m udp --dport 2055 -j ACCEPT To det erm ine whet her t raffic is com ing from t hese gat eways on t he expect ed port s ( per t he iptables configurat ion) , look at t he accum ulat ed packet s from each rule on t he line next t o each rout er. Com pare t hat t o t he values from t he last check t o det erm ine whet her m ore packet s have arrived. Use t he -nvL opt ions on t he iptables com m and and observe t he accum ulat ed packet s for each gat eway.

root@nfc ~]# iptables -nvL INPUT Chain INPUT (policy ACCEPT 125K packets, 180M pkts bytes target prot opt in out source 21986 32M ACCEPT udp -* * dc-gw-1 50059 72M ACCEPT udp -* * dc-gw-2

bytes) destination 0.0.0.0/0 udp dpt:2055 0.0.0.0/0 udp dpt:2055

As described earlier in " Monit or collect or net work configurat ion," t he script will zero t he count ers aft er each check t o m ake com parison easier.

7 .5 .5 . M on it or Eve n t Log Colle ct ion The event log collect ors for Blanco are running syslog-ng . These collect ors are receiving syslog from Unix servers, Windows servers, and t he Apache web servers. Blanco m ust m onit or t o ensure t hat t he event logs are in good healt h, and t hat event logs are being collect ed and st ored. Blanco has configured m onit oring for t he following:

Collect ors are cont inuously receiving event s from all expect ed servers. Collect ors are in good healt h, able t o receive and st ore event s properly. Collect ion processes are running and funct ioning properly. Using t he NRPE plug- ins, Nagios will execut e com m ands on t he syslog collect or and process t he result s on t he cent ral server. 7 .5 .5 .1 . M on it or colle ct or h e a lt h Blanco will use t he sam e t echniques for m onit oring collect or healt h t hat it used for m onit oring t he Net Flow collect or. This will involve wat ching load, m em ory, and disk t o ensure t hat t he syst em is funct ioning norm ally and can receive and st ore event logs. Configurat ion for checking load, m em ory, and swap will rem ain exact ly t he sam e, but we'll need t o m onit or t he specific file syst em direct ories for syslog collect ion on t his syst em . 7 .5 .5 .2 . Ve r ify disk spa ce Check t o verify t hat t here's adequat e disk space by running t he check_disk plug- in and wat ching t he collect ion direct ory ( / apps/ logs) . Warn if t here's less t han 20% available, and send a crit ical alert if t here's less t han 10% :

check_disk -w 20 -c 10 -p /apps/logs 7 .5 .5 .3 . En su r e pe r m ission s Ensure t hat t he direct ory st oring logs rem ains writ able so t hat t he event s can be st ored as t hey arrive:

check_diskrw /apps/logs 7 .5 .5 .4 . M on it or colle ct ion pr oce sse s Make sure t he syslog-ng process is running, raising a crit ical alert if t here isn't exact ly one process running:

check_procs -c 1:1 -C syslog-ng 7 .5 .5 .5 . M a in t a in con t in u ou s logs

Configure t he Nagios check_lat est file plug- in t o wat ch t he / apps/ logs collect ion direct ory. Set t he script t o warn if t he m ost recent file is m ore t han 30 m inut es old. 7 .5 .5 .6 . M on it or colle ct ion fr om se r ve r s When st oring logs t o t he file syst em , syslog-ng collat es t he logs from all sources int o a single file during collect ion. This m akes it difficult t o discern whet her a recent ly writ t en file cont ains event s from every source. To m onit or for event s from each of Blanco's m onit ored servers, Blanco used t he sam e approach t aken wit h Net Flow m onit oring: using iptables t o wat ch for t raffic volum e. The iptables configurat ion file will look sim ilar t o t he one used t o wat ch for Net Flow:

-A -A -A -A -A

INPUT INPUT INPUT INPUT INPUT

-s -s -s -s -s

webserver-1 webserver-2 appserver-1 appserver-2 dbserver -p

-p udp -p udp -p udp -p udp udp -m

-m udp --dport 514 -m udp --dport 514 -m udp --dport 514 -m udp --dport 514 udp --dport 514 -j

-j ACCEPT -j ACCEPT -j ACCEPT -j ACCEPT ACCEPT

A cust om script regularly checks t he accum ulat ed dat a from each server via t he iptables com m and, subsequent ly zeroing t he count ers in preparat ion for t he next check.

7 .5 .6 . M on it or N I D S To ensure t hat Blanco receives a cont inuous st ream of NI DS alert s, Nagios m onit ors NI DS device healt h, t raffic feeds, sensor processes, and alert s from t he NI DS. Because Blanco is using t he Cisco NI DS, it 's not possible t o run script s direct ly on t he sensors. All Nagios m onit oring is t herefore done using Expect script s on t he Nagios server, connect ing t o t he sensors via SSH. The sensor m ust explicit ly perm it connect ions from t he Nagios server for t his t o work. This is configured via an access list on t he sensor.

7 .5 .6 .1 . M on it or de vice h e a lt h Blanco m onit ors t he healt h of it s sensors using SNMP and Expect script s execut ed via SSH. To m onit or t he CPU, Blanco uses a built - in Nagios plug- in called check_snm p, which queries t he sensor via SNMP, sending a warning if CPU usage is above 75% . The -H param et er specifies t he sensor nam e ( ids-1) , and t he -o param et er requires t he SNMP obj ect ident ifier corresponding t o a check for t he CPU value of t he sensor:

check_snmp -H ids-1 -o "1.3.6.1.4.1.9.9.109.1.1.1.1.8.1" -w 75 To m onit or available m em ory and disk space, Blanco execut es an Expect script via SSH. The script connect s t o t he sensor and parses t he out put of t he show version com m and:

ids-1# show version Application Partition: Cisco Intrusion Prevention System, Version 6.1(1)E2 -- output clipped for brevity -Using 1901985792 out of 4100345856 bytes of available memory (46% usage) system is using 17.7M out of 29.0M bytes of available disk space (61% usage) boot is using 40.6M out of 69.5M bytes of available disk space (62% usage) -- output clipped for brevity --

The script parses t he out put , raising a crit ical alert t o Nagios if m em ory or disk space is consum ed beyond 90% . I n t he preceding out put , all values are safely below t his t hreshold. 7 .5 .6 .2 . M on it or t r a ffic fe e ds Because Blanco's sensors are receiving feeds direct ly from gat eway rout ers, script s are used t o periodically verify t hat SPANs are properly configured. This confirm s t hat t he rout ers are properly enabled t o m irror t raffic t o t he sensor. Using Expect , Blanco script ed a regular com parison of t he show monitor session all com m and wit h a st ored baseline. This is best accom plished by st oring an MD5 hash of t he com m and out put and subsequent ly com paring it t o t he new value upon each connect . The script raises a Nagios alert if t he hashes don't m at ch. Blanco also m onit ors t he percent age of m issed packet s. I f t he value is higher t han 10% , it can indicat e t hat t he net work has grown beyond t he sensor's capacit y t o m onit or t raffic. Using t he show interface com m and, t he script parses t he percent age of m issed packet s from t he out put :

ids-1# show interface Interface Statistics Total Packets Received = 109049288548 Total Bytes Received = 67536504907892 Missed Packet Percentage = 2 Current Bypass Mode = Auto_off -- output clipped for brevity -Not e t he value Missed Packet Percentage = 2. Because t his value is less t han 10, t he script will not raise an alert t o Nagios ( recall t hat t he script should alert only if t he percent age m issed is great er t han 10% ) . 7 .5 .6 .3 . Ch e ck se n sor pr oce sse s Most crit ically, Blanco want s t o ensure t hat t he AnalysisEngine and MainApp processes are running on it s CS- I PS 6.1 sensor. I f t hey're not running, t he script raises a crit ical alert t o Nagios. Using t he show version com m and, t he result is parsed t o find an indicat ion t hat t hese processes are running:

ids-1# show version Application Partition: Cisco Intrusion Prevention System, Version 6.1(1)E2 -- output clipped for brevity -MainApp M-2008_APR_24_19_16 (Release) 2008-04-24T19:49:05-0500 AnalysisEngine ME-2008_JUN_05_18_26 (Release) 2008-06-05T18:55:02-0500 CLI M-2008_APR_24_19_16 (Release) 2008-04-24T19:49:05-0500 -- output clipped for brevity --

Running Running

The preceding out put shows t he Running keyword on t he line wit h each of t he t wo crit ical processes: MainApp and AnalysisEngine . I f t he Running keyword is not present , t he script raises a crit ical alert in Nagios. 7 .5 .6 .4 . M on it or a le r t ge n e r a t ion Last ly, Blanco want s t o m onit or t he sensors t o ensure t hat t hey are st ill generat ing alert s. To check for recent alert s, t he script checks t he alert dest inat ion—t he Securit y I nform at ion Manager ( SI M) , where alert s are reviewed by t he m onit oring st aff. Blanco's SI M uses an Oracle dat abase t o st ore t he alert s for processing, so t he com pany uses a script t o query t he pert inent t able looking for alert s generat ed wit hin t he past five m inut es. I f no alert s have been logged wit hin t he past five m inut es, t he script raises a crit ical alert t o Nagios via t he NSCA fram ework.

7 .5 .7 . M on it or Or a cle Loggin g Because Oracle is configured t o send event s via syslog t o t he log collect or, t here's no need t o direct ly m onit or t he dat abases. Rat her, t his inform at ion is logged via syslog direct ly t o t he event collect ors.

7 .5 .8 . M on it or An t ivir u s/ H I D S Loggin g Ant ivirus and host int rusion det ect ion syst em ( HI DS) logs are collect ed t o t heir respect ive servers using t he nat ive facilit ies of t he ant ivirus and HI DS soft ware. Monit oring collect ion on t hose servers is ident ical t o t he t echniques used for m onit oring t he syslog collect ion server.

7 .6 . Con clu sion To t his point , you've developed t he policies upon which you will base your securit y m onit oring, and you've m apped your net work and infrast ruct ure. Using net work m et adat a as a backdrop for m onit oring, you've select ed m onit oring t arget s, chosen event sources, and fed t hem int o your m onit oring syst em s. This chapt er aim ed t o professionalize your m onit oring, prevent ing gaps t hat could allow an int rusion t o succeed wit hout not ice. Wit h t hese finishing t ouches in place, you've enabled m onit oring for your syst em s wit h t he confidence t hat event s can be collect ed r eliably .

Ch a pt e r 8 . Con clu sion : Ke e pin g I t Re a l For t he past t wo years, I 've been t eaching m y t eenagers t o drive. As m y pat ience st ret ched, I began t o wonder whet her I could sim plify t hings by writ ing down t he st eps for t hem t o em ploy. Once I had docum ent ed t he st eps, m y younger kids could event ually use t hem —m aybe I could even sell t he publishing right s! The st eps would look som et hing like t his:

Verify t hat all t ires are properly inflat ed t o wit hin 35–37 PSI . Check for t raffic on t he st reet —do not proceed if t here is a vehicle in m ot ion wit hin 750 feet . Open driver door. Be seat ed in t he driver's seat . Adj ust seat so t hat your t orso is no m ore t han 18 inches from t he wheel and your feet can easily work t he foot pedals. Adj ust rear- view m irror so t hat t he rear window is cent ered in your view. Place your hands on t he st eering wheel at t he 10: 00 and 2: 00 posit ions. Verify t hat you know t he locat ion of headlight s, windshield wipers, brake, and accelerat or. Place key in ignit ion and st art car. You get t he idea: t his is t edious. Experienced drivers don't follow every st ep. Sure, on som e occasions I plan m ore carefully, such as when leaving for a long j ourney. Most of t he t im e, however, I skip m ost of t hese st eps, such as when heading off t o work. Som e st eps are execut ed subconsciously—I 've done t hem for so long t hat I don't even t hink about t hem —whereas ot hers I perform only occasionally. Here's t he point : t he aut hors have a confession t o m ake. We rarely follow t he det ails of our always goes wrong, and we find ourselves speeding t hrough t he set up so t hat we can m eet alm ost never able t o define policies or t ake t im e t o choose t he right event sources. Usually, com e up wit h a funding est im at e, so we t ake an educat ed guess at our requirem ent s, writ e m onit oring playbooks, and begin ordering hardware.

own advice. Som et hing a deadline. We're we have t wo weeks t o up som e brief

I n t his final chapt er, we'll give exam ples where m onit oring ideals haven't always aligned wit h pract ical experience, including t he consequences of t hose expediencies. We'll share t he result s of t wo case st udies, including how t he organizat ions deployed t arget ed m onit oring. We'll conclude by st ripping down t he advice of t his book t o barem inim um t asks for each st ep, leaving you wit h a checklist t o st art your own t arget ed m onit oring.

8 .1 . W h a t Ca n Go W r on g We don't always follow our own advice. This sect ion gives exam ples of t eam s t hat failed t o im plem ent a st ep of our m onit oring approach, including t he consequences t hat followed. Som e of t hese exam ples are our very own, whereas ot hers are t he anecdot es of fellow incident response professionals. I n every case, we've changed t he nam es t o avoid disclosing confident ial det ails, and t o avoid em barrassing any individuals.

8 .1 .1 . Cr e a t e Policy Here are t wo st ories t hat illust rat e t he im port ance of t he first and m ost basic requirem ent : build and clarify policies before you begin m onit oring. 8 .1 .1 .1 . Rya n m on it or s t h e r isk y ve n t u r e Ansalya Net works acquired t echnology t hat allowed m obile phone users t o access t he services of t heir deskt op VoI P phones over t he m obile carrier's net work. Ansalya rapidly int egrat ed t he t echnology and pilot ed t he product for cust om ers, deploying a proxy server t o provide access from t he carrier's servers t o Ansalya's VoI P infrast ruct ure, carried via t he public I nt ernet ( see Figure 8- 1) . Because t he product t eam could not properly secure t he proxy wit hout archit ect ural changes and long delays, Ansalya's chief securit y officer t asked Ryan, Ansalya's m onit oring lead, t o address t he com pany's securit y concerns by m onit oring t he pilot syst em s. Because t he pilot was deployed before policies could be discussed, no rules were docum ent ed t o discern securit y incident s from t he vast num ber of generat ed securit y event s in t his environm ent . This lack of docum ent ed policies left Ryan wit h few opt ions, so he focused t he t eam 's effort s on t he following available event sources:

St andard, high- fidelit y signat ures on t he exist ing perim et er NI DS Net Flow records highlight ing I nt ernet access originat ing from t he proxy servers Figu r e 8 - 1 . A VoI P pr ox y w h ich pr ovide d a ga t e w a y be t w e e n t h e m obile ca r r ie r 's n e t w or k a n d t h e VoI P se r ve r , bu t pr e se n t e d r isk t o An sa lya 's n e t w or k

As alert s were generat ed, t he m onit oring t eam analyzed and escalat ed incident s t o t he pilot support st aff, but never received replies t o any of t heir m essages. Occasionally, t raffic was int erest ing enough t o invest igat e furt her—NI DS alert s of at t acks against t he proxy and Net Flow records showing suspicious t raffic from t he proxy. The pilot support st aff, however, received t he alert s but never t ook rem edial act ion. Result : The pilot support st aff never m it igat ed issues escalat ed by Ryan's t eam . I n t he end, t he pilot was concluded wit hout knowledge of any known incident s, and Ansalya cont inued t o pilot solut ions wit hout m it igat ing securit y

risks. The lack of docum ent ed securit y policy prevent ed Ryan from discerning which act ivit y could indicat e securit y problem s. 8 .1 .1 .2 . Pa m discove r s n e t w or k a bu se by a n e x t r a n e t pa r t n e r I n est ablishing ext ranet net work connect ivit y t o a cont ract m anufact urer, Special Elect ric ( SE) unwisely provided direct I nt ernet access t hrough it s corporat e I nt ernet connect ion. This connect ion was quickly arranged and deem ed crit ical t o leverage t he cont ract m anufact urer's syst em s for a crucial product launch. While t he securit y t eam was m onit oring t he ext ranet environm ent for SE, Pam discovered peer- t o- peer file sharing and t raffic obfuscat ion via Tor. [ 6 7 ] [ 67]

See ht t p: / / www.t orproj ect .org/ for m ore inform at ion.

Pam t raced t he access via SE's I nt ernet connect ion t o t he new cont ract m anufact urer, docum ent ing hundreds of connect ions from t wo host s at t he part ner's sit e. Pam knew t hat t his access was an abuse of t he part ner's int erconnect ion t o SE, and raised t he issue wit h t he m anagers who had arranged t he connect ion. Result : Because policies had not been est ablished t o lim it t he part ner's access, Pam 's not ificat ions t o m anagem ent regarding net work abuse fell on deaf ears.

8 .1 .2 . Kn ow You r N e t w or k Here are t wo t rue st ories t hat illust rat e t he im port ance of docum ent ing your net work boundaries and syst em s. 8 .1 .2 .1 . M ich a e l m on it or s a n a cqu isit ion When Denat a Financials acquired a sm all brokerage nam ed MM, t he acquisit ion was perm it t ed t o conduct it s own incident response and m onit oring for t he first t wo years. MM had out sourced securit y m onit oring t o a service part ner, and Michael was direct ed t o ret ire t hat arrangem ent , int egrat ing MM's securit y m onit oring wit h Denat a's ent erprise m onit oring. As he st udied t he exist ing set up, Michael learned t hat MM's service part ner provided no cont ext t o alert s—it was sim ply escalat ing every high- fidelit y NI DS alert t o securit y analyst s at MM. Because t he work had been out sourced, MM had m ade no effort t o docum ent essent ial segm ent s of t he net work so t hat it could apply cont ext t o it s securit y m onit oring. When Michael int egrat ed MM int o Denat a's ent erprise m onit oring, t his deficiency prevent ed cont ext - based alert ing. To rem edy t his, Michael docum ent ed t he m ost im port ant net work segm ent s—dat a cent ers, labs, and deskt op net works—creat ed variables in t he NI DS t o highlight t hem , and im m ediat ely im proved t he escalat ion pat h and fidelit y of t he securit y m onit oring. Result : Michael vast ly im proved MM's securit y m onit oring, dem onst rat ing t he superior securit y provided by docum ent ing and leveraging net work cont ext t o securit y m onit oring at Denat a. 8 .1 .2 .2 . H e le n a dds con t e x t t o t h e N I D S Helen was get t ing frust rat ed. The NI DS t eam was escalat ing every " high- fidelit y" securit y alert t o her. She analyzed scores of alert s every day, but as she gat hered incident det ails, she always seem ed t o uncover a m undane realit y: anot her lab- infect ed syst em at t acking a deskt op PC. Helen pat ient ly t raced t he alert det ails t o find t he pat h of at t ack, inferring inform at ion about t he source and dest inat ion based on t heir net work posit ions. Once she realized how m uch t im e she was wast ing, she set out t o fix t he problem for good. Helen developed a proj ect t o ident ify all net work segm ent s in an I nt ernet Prot ocol Address Managem ent ( I PAM) regist rat ion t ool. Once com plet e, she leveraged t he I PAM inform at ion reposit ory by configuring t he NI DS t o incorporat e net work cont ext int o each alert . Result : Pam 's work t o docum ent net work boundaries saved her t eam valuable m inut es on each alert , and allowed t hem t o priorit ize t heir m onit oring report s. N OTE

I n 2004, Cisco I T had a reasonably st rong I PAM solut ion, but it lacked t he abilit y t o program m at ically ret rieve inform at ion about addresses, subnet s, and descript ions. The dat a cont ained wit hin I PAM was locked up and not configured int o Cisco's NI DS. When NI DS alert s fired, t hey cont ained only t he I P address and DNS nam e of t he host s involved. Once t he Cisco NI DS product perm it t ed configurat ions t o accom m odat e hundreds of variables, t he Cisco com put er securit y incident response t eam ( CSI RT) added net work cont ext from t his cent ral reposit ory int o t he alert s. CSI RT t hen leveraged t his configurat ion t o t une hundreds of alert s, creat ing m onit oring playbooks, and launched t heir first global m onit oring t eam .

8 .1 .3 . Ch oose Ta r ge t s for Se cu r it y M on it or in g

Many effort s at securit y m onit oring fail because t hey don't focus on crit ical risks. They " boil t he ocean," forget t ing t o narrow m onit oring t o specific t arget s. 8 .1 .3 .1 . Pa m a n d t h e fa ile d pilot The alert s were arranged by color—t he m ost severe alert s colored red. St ill, t he t eam observed nearly 60 highseverit y alert s per hour. Pam 's t eam had deployed a Securit y I nform at ion Manager ( SI M) , which was supposed t o provide correlat ion, de-duplicat ion, and cont ext ualizat ion, aim ing t o reduce t he num ber of alert s. St aring at t he hundreds of " crit ical" alert s t he t eam received each day, Pam realized t his wasn't going t o work. Wit h nearly 100 NI DSs deployed, hundreds of servers sending logs, and billions of Net Flow report s from gat eways all over t he globe, t he new SI M bogged down; t here was sim ply no way t o priorit ize and keep up wit h t he alert s as t hey appeared ( see Figure 8- 2) . Figu r e 8 - 2 . A SI M , sh ow in g t h e va st n u m be r of a le r t s w h ich m a de it im possible for Pa m 's t e a m t o focu s t h e ir m on it or in g

Result : Pam 's SI M proj ect failed because t he t eam did not select t arget s for m onit oring, overwhelm ing t hem wit h " crit ical" alert s. However, t he t eam lat er t ook on a well- defined proj ect wit h clear policies and dist inct net work boundaries. The t eam select ed t hree servers and an applicat ion against which t o align securit y m onit oring. Wit h t he sam e SI M, t hey found t hey were able t o m ake product ive use of t he event s, and alert s t rickled in at a m anageable rat e of t hree t o four per hour .

8 .1 .4 . Ch oose Eve n t Sou r ce s I n t his sect ion, we'll highlight what went wrong when an em ployee t ried t o conduct securit y m onit oring wit hout t he necessary event sources. 8 .1 .4 .1 . D on a ld m on it or s h igh - r isk e m ploye e s Donald's t eam received advance warning of an im pending layoff. Because som e of t he em ployees held privileged account s on crit ical infrast ruct ure, he was advised t o conduct m onit oring of t hese em ployees t o prevent " abuse of privileges" and " t rust ed insider" at t acks. Donald began by reviewing t he securit y policies surrounding em ployee

logins, and found it easy t o draw up plans for policy- based m onit oring using logs from crit ical servers. He creat ed a list of crit ical servers based on t he rat ed priorit ies in t he com pany's infrast ruct ure regist rat ion syst em . He t hen focused his effort s on a few crit ical servers support ed by t he fat ed em ployees. As he began assem bling a plan t o m onit or t he em ployees, he discovered t hat NI DS and Net Flow feeds were working perfect ly, but none of t he servers were configured t o log event s for securit y review. Knowing t hat a request t o enable logging on t he servers would raise suspicion by t he very em ployees he needed t o m onit or, Donald found him self wit hout opt ions. Result : Donald failed t o det ect abuse of privileges by t hese em ployees. Lacking t im e t o deploy appropriat e event feeds, his t eam m em bers found t hem selves sift ing vast NI DS and Net Flow records looking for signs of abuse, but discovered not hing.

8 .1 .5 . Fe e d a n d Tu n e Here are t wo st ories illust rat ing what can go wrong when event sources are not properly deployed. 8 .1 .5 .1 . Ja n e t a n d t h e ca r e e r - lim it in g fa lse posit ive Once t he new NI DSs were deployed on t he Toront o dat a cent er gat eways, Janet began analyzing t he event s t hey generat ed. She had let t he sensors run for a few days t o ensure t hat t hey were funct ioning properly, but was eager t o see new act ivit y. She was alarm ed by t he sudden appearance of alert s indicat ing exploit at ion of RPC race condit ion vulnerabilit ies against t he Windows dom ain cont rollers. As she furt her analyzed t he t raffic, she t raced t he source t o ot her dom ain cont rollers. She had seen t his act ivit y before as com prom ised m achines on t he I nt ernet at t acked t he organizat ion's int ernal syst em s, but t he firewalls always dropped t he t raffic. Confident t hat t his indicat ed m alicious, dam aging at t acks, she im m ediat ely engaged t he Windows adm inist rat ors, opening an urgent case wit h t he net work operat ions cent er. Two Windows adm ins j oined t he call, groggy due t o t he lat e hour at com pany headquart ers in London. Janet relat ed t he problem t o t he adm ins, who looked for indicat ions t hat som et hing was wrong on t he servers. Convinced t hat som et hing was wrong but lacking a full explanat ion, Janet insist ed t hat t he servers be backed up and rebuilt , requiring m ore t han 20 hours of work by t he London Windows adm ins. Once t he servers were brought back online, t he alert s reappeared t he following week near t he sam e dat e and t im e. This t im e, when t he adm in j oined t he call, he was m ore skept ical. He not ed t hat t he t im ing was fishy, since it coincided wit h syst em m aint enance of t he dom ain cont rollers. Upon furt her research, he dem onst rat ed t hat because t he Windows servers exchange direct ory updat es at regular int ervals, it 's com m on t o see rapid DCOM connect ions, which can easily be m ist aken for exploit at ion of t he RPC vulnerabilit y. Result : Janet was reassigned t o anot her t eam . As a result of t his m isunderst anding, t he NI DS t eam realized t hat due t o t he m assive num ber of prot ocols and t raffic passing t hrough t he dat a cent er gat eways, t hey would require several m ont hs t o t une t heir NI DSs before t hey could reliably raise alert s for m onit oring. 8 .1 .5 .2 . D w igh t ove r w h e lm s t h e e ve n t colle ct or s Dwight had spent seven m ont hs convincing syst em adm inist rat ors t o send t heir event logs t o his new log collect ors. He had deployed a syslog server and spent several m ont hs preparing t o m onit or for signs t hat em ployees were sharing account s on crit ical syst em s. Now t hat he was collect ing t he needed event s, he found it a sim ple t ask t o forward t he event s t o t he SI M, and configure it t o st ore and relay t he logs. When he was finished, he wait ed a few days and t hen logged in t o t he SI M t o look for indicat ions of account sharing—calling out sim ult aneous inst ances of user login from m ore t han one I P at a t im e. He searched t hrough t he dat a, but found no init ial indicat ions of such policy violat ions. Three weeks lat er, Dwight was called in t o invest igat e an em ployee suspect ed of hacking a colleague's account t o download sensit ive files. He searched t hrough t he collect ed logs t o t race t he access, but it appeared t hat event s for t hat dat e were m issing. Upon furt her analysis, he found t hat t he dat a on his new collect or was ret ained for only t hree hours. Dwight had failed t o account for t he vast st orage needed t o support such " noisy" collect ion; his log collect or was discarding dat a m ore t han t hree hours old t o allow newly collect ed event s t o be st ored wit hin t he 500 GB allocat ed lim it . I n lieu of dat a support ing t he invest igat ion, he found t he event collect or clogged wit h m essages about NTP sync and debugging m essages from dozens of in- house applicat ions. Result : Dwight was unable t o offer dat a in support of t he invest igat ion, and it was closed for lack of evidence. Had he been careful t o t une his event feeds t o capt ure only what was needed t o support securit y m onit oring and invest igat ions, his t eam would have been able t o support t his invest igat ion .

8 .1 .6 . M a in t a in D e pe n da ble Eve n t Sou r ce s Here are t wo st ories t o illust rat e what went wrong when securit y m onit oring t ools were not properly m aint ained. 8 .1 .6 .1 . Lyle a n d t h e br ok e n N e t Flow colle ct or s A lab had been com prom ised, and Lyle's analysis of t he exploit m ade it clear t hat a det erm ined at t acker had com prom ised t he lab, m ost likely from a rem ot e connect ion. The t eam t ried t o analyze t he event s on t he local

m achine, but found t hat t he at t acker had erased t hem . The NI DS ident ified t he com prom ise, indicat ing a source originat ing from anot her developm ent lab at a rem ot e locat ion. Lyle's effort now cent ered on finding t he at t acker, and he knew t he best m et hod for t racking down t he at t acker was t o analyze Net Flow records from t he lab over t he days since t he at t ack. He logged in t o t he Net Flow collect or, querying records corresponding t o t he lab's I nt ernet connect ion. Lyle squint ed at t he screen—no Net Flow records had been logged for t hree weeks. He didn't underst and what had happened, but engaged t he net work engineering t eam t o check t he configurat ions on t he rout ers connect ing t he lab t o t he int ernal net work. They had never heard of Lyle's Net Flow collect ion servers, but had recent ly configured t he rout ers t o send all Net Flow t o t heir own collect ors for t raffic analysis and account ing. Unfort unat ely, t he net work t eam 's Net Flow collect ors hadn't ret ained t he dat a, and Lyle's invest igat ion hit a dead end. Result : Lyle's invest igat ion failed. Though Net Flow had been collect ed, t his gap in records prevent ed him from t racing t he act ivit y back t o a clear at t acker. 8 .1 .6 .2 . M a r ia n a n d t h e t h r e a t e n in g n ot e Marian was called int o her direct or's office. Facebook, a popular social net working sit e, had j ust cont act ed her regarding a t hreat ening not e post ed t o an em ployee's profile. The not e m ade reference t o t heir com pany, St ar Labs; Facebook's records proved t hat t he connect ions originat ed from St ar Labs' I P address space. Marian quickly t raced t he connect ion t o t he com pany's Pit t sburg cam pus. I nt ersect ing t he I P locat ion wit h t he t im e of t he incident , she was able t o isolat e t he t raffic t o a wireless net work in Building 22. Since it was a wireless net work, however, t he addresses were assigned dynam ically—an address is reused several t im es t hroughout t he day. She was relieved t o find t hat t he event s had been capt ured from t he DHCP servers t o St ar Labs' collect ion servers, but found t hat recent changes t o t he configurat ion of t he servers' NFS m ount s had prevent ed t he event s from being writ t en properly. Result : Marian's invest igat ion hit a dead end because t here was no way t o isolat e t he source of t he incident and correlat e it wit h a real user.

8 .2 . Ca se St u die s As we were com plet ing t his book, we want ed t o t est our m et hodology against real experiences t hroughout t he securit y com m unit y. As m em bers of t he Forum for I ncident Response and Securit y Team s ( FI RST) , Cisco has est ablished a t rust ed relat ionship wit h fellow incident response t eam s. Through FI RST, we found t wo ot her securit y t eam s int erest ed in sharing som e perspect ive regarding t heir securit y m onit oring. This highlight s how t heir securit y m onit oring aligns wit h t he m et hodology present ed in t his book. Here are case st udies from t wo respect ed securit y t eam s: KPN- CERT and Nort hrop Grum m an.

8 .2 .1 . KPN - CERT KPN is a Dut ch t elecom m unicat ions com pany t hat operat es fixed- line and m obile t elephony, I nt ernet , wireless t elevision, I CT, ret ail, and I PTV services. The com pany, which em ploys nearly 30,000 people, has an act ive com put er securit y incident response t eam called t he Com put er Em ergency Response t eam ( KPN- CERT) . This t eam provides securit y m onit oring and response for KPN's com pany net work as well as t he net works over which KPN offers it s services, including I nt ernet services. Like all com panies in t he Net herlands, KPN is governed by Dut ch and European Union laws. These regulat ions lim it t he dept h of KPN's securit y m onit oring, and require ret ent ion of som e event records. Because KPN is a t elecom m unicat ions com pany, t he cust om er dat a it st ores is furt her regulat ed, and t he securit y t eam m ust act ively m onit or for securit y breaches. N OTE

KPN has several divisions, wit h policies and response processes applied different ly t o each. The inform at ion represent ed in t his sect ion is specifically for m onit oring XS4ALL, an I nt ernet service provider ( I SP) owned by KPN. 8 .2 .1 .1 . Policie s KPN's cult ure em powers em ployees, allowing t hem t o do t heir j obs as t hey see fit . The securit y t eam , t herefore, t ries not t o lim it t heir access any m ore t han necessary. To develop policies, KPN- CERT analyzed it s net work t raffic t o charact erize t hreat s t o it s environm ent ; t he securit y officers t hen developed policies based on t he legit im at e t raffic generat ed by it s em ployees and cust om ers. Here are t heir five prim ary policies, m ainly direct ed at em ployees:

Securit y, which creat es a fram ework for all ot her policies. This covers t he role of I nfoSec, how inform at ion is t o be classified, and t he dut ies of em ployees. Dat a Sharing, which governs how dat a is t o be handled and prot ect ed. Logging, which governs what event s should be logged, how t hey are t o be recorded , and secure m et hods of access. Office Net work , which direct s accept able use of com put ers and t he net work. This policy prescribes which com put ers are allowed t o connect t o KPN's net work, and explicit ly bans som e net work act ivit ies, including t he use of peer- t o- peer applicat ions. This policy also requires t hat every device regist er it s MAC address as a condit ion for using t he KPN net work, but will be updat ed t o require 802.1x aut hent icat ion in 2009. Third Part y, which governs how KPN- CERT int eract s wit h ext ernal organizat ions, and declares t heir right t o audit act ivit y and services provided by t hose organizat ions. Securit y, which covers t he role of I nfoSec, how inform at ion is t o be classified, and t he dut ies of em ployees. 8 .2 .1 .2 . N e t w or k KPN- CERT applies different t ypes of m onit oring based on t he t ype of net work being m onit ored, including wireless, deskt op, server, and so fort h. 8 .2 .1 .3 . M on it or in g t a r ge t s KPN- CERT applies m onit oring where securit y cont rols are weakest . This philosophy direct s it s policies and select ion of t arget s for securit y m onit oring. Such priorit ized m onit oring allows KPN- CERT t o quickly react t o new t hreat s.

This priorit ized t hreat m onit oring was em ployed when an I RC t roj an em erged on EFnet , causing securit y adm inist rat ors t o ponder whet her an I RC server binary wit hin t heir own net work was t roj aned during t he incident . The CERT t eam provided m onit oring for t he hot t hreat , leveraging t heir knowledge of t he net work segm ent s t o apply t arget ed m onit oring against t he likely I RC servers running wit hin KPN. Such knowledge allows CERT t o priorit ize based on risk, even t hose t hat rapidly escalat e as t his one did. 8 .2 .1 .4 . Eve n t sou r ce s KPN- CERT collect s NI DS, Net Flow, and syslog dat a for securit y m onit oring, m aking heaviest use of NI DS dat a. KPN t raffic volum es require t hat t he NI DS support m ult igigabit t hroughput , requiring deploym ent of NI DS gear appropriat e t o t he t ask. The t eam also m akes use of net work int rusion prevent ion syst em s ( NI PSs) , t o aut om at ically block cert ain " known evils" and recreat ional t raffic. The NI PS rules are based on CERT's own securit y policies highlight ed earlier. Traffic t hat isn't aut om at ically blocked is inspect ed via t he NI DS for securit y m onit oring and forensics. For exam ple, when CERT ident ifies SSH access over a nonst andard port ( som et hing ot her t han port 22) , securit y analyst s m onit or and analyze t he NI DS alert s. Net Flow is collect ed regionally, sam pling Net Flow v5 from all choke point s. Using Arbor Peakflow t he CERT t eam wat ches for anom alies in t raffic t o alert t hem of suspicious behavior. Net Flow is furt her used as a short - t erm audit hist ory for net work t raffic, which is analyzed via Argus. Net Flow m onit oring was inst rum ent al in discovering I Pv6 experim ent at ion on t he office net work, as Peakflow highlight ed a spike in I Pv6 t raffic. This allowed t he securit y t eam t o ident ify t he source of t he t raffic and rest rict it before t he t raffic disrupt ed t he net work. Syslog is collect ed t o a cent ral server from rout ers, swit ches, firewalls, and Unix servers. CERT uses syslog from swit ches t o find securit y problem s such as DHCP snooping and rogue DHCP servers. Firewall logs are m onit ored t o ensure proper funct ioning, and t o wat ch for signs of int rusion at t em pt s. Unix servers, which account for t he largest proport ion of servers on t he net work, are m onit ored by CERT for st andard securit y problem s, such as repeat ed aut hent icat ion failures. The CERT t eam does not m ake use of a SI M, as it has not dem onst rat ed value for t heir m onit oring. Securit y analyst s require powerful flexibilit y and hum an analysis t o pick out anom alous pat t erns. I n lieu of a SI M, t hey observe t he event sources in real t im e, wit h som e lim it ed aut o- alert ing for known bad securit y event s. 8 .2 .1 .5 . M a in t e n a n ce KPN- CERT conduct s lim it ed m onit oring of it s securit y event feeds via hum an analysis, built - in t ools, and cust om script s. The t eam 's careful observat ion is oft en useful in highlight ing genuine securit y incident s. For exam ple, when an em ail virus disrupt ed t he KPN net work, CERT analyst s were observing t he NI DS. The alert s were indicat ing t he problem , but t he NI DS was m issing m uch of t he dat a and wasn't capt uring t rigger packet s. Based on t he analyst s' experience, t his problem indicat ed t hat a configurat ion change had been im plem ent ed on t he rout ers feeding t he NI DS. This m isconfigurat ion lim it ed CERT's abilit y t o invest igat e t he im pact ed users, as well as it s abilit y t o t race t he act ivit y t o it s source. Aft er invest igat ing t he NI DS configurat ion, t he CERT analyst s discovered t hat an em ployee had enabled a new link on t he office net work, changing t he t raffic pat h and causing t heir NI DS t o m iss half t he t raffic. To prevent t his problem in t he fut ure, CERT changed access perm issions on t he rout ers t o lim it access. N OTE

Though NI DS t raffic was im pact ed by t his incident , Net Flow was not . This highlight s t he addit ional value in using m ult iple sources of securit y event dat a. The syst em used for m onit oring anom alous act ivit y via Net Flow also m onit ors it self, raising an alert when act ivit y exceeds or falls below defined t hresholds. A recent alert from t his syst em showed t hat act ivit y had st opped com plet ely. CERT's analysis uncovered a rout ing problem —rout ing had been reassigned t hrough a different int erface, but was quickly repaired. Syslog feeds are m onit ored via Perl script s, which wat ch for subst ant ive variat ion in t he num ber of event s logged t o t heir cent ral collect ors. Allowance is m ade for peak periods, when em ployees are arriving or depart ing work. 8 .2 .1 .6 . An a ppr oa ch t o pr ot e ct cu st om e r da t a KPN- CERT recognizes t he value of it s cust om er dat a, and balances it s responsibilit y t o m onit or for securit y breaches against a cult ure t hat prom ot es creat ivit y in it s workforce.

8 .2 .2 . N or t h r op Gr u m m a n One of t he largest defense cont ract ors in t he world, Nort hrop Grum m an em ploys 120,000 people worldwide. I t s

large I nfoSec organizat ion includes t eam s t o address advanced t hreat analysis, int elligence, inform at ion assurance, m onit oring, and incident response. Due t o st at e- sponsored espionage aim ed at defense cont ract ors, Nort hrop Grum m an's securit y t eam s address t hreat s beyond t hose of m ost ent erprises, while t ypical organizat ions focus on prot ect ing against opport unist ic at t ackers. [ 6 8 ] [ 6 8 ] McNab, C. 2007. " Classifying I nt ernet - based At t ackers," in Net work Securit y Assessm ent . Sebast opol, CA: O'Reilly Media, I nc.

8 .2 .2 .1 . Policie s Like m ost com panies, Nort hrop Grum m an provides accept able use policies t o guide em ployee behavior, including direct ives for use of em ail and at t achm ent s. Nort hrop Grum m an's m onit oring is not prim arily direct ed by em ployee policies, but rat her by t hreat s and t he value of prot ect ed dat a. 8 .2 .2 .2 . N e t w or k t opology, m e t a da t a , a n d m on it or in g t a r ge t s Nort hrop Grum m an leverages it s docum ent ed net work t opology t o priorit ize m onit oring, but in a unique way. Most organizat ions delineat e server net works from deskt op and lab net works, but Nort hrop Grum m an priorit izes based on I nt ernet exposure and business crit icalit y. This inform at ion is int ersect ed wit h a dynam ic t hreat landscape using CERT's OCTAVE[ 6 9 ] risk assessm ent m et hodology. OCTAVE highlight s crit ical, sensit ive areas, allowing Nort hrop Grum m an t o label t he net work segm ent s for priorit y m onit oring. [ 69]

See ht t p: / / www.cert .org/ oct ave/ for m ore inform at ion.

8 .2 .2 .3 . Eve n t sou r ce s Nort hrop Grum m an uses a SI M t o gat her securit y event s from m ost sources, allowing t he securit y analyst s t o correlat e and report securit y event s against known t hreat s. The SI M is fed by NI DS and NI PS dat a, along wit h firewall and proxy logs. Like m ost ent erprises, Nort hrop Grum m an has found perim et er m onit oring less valuable as it has becom e m ore porous; correspondingly, t hreat s have driven deeper int o t he infrast ruct ure. Nort hrop Grum m an's CSI RT finds it self m onit oring end host s m ore frequent ly, especially as m ore virt ual m achines ( VMs) are deployed int o t he dat a cent er. ( A com prom ised VM could have access t o all VMs on t he sam e physical host wit hout t raversing rout ers or net work m onit oring t ools.) Nort hrop Grum m an collect s and analyzes any dat a t hat can reflect a change t o t he infrast ruct ure:

Net Flow and syslog have proven useful for det ect ing changes and alert ing. Windows event logs are collect ed but not gat hered int o t he SI M for aut om at ic correlat ion and analysis. Full packet capt ure is collect ed at net work choke point s. Expert s analyze t his t raffic, searching for behavior pat t erns and problem s. Dat a indicat ors spot t ed via packet capt ure som et im es end up as prim ary indicat ors in t he SI M. Two addit ional event sources are used t o augm ent Nort hrop Grum m an's real- t im e det ect ion capabilit ies, leveraged during incident response and invest igat ions:

Vulnerabilit y scans, used t o correlat e host dat a wit h real- t im e alert s and discover fresh t hreat s. Scans include all servers, net work devices, applicat ions, deskt ops, and dat abases. Windows dom ain cont roller logs and host int rusion prevent ion syst em ( HI PS) logs. 8 .2 .2 .4 . M a in t e n a n ce Syst em adm inist rat ors at Nort hrop Grum m an are responsible for m onit oring all syst em s, including securit y m onit oring t ools. Their support addresses t he overall t hroughput and healt h of such syst em s. To supplem ent t he syst em adm inist rat ors' support , one net work engineer is assigned t o observe event s from t he NI DS and NI PS. Availabilit y m et rics from event sources are analyzed and report ed via a daily st at us sum m ary, which shows:

Ut ilizat ion Throughput

General syst em healt h ( upt im e, et c.) Event volum es Select ed event t ypes are t rended and correlat ed wit h announced vulnerabilit ies t o ensure t hat Nort hrop Grum m an st ays on t op of hot t hreat s. This allows Nort hrop Grum m an t o priorit ize pat ch rollout s and awareness cam paigns wit hin t he organizat ion. Though Nort hrop Grum m an m onit ors securit y event sources, m aint aining adequat e st orage for all event sources is challenging, especially for packet capt ures. During a recent m ult ist age at t ack, t he NI DS sensors raised unique alert s, leading securit y analyst s t o quickly t rigger full packet capt ure t o support t heir invest igat ion. Due t o t he nat ure of t he at t ack—a coordinat ed int rusion against m ult iple com pany sit es—t he incident was not im m ediat ely recognized. While t he t raffic was being analyzed, t he packet capt ure devices rolled t heir st orage, overwrit ing dat a from t he beginning of t he at t ack. I n t he end, t he analyst s were able t o init iat e a response, but t hey lacked t he dept h t hey needed for a full invest igat ion. 8 .2 .2 .5 . A dyn a m ic- t h r e a t - or ie n t e d se cu r it y t e a m Nort hrop Grum m an places value in dynam ic t hreat analysis, and orient s a large proport ion of it s m onit oring t oward responding t o such t hreat s. Much of it s approach aligns wit h t he t arget ed m onit oring principles we've encouraged in t his book, but you can clearly see how t hey are t ilt ed t oward sophist icat ed t hreat s.

8 .3 . Re a l St or ie s of t h e CSI RT To illust rat e t he m onit oring st eps we've recom m ended in t his book, we collect ed st ories from securit y m onit oring t eam s in peer organizat ions. These st ories illust rat e incident response and som e of t he lim it at ions of ent erprise securit y m onit oring. N OTE

Because we were asked t o keep som e st ories anonym ous, we've added sm all em bellishm ent s t o prevent ident ificat ion of affect ed organizat ions as necessary. Sim ilarit ies wit h act ual event s, individuals, or corporat ions are neit her int ent ional nor im plied.

8 .3 .1 . St ole n I n t e lle ct u a l Pr ope r t y A cust om er alert ed Mike, a securit y invest igat or for Wirespeed, t hat source code belonging t o his com pany had been post ed on t he I nt ernet . Following t he leads provided, he visit ed a web address t hat confirm ed t he inform at ion. Mike not ed t hat t he source code was visible due t o a m isconfigurat ion of t he sit e's Apache web server, allowing unaut hent icat ed direct ory t raversal. Mike perused t he code, im m ediat ely recognizing it as Wirespeed's order processing soft ware. He even found som e em bedded syst em credent ials, dat abase account s, and passwords wit hin t he open direct ories. Mike t raced t he web server's I P address t o a hom e I SP connect ion, and was surprised when it correlat ed t o t he hom e I P address of a Wirespeed em ployee. Because Wirespeed m anaged t he I nt ernet connect ions of it s em ployees, he was able t o im m ediat ely disable t he em ployee's net work connect ion and invest igat e furt her. Wit h t he websit e now disabled, he t urned his at t ent ion t o t he search engines now caching t he source code. Using his ext ensive indust ry cont act s, Mike quickly secured agreem ent from Google, Yahoo! , and MSN t o delet e t he inform at ion. Thankfully, t here were no indicat ions t hat t he code had been discovered and repost ed by ot hers. I nvest igat ing how t he source code was leaked, Mike found t hat t he em ployee had " borrowed" t he source code t o st art his own com pany, forget t ing t o rem ove ident ifying and sensit ive inform at ion. Mike direct ed t he applicat ion support t eam t o change t heir dat abase passwords, added net work access cont rols t o prevent ext ernal connect ions t o t he im pact ed dat abase, and direct ed t he em ployee and his boss t o hum an resources for review of t he em ployee's policy violat ions. A post - m ort em review of t he m onit oring syst em s uncovered no alert s logged for t his incident . Mike concluded t hat t he event was m issed because t he em ployee's I SP connect ion was not in t he scope of t he com pany's net work m onit oring. Wirespeed's net work access cont rols allowed all em ployees' hom e net works t o access it s int ernal net work as part of t heir support ed configurat ion. The incident report did not recom m end any changes t o t he com pany's m onit oring syst em s or access cont rols, but direct ed t he awareness t eam t o rem ind em ployees of t heir dut y regarding t he prot ect ion of Wirespeed's confident ial inform at ion.

8 .3 .2 . Ta r ge t e d At t a ck Aga in st Em ploye e s Due t o t he sensit ive nat ure of it s dat a and t he unique t hreat s faced by it s ent erprise, Nort hrop Grum m an m aint ains an act ive relat ionship wit h it s user base. Users are em ployed as a first line of defense against social engineering at t acks and suspicious act ivit y. I n Oct ober 2008, t he com pany's CSI RT received a user report concerning a suspicious em ail at t achm ent . CSI RT passed t he report t o George, who coordinat es deep t hreat analysis for Nort hrop Grum m an. George searched for correlat ion wit h com plem ent ary dat a sources, querying t he SI M for Net Flow records and NI DS alert s during t he sam e t im e period, and com paring t he t rapped binary against capt ures from t he proxy server. George's analysis of proxy logs clearly indicat ed t hat m ore t han 20 em ployees had received t he sam e at t achm ent wit hin t he past few hours. Using a propriet ary dat abase, George found an im m ediat e connect ion shared by all affect ed em ployees: t heir work for Nort hrop Grum m an was all wit hin t he sam e program . As t he invest igat ion t urned t oward response and rem ediat ion of t his t arget ed at t ack, George t urned t o CSI RT, whose analysis confirm ed t hat none of t he vict im em ployees were com prom ised. The com pany's wide deploym ent of aut om at ed pat ching and host - based int rusion prevent ion t ools t o t he deskt ops had prevent ed com prom ise. To ensure t hat Nort hrop Grum m an had addressed t he problem , t he I nfoSec t eam built a m icro- awareness cam paign, quickly inform ing users of t he specific risks posed by t he t roj an. I nfoSec furt her conduct ed a risk assessm ent against t he affect ed program , highlight ing t hat t he com pany was sufficient ly prot ect ed from t he t hreat . I n it s post - m ort em assessm ent , Nort hrop Grum m an found t hat st rong collaborat ion and t im ely, in- dept h t hreat analysis allowed for a rapid and effect ive response, cont aining t he t hreat .

8 .4 . Ba r e M in im u m Re qu ir e m e n t s When m y first child was born, our annual Christ m as t rek t o I owa required m e t o st uff our 1989 Ford Taurus wit h every it em of clot hing, t oy, and art icle of port able baby furnit ure we owned. Three children lat er, we've discovered how lit t le we t ruly need for a week at Grandm a's. Likewise, you can oft en succeed wit h far less t han your ideals, especially when you reduce your scope. I n t he spirit of efficiency, here are a few essent ials t o apply for success in t arget ed m onit oring.

8 .4 .1 . Policy You can't escape policy—it 's your securit y m onit oring anchor. St ill, it 's hard t o know where t o begin. Here are t he essent ial policies for m ost organizat ions, against which you can conduct product ive securit y m onit oring. 8 .4 .1 .1 . Policy 1 : Allow e d n e t w or k a ct ivit y Be clear what net work access is allowed and what isn't . This is especially t rue of t he m ost sensit ive and crit ical net work segm ent s. When analyst s det ect act ivit y t oward t he I nt ernet from dat a cent ers, t hey need clear, docum ent ed policies regarding what net work act ivit y is allowed so t hat t hey can conduct fruit ful invest igat ions. 8 .4 .1 .2 . Policy 2 : Allow e d a cce ss Docum ent who can and should access t he organizat ion's m ost crit ical, sensit ive servers. Docum ent ing who is allowed access creat es a reference point against unaut horized access. I t perm it s discovery and enforcem ent of access t hat is out of alignm ent wit h securit y policy. 8 .4 .1 .3 . Policy 3 : M in im u m a cce ss st a n da r ds Dict at e t he securit y st andards expect ed of devices present on t he net work. This m ust include:

I dent ificat ion and nam ing I f t he device doesn't t ie act ivit y back t o individuals, invest igat ions will hit dead ends.

Logging I f t he device isn't required t o m aint ain logging or unique logins, it 's not possible t o effect ively t race act ivit y.

Aut hent icat ion I f t here's no st andard for access cont rol, m isconfigurat ions and weaknesses m ay perm it securit y incident s.

8 .4 .2 . Kn ow t h e N e t w or k Wit hout basic knowledge of t he organizat ion's net work segm ent s, securit y analyst s lack t he m ost fundam ent al m eans of priorit izing and cont ext ualizing securit y alert s. Great er det ail is helpful, but in a pinch, at least set up an I PAM solut ion and docum ent t he m ost basic dem arcat ions. 8 .4 .2 .1 . St e p 1 : Se t u p a n I PAM solu t ion Organizat ions need a m eans of docum ent ing t heir hierarchy of I P addresses. For m ost net works, it 's im pract ical t o represent a hierarchy of addresses in anyt hing but a dat abase. Rat her t han t rying t o see how m any rows Excel 2007 will support , spend t im e set t ing up an I PAM solut ion. I Pplan is a free and open source solut ion t hat runs on LAMP ( a Linux server wit h Apache, MySQL, and PHP) . 8 .4 .2 .2 . St e p 2 : D ocu m e n t ba sic I P de m a r ca t ion s I n t he I PAM solut ion, docum ent all net work address space owned by t he organizat ion, grouping and ident ifying t he following net works:

DMZ

Docum ent t he space assigned t o I nt ernet - accessible m achines, as you can expect t hem t o see a lot m ore at t acks.

Dat a cent er Dat a cent ers are hom e t o t he m ost crit ical infrast ruct ure. At least docum ent t he subnet s corresponding t o t he dat a cent er. I t will overlap ( or fully cont ain) t he DMZ address space, but it 's good t o know where it is.

Labs Docum ent t he address space used by labs ( bot h DMZ and int ernal) on t he net work, as t hey will generat e event s t hat are m ore unusual ( hopefully) t han t hose seen on t he rest of t he net work.

Part ners I f t here are ext ranet connect ions—places where part ners are connect ed direct ly t o t he organizat ion's net work—segm ent and docum ent t hem .

Rem ot e access Many organizat ions allow rem ot e access via a virt ual privat e net work ( VPN) of som e kind ( soft ware or hardware) . Docum ent t hose net work subnet s here.

Deskt op Last ly, t he " rest of" t he net work is probably available for end users as t he " deskt op" net works; docum ent t he address ranges of t hose subnet s. Figure 8- 3 depict s t he net work ranges and t heir relat ionships t hat you should docum ent in t he I PAM syst em . Figu r e 8 - 3 . Ba sic n e t w or k de m a r ca t ion s t o docu m e n t in a n I PAM syst e m

8 .4 .3 . Se le ct Ta r ge t s for Effe ct ive M on it or in g

Target m onit oring t oward areas of risk and sensit ivit y:

Risky syst em s Syst em s t hat present clear risk—such as t hose called out by a form al risk assessm ent —are excellent t arget s for securit y m onit oring. This is especially helpful where t he risk cannot be sufficient ly m it igat ed wit h cont rols such as net work ACLs or account rest rict ions.

Syst em s wit h sensit ive dat a Wherever sensit ive dat a is st ored, such as t he Social Securit y num bers st ored by Blanco Wireless, it 's vit al t o conduct securit y m onit oring t o prevent dat a loss and liabilit y. Be aware of governm ent regulat ions, which m ay specify and det ail t he t ype of m onit oring t hat is required.

8 .4 .4 . Ch oose Eve n t Sou r ce s Choose event sources t hat will allow effect ive m onit oring of t he select ed t arget s. This requires records t o det ect act ivit y on t he net work around t he t arget s, and on t he servers t hem selves. We believe t he m ost essent ial event sources are t hose which provide t he best alert - t o- m essage rat io. I n ot her words, event sources t hat produce a lot of un- act ionable m essages should not t ake first priorit y. The best event sources for t arget ed securit y m onit oring are oft en NI DS alert s, net work flows, and server logs. 8 .4 .4 .1 . N I D S a le r t s Alert s from a NI DS can be t uned t o t he organizat ion's specific environm ent , and can det ect specific pat t erns of act ivit y t hat will highlight policy violat ions for act ion. 8 .4 .4 .2 . N e t w or k flow s Cisco records net work flows very efficient ly wit h Net Flow, and sim ilar event t ypes can be capt ured on ot her vendors' hardware, including Juniper and Huawei. Net work flows capt ure j ust a few im port ant at t ribut es of I P packet s, allowing efficient st orage and analysis of net work t raffic. 8 .4 .4 .3 . Se r ve r logs Collect t he event s generat ed by servers via syslog. This is nat ive on Unix servers, and can be capt ured on Windows syst em s via t hird- part y t ools such as Snare. Collect t he following event s: aut hent icat ion, config changes, and daem on st art / st op event s. You should collect t hese event s from every server t hat is m at erial t o your t arget ed m onit oring.

8 .4 .5 . Fe e d a n d Tu n e Here are som e sim ple guidelines for deploying equipm ent t o collect and analyze NI DS alert s, net work flows, and server logs. 8 .4 .5 .1 . Se t u p a Se cu r it y I n for m a t ion M a n a ge r ( SI M ) Not everyone believes a SI M is necessary for securit y m onit oring, but it 's a sim ple solut ion for aggregat ing t he event sources art iculat ed in t his sect ion. A SI M will receive t raffic from each event source, highlight ing and correlat ing relat ed securit y event s. 8 .4 .5 .2 . D e ploy t h e N I D S Deploy t he NI DS t o net work choke point s in prom iscuous m ode ( so t hat it can see all t he t raffic) . Choke point s are norm ally at t he I nt ernet connect ion and dat a cent er gat eways, and m ay be found at branch offices and lab gat eways in som e organizat ions. 1. 2. 3. 4.

Configure t he NI DS t o send alert s t o t he SI M ( or t he SI M t o pull alert s from t he NI DS) . Tune t he NI DS, beginning wit h alert s t hat fire t he m ost . Configure t he NI DS t o use inform at ion about basic net work dem arcat ions of t he organizat ion, adding t he dat a t o t he NI DS alert s. Ret ire unwant ed or irrelevant NI DS signat ures.

8 .4 .5 .3 . Poin t N e t Flow a t t h e SI M

Though we recom m end int erm ediat e Net Flow collect ion via t he open source OSU flow- t ools, t he sim plest solut ion is t o point Net Flow at t he SI M and t o configure it t o not ify you of t raffic pat t erns forbidden by policies. Once it is deployed, configure t he choke point rout ers t o send t heir net work flows t o t he SI M. 8 .4 .5 .4 . Con figu r e se r ve r logs Configure t arget ed servers t o log aut hent icat ion, services t hat are st art ing/ st opping, and configurat ion changes t o t he SI M.

8 .4 .6 . M a in t a in D e pe n da ble Eve n t Sou r ce s To ensure t hat securit y event s are reliably and cont inuously capt ured, m onit or t he SI M t o m ake sure event s are arriving from each expect ed event source regularly. For each event source, query for a recent event and alert support st aff if a recent event has not been post ed from each source.

M on it or in g Se t u p Ch e ck list Finally, here's a checklist t o guide t he applicat ion of t he st eps present ed in t his book: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.

Docum ent t op t hreat s. This includes t he agent s t hat can harm t he organizat ion, and t he expect ed m et hods or circum st ances em ployed t o bring harm . Est ablish policies. Base policies on t hreat s, regulat ions, and st andards t o which t he organizat ion subscribes ( such as I SO) . Place policies on a cent ral websit e and com m unicat e t hem clearly t o st aff. Det erm ine which policies and t hreat s should be m onit ored. Conduct a Business I m pact Analysis ( BI A) t o det erm ine t he organizat ion's m ost crit ical infrast ruct ure. Conduct a risk assessm ent against t he crit ical infrast ruct ure highlight ed in t he BI A. Based on risk assessm ent and t op t hreat s, det erm ine which t arget s ( applicat ions, net works, et c.) t o m onit or. Choose event sources t o support securit y m onit oring. Det erm ine size requirem ent s for m onit oring equipm ent , based on fact ors such as necessary t hroughput , event ret ent ion t im es, price, and ent erprise fit . Deploy and t une m onit oring equipm ent . Negot iat e service level agreem ent s ( SLAs) wit h I T st aff t o support securit y event sources. Deploy configurat ion t em plat es for devices t hat are sending securit y event s t o keep event s flowing reliably. Choose and deploy a SI M. Configure securit y event sources t o feed int o t he select ed SI M. Develop m onit oring, escalat ion, and rem ediat ion procedures for securit y m onit oring. Acquire and t rain m onit oring st aff. Choose and deploy a syst em t o m onit or securit y event sources. Begin execut ing m onit oring procedures.

8 .5 . Con clu sion Each year, net work securit y m onit oring gains m ore capabilit ies for finding and m it igat ing securit y risks. Wit h a focus on act ionable m onit oring ( which you can achieve only if m onit oring is priorit ized and aligned wit h policies) , organizat ions can m eet t heir goals wit hout com prom ising securit y. We hope you've benefit ed from t his book, and t hat we'll see you at a securit y conference soon!

Appe n dix A. D e t a ile d OSU flow - t ools Colle ct or Se t u p This appendix gives det ailed inform at ion on set t ing up and running a Net Flow collect or based on OSU flow- t ools, followed by som e sim ple com m ands t o enable Net Flow generat ion from a Cisco I OS rout er. OSU flow- t ools is a set of open source Net Flow collect ion ut ilit ies, which you can reference at ht t p: / / www.splint ered.net / sw/ flow- t ools/ . Before you begin, ensure t hat your hardware m eet s t he inst allat ion requirem ent s, which are as sim ple as t he following:

One server ( or virt ual server inst ance) running t he * nix operat ing syst em An appropriat e am ount of disk space ( 250 GB t o 500 GB is a good st art ing point , t hough we've run som e low- t raffic environm ent s on as lit t le as 100 GB)

A.1 . Se t Up t h e Se r ve r To prepare your server for Net Flow collect ion, follow t hese st eps: 1. 2.

Download t he lat est package of flow- t ools ut ilit ies ( in t his case, t he version is 0.66) from ft p: / / ft p.eng.oar.net / pub/ flow- t ools/ flow- t ools- 0.66.t ar.gz. Place t he file in t he / t m p direct ory of your server. Ext ract t he files in / t m p wit h t he following com m and:

tar -xzvf flow-tools-0.66.tar.gz 3. 4. 5.

This creat es a flow- t ools- 0.66 direct ory. Run t he inst all- sh shell script in t hat direct ory as root . I t will inst all flow- t ools t o / usr/ local/ net flow , cont aining all t he flow- t ools binaries. Creat e a netflow user t o run t he collect ion soft ware. su t o t he netflow user and st art t he flow-capture process, which prepares t he syst em t o receive forwarded flows. There are several opt ions in t he st art up com m and which can be found on t he flow-capture m anpage. Of course, you m ust det erm ine t he correct set t ings for your own environm ent , but here is an exam ple t o get st art ed:

/usr/local/netflow/bin/flow-capture -w /var/local/flows/data -E90G -V5 -A1134 0/0/9999 -S5 -p /var/run/netflow/flow-capture.pid -N-1 -n288

The preceding line of code will:

Capt ure flows t o t he / var/ local/ flows/ dat a direct ory ( m ake sure t he netflow user can writ e here! ) . Begin rolling over once t here is 90 GB of flow dat a ( useful, for exam ple, if you have a 95 GB drive) . Collect and st ore Net Flow in version 5 form at . Tag flows wit h your AS num ber—for exam ple, AS1134. List en and capt ure on UDP 9999. Not e: you m ay have t o poke a hole in your firewall if you are collect ing from rout ers in t he DMZ, but t his collect or is inside t he firewall. Log a stats m essage every five m inut es. Specify t he process I D flow- capt ure.pid in locat ion / var/ run/ net flow/ . Set t he nest ing level t o YYYY- MM- DD/ flow- file ( see man flow-capture for opt ions) . Creat e a flow file 288 t im es per day ( every five m inut es) . You need t o ensure t hat flow- capt ure st art s when t he syst em reboot s. To do so, sim ply add t he following init script t o / et c/ init .d/ :

---------------------------------BEGIN init script----------------------------#! /bin/bash # # file to stop and start netflow prog="flow_capture" start() { echo -n $"Starting $prog: " su - netflow -c "/usr/local/netflow/bin/flow-capture -w /var/local/flows/data E90G -V5 -A 0/0/9999 -S5 -p /var/run/netflow/flow-capture.pid -N-1 -n288" RETVAL=$?

echo return $RETVAL } stop() { echo -n $"Stopping $prog: " kill -9 'ps -ef|grep flow-capture|grep -v grep|awk '{print $2}'` kill -9 'ps -ef|grep flow-fanout|grep -v grep|awk '{print $2}'` RETVAL=$? echo return $RETVAL } restart() { stop start } case "$1" in start) start ;; stop) stop ;; restart) restart ;; reload) reload ;; *) echo $"Usage: $0 {start|stop|restart}" exit 1 esac ---------------------------------END init script-------------------------------

A.2 . Con figu r in g N e t Flow Ex por t fr om t h e Rou t e r The following is a sim ple configurat ion st anza t o enable Net Flow generat ion and export from a Cisco I OS 12.x rout er. Refer t o your rout er docum ent at ion for soft ware and plat form - specific com m ands:

Router1(config)#ip route-cache flow Router1(config)#ip flow-export source Loopback 0 Router1(config)#ip flow-export destination 10.1.1.1 9999 Router1(config)#interface FastEthernet0/0 Router1(config-if)#ip route-cache flow

Appe n dix B. SLA Te m pla t e I n t his appendix, you will find a sam ple service level agreem ent ( SLA) for support ing securit y event feeds from net work devices. This sam ple SLA is arranged bet ween t he net work support t eam ( Net Eng) and t he t eam t o whom securit y m onit oring is assigned ( I nfoSec) . Following t he pract ice of t his book, t he scope belongs t o our fict it ious com pany, Blanco Wireless.

B.1 . Se r vice Le ve l Agr e e m e n t : I n for m a t ion Se cu r it y a n d N e t w or k En gin e e r in g B.1 .1 . Ove r vie w This is a service level agreem ent ( SLA) bet ween I nform at ion Securit y ( I nfoSec) and Net work Engineering ( Net Eng) . The purpose of t his docum ent is t o clarify support responsibilit ies and expect at ions. Specifically, it out lines:

Services provided by Net Eng t o support net work securit y event recording for m onit oring and incident response General levels of response, availabilit y, and m aint enance associat ed wit h t hese services Responsibilit ies of Net Eng as a provider of t hese services Responsibilit ies of I nfoSec as t he client and request er of t hese services Processes for request ing and com m unicat ing st at us of services This SLA shall rem ain valid unt il t erm inat ed. Approval and t erm inat ion indicat ions are not ed by signat ures in " 8.1: Approvals."

B.1 .2 . Se r vice D e scr ipt ion This service includes configurat ion of net work devices t o support securit y m onit oring. I t specifically requires:

Net Flow configurat ion t o I nfoSec Net Flow collect ors Logging configurat ion t o log appropriat e syslog m essages t o I nfoSec syslog collect ors SPAN configurat ion on rout ers t o m irror t raffic t o net work int rusion det ect ion syst em s ( NI DSs)

B.1 .3 . Scope The scope of t his agreem ent includes t he following devices where regist ered in Blanco's device m anagem ent syst em , and operat ing wit hin t he bounds of Blanco's global net work :

All Net Eng- support ed dist ribut ion layer aggregat ion rout ers ( choke point s) including, but not lim it ed t o, t he perim et ers of t he DMZ, product ion, ext ranet , and dat a cent er net works All I nfoSec- support ed NI DSs

B.1 .4 . Role s a n d Re spon sibilit ie s The Net Eng t eam will support t he process in cooperat ion wit h I nfoSec. B.1 .4 .1 . N e t En g r e spon sibilit ie s Net Eng will m aint ain t he following configurat ion on every Blanco choke point rout er:

Log Net Flow v5 t o port 2055 of t he I nfoSec- designat ed Net Flow collect ion server. Log auth and daemon m essages t o t he I nfoSec- designat ed syslog collect ion server. Configure one SPAN t o m irror bot h Rx and Tx t raffic t o t he NI DS. For rout ers in HSRP, RSPAN m ust be configured t o m irror all t raffic. This configurat ion will be m aint ained during norm al operat ions of all net work devices. Net Eng will coordinat e configurat ion changes and downt im e wit h I nfoSec via Blanco's change m anagem ent process. B.1 .4 .2 . I n foSe c r e spon sibilit ie s I nfoSec will m aint ain collect ion of securit y event s in support of incident response, m onit oring, and invest igat ions on

Blanco's net work. I nfoSec will also:

Provide access t o Net Flow and net work device log m essages st ored on collect ion servers. Monit or for securit y event s on net work infrast ruct ure. Provide incident response and invest igat ions during securit y incident s involving net work infrast ruct ure.

B.1 .5 . Se r vice Ope r a t ion s This sect ion det ails how service is request ed, hours of operat ion, expect ed response t im es, and escalat ion pat hs. B.1 .5 .1 . Re qu e st in g se r vice Service request s and change m anagem ent will use Blanco's in- house t ools t o log and rout e inform at ion.

I nfoSec will request service by logging cases t o Net Eng via t he Blanco Service Request Syst em ( BSR) . Urgent request s will be escalat ed via Global Operat ions. Net Eng will com m unicat e all out ages and configurat ion changes by adding t he group " I nfoSec" t o t he approval group on all change request s. B.1 .5 .2 . H ou r s of ope r a t ion Bot h I nfoSec and Net Eng will m aint ain 24/ 7 operat ions and support for t he services not ed in t his SLA. B.1 .5 .3 . Re spon se t im e s Net Eng agrees t o support t he securit y event feeds as a P2 service, which allows for up t o four hours of downt im e t o resolve problem s. B.1 .5 .4 . Esca la t ion s Should eit her part y require urgent at t ent ion t o a problem , Global Operat ions will conduct priorit y adj ust m ent s and coordinat ion of response. Assist ance wit h resolut ion of ongoing but nonurgent problem s will be handled by engaging t he m anagem ent of each respect ive organizat ion. B.1 .5 .5 . M a in t e n a n ce a n d se r vice ch a n ge s Rout ers support ing securit y event feeds will m aint ain 24/ 7 operat ions. There will be no regularly scheduled m aint enance, but necessary service out ages will be request ed and com m unicat ed via t he change m anagem ent syst em . Securit y event collect ors support ed by I nfoSec will m aint ain 24/ 7 operat ions wit h scheduled downt im e on Sundays from 1: 00 a.m . t o 2: 30 a.m . PST.

B.1 .6 . Agr e e m e n t D a t e s a n d Ch a n ge s This docum ent has been placed int o effect January 20, 2009 and will rem ain in perpet uit y. This docum ent will be reviewed for changes and new approvals every t wo years or when direct or- level m anagem ent changes are m ade t o eit her t he Net Eng or I nfoSec organizat ion, whichever com es first .

B.1 .7 . Su ppor t in g Policie s a n d Te m pla t e s This docum ent is in support of t he following Blanco Wireless policies:

Device Logging Policy Net work Securit y I ncident Response Policy Net work Securit y Monit oring Policy This docum ent requires t hat t he following t em plat es be applied t o all devices wit hin t he scope of t his SLA. These t em plat es will support t he configurat ion required by t his docum ent :

Net Flow Logging Tem plat e for Cisco I OS 12 Rout ers Event Logging Tem plat e for Cisco I OS 12 Rout ers

B.1 .8 . Appr ova ls, Te r m in a t ion s, a n d Re vie w s This docum ent m ust be elect ronically signed by a direct or in bot h t he Net Eng and I nfoSec organizat ions. B.1 .8 .1 . Appr ova ls This sect ion should not e t he approver, t it le, and effect ive dat e. Appr ove r

Tit le

John McCain

Direct or, Net work Engineering 1/ 20/ 09

Date

Barack Obam a Direct or, I nform at ion Securit y 1/ 20/ 09 B.1 .8 .2 . Te r m in a t ion s This sect ion should not e t he t erm inat ing direct or's nam e, t it le, and effect ive dat e. This sect ion is left blank unt il t his agreem ent is t erm inat ed. Te r m in a t in g dir e ct or Tit le D a t e

B.1 .8 .3 . Re vie w e r s This sect ion should list t he cont ribut ing edit ors and t hose whose review affect ed m at erial changes t o t he docum ent . Review er

Tit le

Date

Jason Bourne Net work Engineer 12/ 15/ 08 Michael St eele Securit y Engineer 12/ 09/ 08

Appe n dix C. Ca lcu la t in g Ava ila bilit y N OTE

Much of t he inform at ion t hat follows is based on t he concept s present ed in t he book High Availabilit y Net work Fundam ent als, by Chris Oggerino ( Cisco Press) . At t he t im e of t his writ ing, t he book is unfort unat ely out of print . I f you can get your hands on a copy, it is wort h your while. This appendix provides richer det ail t o help you evaluat e t he com ponent s of syst em availabilit y, as an ext ension of what was present ed in Chapt er 6 . You can calculat e t he availabilit y of a single com ponent wit h t he following equat ion:

So, t he availabilit y of a com ponent whose Mean Tim e Bet ween Failures ( MTBF) is 175,000 hours and Mean Tim e To Repair ( MTTR) is 30 m inut es would be:

I n ot her words, according t o t he m anufact urer's t est ing result s, t he com ponent is expect ed t o have only 1.51 m inut es of downt im e per year. Most syst em s are com posed of m ore t han one com ponent , of course. Mult icom ponent syst em s are arranged in a serial or a parallel fashion. For a serial com ponent - based syst em , each com ponent is a single point of failure, and so each com ponent depends on t he ot her for syst em availabilit y. I n cont rast , a parallel com ponent syst em has redundant com ponent s built such t hat t he failure of a single com ponent will not cause t he ent ire syst em t o fail. You can calculat e t he availabilit y of serial redundant com ponent s by m ult iplying t oget her t he availabilit y num bers for each single com ponent:

Here's how t o calculat e t he availabilit y of a serial m ult icom ponent syst em , consist ing of a processor, bus, and I / O

card:

This represent s 99.998% availabilit y, which is also called " four 9s and an 8." That was a sim plified exam ple. Now, let 's look at a redundant syst em availabilit y calculat ion ( see Figure C- 1) . Figu r e C- 1 . Block dia gr a m of a sim ple r e du n da n t syst e m ( Sou r ce : Ch r is Ogge r in o, H igh Ava ila bilit y N e t w or k Fu n da m e n t a ls, Cisco Pr e ss)

Figure C- 1 shows a diagram of a sim ple redundant syst em wit h t wo CPUs, t wo power supplies, and t wo I / O cards. You can calculat e availabilit y on such a syst em in t he sam e way you would calculat e serial availabilit y. The difference here is t hat each redundant syst em is calculat ed as t he difference of 1 m inus t he product of each redundant and serial com ponent . Not e t his key qualifier: a single redundant com ponent ( i.e., t wo power supplies) is 1 m inus t he product of t he individual com ponent 's availabilit y. The following form ula should help clear t his up:

Now t hat you underst and serial versus parallel syst em s, you can begin t o calculat e m ore com plex scenarios, such as what 's shown in t he following calculat ion. Assum e t hat you know your I / O card availabilit y is .99995, your CPU availabilit y is .99996, your power supply availabilit y is .99994, and your chassis availabilit y is .999998. The availabilit y calculat ion would be as follows:

The preceding calculat ion shows t hat , based purely on hardware MTBF num bers, t his scenario should have only 1.05 m inut es of downt im e per year; in ot her words, it is a " five 9s" syst em . You can obt ain t he MTBF com ponent of t he equat ion from your hardware m anufact urer, which, if it is a net work vendor, m ost likely uses t he Telcordia Part s Count Met hod, which is described in docum ent TR- 332 from ht t p: / / www.t elcordia.com / . Cisco list s MTBF inform at ion in it s product dat a sheet s, as do Juniper and ot hers.

Appe n dix . Coloph on The im age on t he cover of Securit y Monit oring is a m an using a t elescope. While t he t elescope is prim arily used for t he viewing of dist ant obj ect s, a host of earlier, cruder t elescopes were used sim ply for t he purposes of m agnificat ion. Euclid wrot e about t he reflect ion and refract ion of light , and Arist ophanes lat er showed t hat a globe filled wit h wat er could enlarge obj ect s. Yet t he invent ion of a proper t elescope was delayed in part because it s effect s were t hought t o be so ast onishing t hat t he inst rum ent and it s creat or were deem ed evil. I n t he 13t h cent ury, Roger Bacon docum ent ed t he effect s of m agnificat ion and wrot e about t he use of lenses t o st udy t he sky: " The Sun, Moon, and St ars m ay be m ade t o descend hit her in appearance…which persons unacquaint ed wit h such t hings would refuse t o believe." Subsequent t o his observat ions, Bacon was labeled a m agician and im prisoned. The use of t he lens for m agnificat ion only becam e accept able wit h t he invent ion and general usage of eyeglasses. Then, in t he lat e 16t h and early 17t h cent uries, eyeglass m aker Hans Lippershey of Holland report edly not iced a church t ower j um p t o t he front doorway of his shop when he st ared at t he t ower t hrough t wo different ly shaped lenses at once. Lippershey t hen succeeded in m aking t he t elescope known m ore widely, and it was he who piqued Galileo Galilei's int erest in t he inst rum ent som et im es dubbed t he " far looker." Galileo and Lippershey each independent ly t hought he could profit from t he dist ribut ion of t elescopes, and bot h m en also foresaw t he m ilit ary advant ages of t he inst rum ent . Galileo fam ously went a st ep furt her wit h his use of t he t elescope and sought out sun spot s, m oons of Jupit er, and new " lands" in t he sky above. Alt hough Galileo was event ually persecut ed for saying t hat t he sun was at t he cent er of t he solar syst em , his and Lippershey's m ilit ary applicat ion of sm aller t elescopes lat er becam e useful t o st rat egist s during t he U.S. Civil War, when m ilit ary personnel oft en used t elescopes designed like t he one on t he cover of t his book t o spy on t heir enem ies. The cover im age is from t he Dover Pict orial Archive. The cover font is Adobe I TC Garam ond. The t ext font is Linot ype Birka; t he heading font is Adobe Myriad Condensed; and t he code font is LucasFont 's TheSansMonoCondensed.

I n de x [ A] [ B] [ C] [ D] [ E] [ F] [ G] [ H] [ I ] [ J] [ K] [ L] [ M] [ N] [ O] [ P] [ Q] [ R] [ S] [ T] [ U] [ V] [ W]

[ A] access cont rols enum erat ion for securit y m onit oring policies, m inim um for securit y m onit oring access_log files ( Apache) account access, det ect ion of ACLs ( access cont rol list s) blocking connect ion from offending I P address creat ing for bot net virus I RC com m and and cont rol server lim it ing negat ive im pact s of ACL logging on syst em s logs, push m et hod of event collect ion net work ACL logs adm inist rat ive privileges, m onit oring for Oracle dat abase adm inist rat or user I Ds aggregat e bandwidt h alert level alert s CS- I PS alert generat ed by web server using Bit Torrent m onit oring from NI DS on Blanco wireless ( exam ple) m onit oring NI DS alert s net work cont ext for NI DS alert s overwhelm ing num bers of, result ing from not choosing m onit oring t arget s securit y alert for configurat ion change securit y alert sources for Blanco wireless net work t uning of NI DS alert s AngryMint St orm / 32 virus anom aly m onit oring 2nd Arbor Peakflow KPN- CERT case st udy Net Flow OSU flow- t ools solut ions for NI DS capabilit ies for 2nd ant ivirus logs Blanco wireless net work ( exam ple) m onit oring on Blanco wireless ( exam ple) querying t o find report ed and unresolved viruses syslog collect ion from Windows securit y applicat ion event s ( exam ple) ant ivirus soft ware, failure of Apache Web Server access_log files configurat ion for logging t o syslog ( exam ple) applicat ion accelerat ion applicat ion event s, securit y [ See securit y applicat ion event s] applicat ion logging Blanco wireless net work ( exam ple) applicat ion logs applicat ion service providers ( ASPs) Arbor Peakflow anom aly m onit oring by KPN- CERT asym m et ric rout ing audit ing m aint aining Oracle audit set t ings on obj ect s m aint aining Oracle syst em wide audit set t ings m onit oring audit configurat ions m onit oring Oracle audit event s perform ance im pact s and operat ional realit ies aut hent icat ion Cisco t erm inal server wit h TACACS+ aut hent icat ion, event collect ion im pact

m inim um st andards for securit y m onit oring aut hent icat ion event s Windows aut hent icat ion aut hent icat ion servers aut horizat ion event s Windows syst em s aut oconfig t em plat es av ailabilit y calculat ing 2nd equat ion for calculat ion of failure scenarios for NI PS devices high- availabilit y NI PS using physical redundancy im pact on NI DS versus NI PS decision m et rics in Nort hrop Grum m an case st udy [ t op]

[ B] bandw idt h analysis for links in exam ple net work t opology assessing aggregat e bandwidt h NI PS and net work bandwidt h banner ad m alware at t acks bidirect ional t raffic on net work int erfaces blacklist m onit oring blacklist ing condit ions for effect ive use of books on securit y t opics bot net s ident ifying infect ed host s part icipat ing in m alware analysis t ools for use of I RC t o rem ot ely cont rol com prom ised syst em s Bro buffer overflow at t acks Snort alert from Oracle syst em m onit oring and business im pact analysis ( BI A) 2nd [ t op]

[ C] CA Unicent er TNG caching servers canary event s, using t o m onit or sources cardholder dat a [ See PCI DSS m onit oring] case st udies KPN- CERT event sources m onit oring t arget s policies Nort hrop Grum m an dynam ic- t hreat - orient ed t eam event sources m aint enance and m onit oring of syst em s net work t opology, m et adat a, and m onit oring t arget s policies Cat bird, websit e m onit oring syst em CERT ( Com put er Em ergency Response Team ) , KPN change m anagem ent syst em , approved configurat ion request

choke point collect ion Cisco Syst em s Cisco Net work Regist rar COBI T configurat ion m onit oring on I OS rout ers I PS soft ware, versions 5 and 6 Securit y Monit oring, Analysis, and Response Syst em ( MARS) classified inform at ion clickj acking COBI T ( Cont rol Obj ect ives for inform at ion and relat ed Technology) code exam ples from t his book collect ion processes m onit oring for event log collect ion m onit oring for event log collect ors m onit oring for net work flow m onit oring on Blanco wireless ( exam ple) collect ion solut ions, syslog collect ors for Net Flow Com pass Bank, insider t heft of dat a from confident ial inform at ion configurat ions audit configurat ions m anaging on event source devices aut om at ed configurat ion m anagem ent est ablishing device logging policy service level agreem ent s ( SLAs) m onit oring net work configurat ion for Net Flow collect or cont ract ual obligat ions, securit y m onit oring for Count rywide Financial Corp., t heft by insider of cust om er dat a CPU of sending device, im pact of event collect ion on crim e, m iscreant econom y and organized crim e CS- I PS [ See NI DS] [ t op]

[ D] daem on st at us event s dashboards, net work m onit or dat a capt ure devices dat a cent ers design of NI DS deploym ent Net Flow collect ion point s Dat a Prot ect ion Policy ( Blanco exam ple) m onit oring dat abase logging Blanco wireless net work ( exam ple) dat abase logs dat abases audit event s, using canary event s t o m onit or sources m onit oring im port ant inform at ion for MySQL servers Oracle capt ure of audit inform at ion Oracle logging DDoS ( dist ribut ed denial- of- service) at t acks Design and Support 9 ( DS9) cont rol obj ect ive deskt op and wireless net works direct ories collect ion process, m onit oring logfiles, m onit oring m onit oring for Net Flow collect ion Net Flow collect ion direct ories, nam ed by dat e disk space m onit oring for m onit oring devices m onit oring for Net Flow collect ors

m onit oring for NI DS verifying for event log collect ors DMZ backbone rout ers, Net Flow collect ion at DMZ net works design of NI DS deploym ent DNS t unneling dom ain cont roller event s ( Windows) DoS ( denial- of- service) at t acks event logging in SQL Slam m er worm downt im e calculat ing for net work com ponent s nonhardware sources of downt im e DS9 ( Design and Support 9) cont rol obj ect ive [ t op]

[ E] egrep com m and elect ric ut ilit ies, securit y of crit ical syst em s for em ployee policies em ployees m onit oring t o prevent abuse of privileges t arget ed at t ack against , real- world case t heft of com pany's source code encrypt ion policies about , enforcem ent of requirem ent for cardholder dat a sent across open, public net works end st at ions ERP syst em , choosing com ponent s for securit y m onit oring event feeds [ See also event sources] gat hering com ponent det ails for event log collect ors m onit oring audit configurat ions Blanco wireless ( exam ple) collect ion direct ories ( logs) collect ion processes log ret ent ion net work t raffic syst em healt h overwhelm ing by failing t o t une event feeds event m essage det ail obt aining appropriat e am ount of det ail event sources choosing choosing for Blanco wireless ( exam ple) det erm ining how t hey will be used event collect ion, push and pull m et hods failure t o choose, causing failure of securit y m onit oring feeding and t uning im pact of event collect ion applicat ion logs dat abase logs host logs Net Flow net work ACL logs Net work I DS ( NI DS) im proper t uning and deploym ent , causing securit y m onit oring failure KPN- CERT case st udy m aint aining dependable 2nd failures caused by not m aint aining m anaging device configurat ions m aint aining reliable

aut om at ed syst em m onit oring KPN- CERT case st udy m onit oring dat abases m onit oring t he m onit ors syst em m onit oring for Blanco wireless ( exam ple) Nort hrop Grum m an case st udy relat ionship bet ween m essage rat e and ret ent ion t im e in event collect ion relat ionships am ong event rat e, m essage size, and disk ut ilizat ion in event collect ion event volum e, im pact on syst em s events key syslog event s m onit oring Oracle audit event s exam ple code from t his book expense im pact analysis for securit y m onit oring expire size or expire count for flows ext ernal net works ext ranet part ner, net work abuse by ( exam ple) ext ranet s Net Flow collect ion point NI PS deploym ent design [ t op]

[ F] Facilit y values, syslog m essages failure scenarios for hardware and soft ware NI PS device failures, analyzing im pact of nonhardware sources of downt im e Federal I nform at ion Securit y Managem ent Act of 2002 ( FI SMA) file t ransfers, flow aggregat ion of firewalls ident ificat ion of blacklist ed it em s securit y m onit oring of flow dat a, aggregat ion of flow ret ent ion flow- capt ure ut ilit y flow- cat ut ilit y flow- filt er ut ilit y flow- print ut ilit y forensics, event sources for forwarding capabilit y, NI PS devices [ t op]

[ G] gat eway rout ers m onit oring flows from Net Flow collect ion at generic account s 2nd sudo access t o Gram m - Leach Blilely Act grep checking Net Flow collect or for correct list ening port searching syslog.conf for a server [ t op]

[ H] Hannaford Bros., t heft of credit and debit card inform at ion from hardware and soft ware failure scenarios Healt h I nsurance Port abilit y and Account abilit y Act of 1996 ( HI PAA) m onit oring HI PAA applicat ions for unaut horized act ivit y healt h m onit oring collect or healt h on Blanco wireless ( exam ple) event log collect ors for healt h of m onit oring syst em for net work flow collect or NI DS sensors, Blanco wireless ( exam ple) syst em healt h of event log collect ors HI DS ( host int rusion det ect ion syst em ) logs Blanco wireless net work ( exam ple) m onit oring on Blanco wireless ( exam ple) Horizon Blue Cross Blue Shield, t heft of dat a from host int rusion prevent ion syst em s ( HI Ps) host I PS logs, m onit oring of host logs host variables for NI DS host nam es for m onit oring t arget s host s, scanning for on net work segm ent HP OpenView Net work Node Manager ( NNM) Huaw ei [ t op]

[I] I BM Tivoli Monit oring ifconfig com m and incident response and invest igat ion, event sources for I nform at ion Securit y Managem ent Syst em s ( I SMSs) inform at ion t echnology securit y assessm ent ( I TSA) insider t hreat s int ellect ual propert y, t heft of int ernal net works I nt ernet connect ion ( direct ) , from product ion servers I nt ernet Relay Chat ( I RC) int rusion det ect ion [ See NI DS] I nt rusion Det ect ion Syst em s Consort ium ( I DSC) I P address assignm ent , Blanco wireless net work ( exam ple) I P address inform at ion server I P addresses using for net work variables in NI DS configurat ion I P net work t ype classificat ion ext ernal net works int ernal net works I P packet s, analysis by NI DS I PAM ( I P address m anagem ent ) dat a Cisco Net work Regist rar st oring I P addresses docum ent ing basic net work dem arcat ions exam ple of list ing of I PAM solut ions using t o add cont ext t o NI DS using t o provide cont ext for NI DS alert ipt ables configurat ion rule for flows from Blanco wireless rout ers ( exam ple) creat ing rules for Net Flow collect or event log collect ors, m onit oring net work t raffic for flushing t he count ers using zero count ers opt ion

running wit h - VL I NPUT wat ching for t raffic volum e in server collect ion m onit oring I RC com m and and cont rol server I P address ( virus) I SO 1799 m onit oring I SP gat eway rout ers, Net Flow collect ion at I TSA ( inform at ion t echnology securit y assessm ent ) [ t op]

[ J] j um bo fram es Juniper [ t op]

[ K] Kerberos error codes, Windows dom ain cont rollers KPN- CERT case st udy event sources m onit oring t arget s policies prot ect ion of cust om er dat a [ t op]

[ L] lab ( net works) Lancope St ealt hWat ch legal request s event sources for exam ple request legal requirem ent s for m onit oring regulat ory com pliance load balancing, NI DS sensor in dat a cent er net work load, m onit oring for a syst em log ret ent ion logging [ See also applicat ion logging; syst em logging] applicat ion logs archiving m onit oring and secondary event s Blanco net work configurat ion ( exam ple) configuring server logs dat abase logs est ablishing policy for event source m onit oring event logging configurat ion, im pact on syst em perform ance event sources for securit y m onit oring, ERP syst em host logs, crit ical det ails capt ured wit h correct logging configurat ion host logs, using canary event t o m onit or sources level of, event volum e and m inim um for securit y m onit oring m onit oring event log collect ors net work ACL logs server logs collect ed via syslog volum e of m essages and

LogLogic syslog collect ors [ t op]

[ M] m alware det ect ion by ant ivirus product s, failure of dist ribut ion by t arget ed websit es prevalence and advanced capabilit ies of use by organized crim e Marshall's discount clot hing st ore, insecure wireless net work m edia, physical m em ory m onit oring availabilit y for NI DS m onit oring for m onit oring devices m onit oring for Net Flow collect ors m et adat a Nort hrop Grum m an case st udy Oracle audit set t ings on a t able pert aining t o I P com m unicat ions bet ween syst em s Microsoft SMS m onit oring 2nd [ See also securit y m onit oring] aut om at ed syst em m onit oring how t o m onit or t he m onit ors m onit oring wit h Nagios t radit ional net work m onit oring and m anagem ent syst em s dat abases MySQL servers Oracle capt ure of audit inform at ion Oracle logging securit y m onit oring syst em event log collect ors net work flow collect ion NI DS syst em healt h 2nd syst em m onit oring for Blanco wireless ( exam ple) MRTG ( Mult i Rout er Traffic Grapher) 2nd m onit oring t raffic volum e wit h MTBF ( m ean t im e bet ween failures) MTTR ( m ean t im e t o repair) Mult i Rout er Traffic Grapher ( MRTG) MySQL dat abase, m onit oring servers [ t op]

[ N] Nagios, syst em m onit oring wit h Blanco wireless net work ( exam ple) NAS ( net work at t achm ent st orage) Nat ional I ndust rial Securit y Program Operat ing Manual ( NI SPOM) Net Flow 2nd analysis solut ions for, considerat ions in select ion Blanco wireless net work ( exam ple) capt ure filt ering wit h OSU flow- t ools choosing collect or for collect ion configurat ion for Blanco net work ( exam ple) collect ion on Blanco wireless net work ( exam ple) configuring collect ion

copying t o ot her devices wit h flow- fanout export ing for collect ion failure of securit y m onit oring caused by broken collect ors header form at ( version 5) ident ifying infect ed host s part icipat ing in bot net s m onit oring collect ion on Blanco wireless net work ( exam ple) m onit oring flows from gat eway rout ers ( Blanco exam ple) m onit oring net work flow collect ion collect ion direct ories collect ion processes flow ret ent ion m onit oring healt h of collect or net work configurat ion of collect or t raffic feeds from rout ers wat ching for new syst em s OSU flow- t ools aggregat ion of flow dat a det ailed collect or set up ident ifying infect ed host s for bot net s ident ifying t raffic wit h phishing sit es repudiat ion and nonrepudiat ion of net work conversat ions perform ance considerat ions for collect ion point ing collect ion t o t he SI M point s for collect ion querying collect or records for canary event record form at ( version 5) records indicat ing connect ions securit y event t em plat e sending event logs t o right collect ors st orage of OSI Layers 3 and 4 I P packet at t ribut es use by KPN- CERT use of recorded connect ivit y inform at ion in exam ple legal request Net QoS Perform ance Cent er net st at checking collect or list ening for Net Flow t raffic on proper port checking for UDP list eners and processes bound t o t hose port s net work at t ached st orage ( NAS) net work bandwidt h, NI PS and net work device logs, Blanco wireless net work ( exam ple) net work flows 2nd [ See also Net Flow] m onit oring collect ion of Net work I PS Product Developers ( NI PD) Consort ium net work prot ocol analyzers net work scans, using t o wat ch for new rout ers or syst em s net work t raffic m ix net work variables, defining from I P address dat a net works Blanco wireless net work ( exam ple) I P address assignm ent Net Flow collect ion rout ing inform at ion charact erist ics of self- defeat ing net work classificat ion by t ype ext ernal net works int ernal net works im port ance of docum ent ing boundaries and syst em s ( exam ples) knowing your net work docum ent ing basic I P dem arcat ions KPN- CERT case st udy m onit oring and m anagem ent syst em s how syst em m onit oring works m onit oring net work perform ance new net works and syst em s added, securit y gaps rout ing and net work t opologies t axonom y classificat ion by t ype I P address m anagem ent dat a t elem et ry Net Flow SNMP

t opology analy zing exam ple net work t opology Nort hrom Grum m an case st udy nfdum p t ool NI DS ( net work int rusion det ect ion syst em s) alert s as event source alert s using I P address dat a available open source and com m ercial solut ions Blanco Wireless ( exam ple) choosing bet ween NI DS and NI PS av ailabilit y net work bandwidt h and NI PS nonhardware sources of downt im e span of cont rol deploy ing deploym ent fram ework analysis of net work environm ent deploying your NI DS designing t uning at t he sensor t uning at t he SI M t uning wit h cust om signat ures t uning wit h host variables t uning wit h net work variables t uning, docum ent at ion of failure t o properly t une and deploy m onit oring configurat ion for Blanco net work ( exam ple) m onit oring for proper funct ioning alert s sensor processes t raffic feeds ( uplinks) m onit oring HI PAA applicat ions for unaut horized act ivit y m onit oring on Blanco wireless ( exam ple) net work cont ext for NI DS alert s net work int rusion prevent ion syst em s ( NI PS) packet analysis and alert ing SLA for support ing NI DS feeds t uning t o lim it collect ion and st orage of false- posit ive alert s use by KPN- CERT using canary evenat s t o m onit or sources NI PD ( Net work I PS Product Developers) Consort ium NI PS ( net work int rusion prevent ion syst em s) choosing bet ween NI DS and NI ST risk assessm ent process Nm ap, sim ple ping net work scan Nort hrop Grum m an case st udy event sources dynam ic- t hreat - orient ed t eam m aint enance and m onit oring of syst em s policies t arget ed at t ack against em ployees NTUM error codes, Windows dom ain cont rollers [ t op]

[ O] OCTAVE risk assessm ent m et hodology OpenView Net work Node Manager ( NNM) opt im izat ion t ools using Net Flow Oracle applicat ion server, alert from Oracle dat abases audit log audit ing, configurat ion for Blanco net work ( exam ple)

dat abase logging dat abase logging on Blanco wireless net work ( exam ple) m onit oring capt ure of audit inform at ion m onit oring logging Oracle syst em , exam ple Snort alert from m onit oring order ent ry syst em s order fulfillm ent syst em s OS fingerprint ing OSSEC OSU flow- t ools 2nd 3rd [ See also Net Flow] configuring Net Flow collect ion det ailed collect or set up filt ering Net Flow capt ure flow aggregat ion flow- capt ure com m and flow- fanout com m and ident ifying infect ed host s part icipat ing in bot net s 2nd m onit oring collect ion processes querying Net Flow collect or records for canary event repudiat ion and nonrepudiat ion of net work t ransact ions out sourcing securit y m onit oring [ t op]

[ P] P2P ( peer- t o- peer) net working, policy m onit oring and packet capt ure packet inspect ion packet inspect ion capabilit y, NI PS passwords at t em pt s t o guess, syst em m onit oring for dat a st olen t hrough guessing weak passwords pat t ern m at ching by NI DS PCI DSS ( Paym ent Card I ndust ry Dat a Securit y St andard) m onit oring 2nd Peakflow anom aly m onit oring by KPN- CERT perform ance analysis on Net Flow dat a audit ing and im pact of event collect ion on syst em s im pact of Net Flow export from net work devices m onit oring for securit y m onit ors m onit oring net work perform ance perim et er ( net work) Periscan, websit e m onit oring syst em perm issions checking on collect ion direct ories direct ories st oring event logs m onit oring for Net Flow collect ors in syst em healt h m onit oring personal inform at ion, prot ect ion of personally ident ifiable inform at ion ( PI I ) prot ect ion policy for Blanco Wireless ( exam ple) phishing fraudulent em ail request ing back account inform at ion ( exam ple) spearphishing sum m er 2008 cam paign, UPS m essage physical m edia point of dem arcat ion or perim et er ( DMZ net work) policies Blanco Wireless ( exam ple) device logging policy for SLA policy m onit oring 2nd allowed access

allowed net work act ivit y condit ions for effect ive use of Dat a Prot ect ion Policy ( Blanco exam ple) against defined policies em ployee policies ident ifying policy violat ions using Net Flow im plem ent ing for Blanco net work policies ( exam ple) im port ance of creat ing policies before m onit oring begins m onit oring a risky vent ure ( exam ple) net work abuse by ext ranet part ner ( exam ple) KPN- CERT case st udy m anagem ent enforcem ent of policy m inim um access st andards Nort hrop Grum m an policies regulat ory com pliance policies Server Securit y Policy ( Blanco exam ple) versus ot her t ypes of m onit oring Priorit y value, syslog m essages privacy concerns, securit y m onit oring and process st at us event s ( Windows) processes [ See also collect ion processes] checking for NI DS sensors on Blanco wireless ( exam ple) m onit oring for correct num ber of collect ion and relay processes m onit oring NI DS sensor processes syst em load m onit oring product ion servers, direct I nt ernet connect ion from propriet ary inform at ion for com panies proxy server logs pull m et hod ( event collect ion) push m et hod ( event collect ion) [ t op]

[ Q] Qualys scan for risk assessm ent [ t op]

[ R] Red Hat Net work ( RHN) , using t o push configurat ion files t o m anaged servers regular expressions m at ching t o find credit card num bers t raversing net work m at ching U.S. Social Securit y num ber regulat ory com pliance regulat ory com pliance policies 2nd COBI T configurat ion cont rol m onit oring cont ract ual obligat ions Gram m - Leach Blilely Act ( exam ple) HI PAA applicat ions m onit oring I SO 1799 m onit oring Paym ent Card I ndust ry Dat a Securit y St andard ( PCI DSS) 2nd SOX m onit oring for financial apps and dat abases st andards for crit ical infrast ruct ure prot ect ion rem ot e access net works resources for furt her inform at ion revenue im pact analysis for securit y m onit oring risk assessm ent m et hodology ( OCTAVE) risk assessm ent s

risk profiles risk, securit y m onit oring t o m inim ize root kit t ed m achines rout ers Cisco rout er ACL log COBI T configurat ion m onit oring on I SP gat eway and DMZ backbone, Net Flow collect ion at m onit oring flows from gat eway rout ers Net Flow configurat ion for Cisco rout er t raffic feeds from , m onit oring for Net Flow collect ion t raffic graph by MRTG showing dram at ic drop in t raffic rout ing t opology asym m et ric rout ing Blanco wireless net work ( exam ple) RPC race condit ion vulnerabilit ies of Windows dom ain cont rollers [ t op]

[ S] SAP R/ 3 m onit oring Sapphire ( worm ) Sarbanes- Oxley Act of 2002 ( SOX) m onit oring for financial apps and dat abases SCADA ( Supervisory Cont rol and Dat a Acquisit ion) syst em s SDBot virus secondary event s, im port ance of securit y applicat ion event s Windows syst em s Securit y Device Event Exchange ( SDEE) Securit y I nform at ion Manager [ See SI M] securit y m onit oring [ See also m onit oring] challenges t o event sources for m inim um requirem ent s for choosing event sources feeding and t uning event sources m aint aining dependable event sources policies select ing m onit oring t arget s open source versus com m ercial product s for out sourcing reasosn for set up checklist securit y m onit oring t eam s, st ories from st olen int ellect ual propert y t arget ed at t ack against em ployees self- defeat ing net work, charact erist ics of sensit ivit y of dat a syst em s accessing classified inform at ion syst em s accessing confident ial inform at ion syst em s accessing personally ident ifiable inform at ion ( PI I ) sensors 2nd [ See also NI DS] net work t raffic m ix and locat ion of sensor t uning and m anaging t uning NI DS depoloym ent at Server Securit y Policy ( Blanco exam ple) m onit oring service level agreem ent s ( SLAs) configurat ion direct ives for event sources and t eam s est ablishing logging policy for devices sect ions or t opics t o include t em plat e for Severit y level indicat or, syslog m essages signat ures

analysis of creat ing host variables for cust om NI DS signat ure on Blanco wireless net work ( exam ple) cust om signat ures for NI DS SI M ( Securit y I nform at ion Manager) failure of, result ing from not choosing m onit oring t arget s point ing Net Flow t o server logging t o set t ing up SI M t ools using Net Flow t uning NI DS at Sit e Securit y Handbook ( RFC 2196) SLAs [ See service level agreem ent s] sniffers SNMP ( Sim ple Net work Managem ent Prot ocol) MRTG analysis t ool pull m et hod of event collect ion Snort alert from m onit oring Oracle syst em alert indicat ing pot ent ial DNS t unneling using I P address allocat ion for net work variables social engineering t echniques t o st eal com pet it ive int elligence soft ware failure scenarios source code, t heft of SOX [ See Sarbanes- Oxley Act of 2002] span of cont rol over net work- connect ed endpoint s spearphishing 2nd Windows securit y applicat ion event s ( exam ple) Splunk, analysis t ool for event s 2nd SQL Slam m er worm SSH daem on, at t ack by t roj aned daem on sudo com m and swap space, m onit oring for collect ors syslog- ng process 2nd m onit oring 2nd syslog.conf file checking configurat ion for event log collect ors syst em logging ( syslog) applicat ion logging dat abase logging aut hent icat ion event s aut horizat ion event s Blanco wireless net work ( exam ple) collect ing syslog configurat ion t em plat es daem on st at us event s det ect ion of syst em access and changes event collect ion by push m et hod exam ple m essage from Linux- based syst em key event s, list ed key Windows log event s KPN- CERT case st udy logging configurat ion for Blanco net work ( exam ple) logging for COBI T configurat ion m onit oring of rout ers logins on Unix server, recording logon and logoff m essages collect ed via m essage facilit ies m essage form at m essage Priorit y value m essage severit ies net work device logs for Blanco net work ( exam ple) Oracle dat abase securit y applicat ion event s server logs collect ed via volum e of log m essages syst em m onit oring Blanco wireless ( exam ple) event log collect ion m onit oring collect or healt h

m onit oring Net Flow collect ion Net Flow collect ion Net Flow collect ion processes Net Flow collect ion/ ant ivirus and HI DS logging Net Flow collect ors' healt h Net Flow, flows from gat eway rout ers NI DS Oracle logging how it works m onit oring t he m onit ors Nort hrop Grum m an case st udy using Nagios syst em wide audit set t ings in Oracle [ t op]

[ T] t ap hardware t arget ed m onit oring t arget s for securit y m onit oring, select ion of 2nd Blanco wireless net work ( exam ple) business im pact analysis m et hod choosing com ponent s wit hin t arget s ERP syst em exam ple gat hering com ponent det ails for event feeds expense im pact analysis m et hod KPN- CERT case st udy legal requirem ent s for m onit oring m et hods for select ing m onit oring failures from not choosing t arget s Nort hrop Grum m an pract ical considerat ions recom m ended m onit oring t arget s revenue im pact analysis m et hod risk profiles sensit ivit y profiles visibilit y profile t elem et ry t im e, synchronizing on servers TJX Com panies, breach of com put er net work t ools for securit y m onit oring, open source versus com m ercial product s t raffic asym m et ry t raffic feeds m onit oring for NI DS sensors on Blanco wireless ( exam ple) uplinks, m onit oring for NI DS t raffic m ix, net work t roj ans, binary analysis of t roj an at t ached t o phishing m essage t unneled t raffic [ t op]

[ U] Unix/ Linux syst em s aut om at ed configurat ion of m onit oring devices Linux syslog ent ry showing user t rying t o becom e root user logfile st orage locat ions m onit oring Unix net work perform ance syslog

URL access logs [ t op]

[ V] vendors of securit y syst em s, prom ises of aut om at ic m onit oring by viruses book about failure of ant ivirus soft ware infect ed host s part icipat ing in bot net s SDBot visibilit y profile [ t op]

[ W] wat er plant pum ping syst em , SCADA screenshot for websit e defacem ent s websit es, t arget ed for dist ribut ion of m alware whit elist m onit oring WI DS ( wireless int rusion det ect ion syst em s) Windows syst em s RPC race condit ion vulnerabilit ies of dom ain cont rollers syslog aut hent icat ion event s aut horizat ion event s collect ing server event s in syslog form at dom ain cont roller event s process st at us event s securit y applicat ion event s WI PS ( wireless int rusion prevent ion syst em ) wireless net works Blanco wireless ( exam ple) choosing event sources for policies policy m onit oring securit y alert sources select ing t arget s for m onit oring syst em m onit oring underst anding insecure wireless net work at Marshall's clot hing st ore [ t op]