BSD Hacks 9780596006792, 0596006799

Citation preview

---

•

Table of Cont ent s

•

I ndex

•

Rev iew s

•

Reader Review s

•

Errat a

•

Academ ic

BSD H a ck s By Dru Lavigne

Publisher: O'Reilly Pub Dat e: May 2004 I SBN: 0- 596- 00679- 9 Pages: 300

Looking for a unique set of pract ical t ips, t ricks, and t ools for adm inist rat ors and power users of BSD syst em s? From hacks t o cust om ize t he user environm ent t o net working, securing t he syst em , and opt im izat ion, BSD Hacks t akes a creat ive approach t o saving t im e and accom plishing m ore wit h fewer resources. I f you want m ore t han t he average BSD user- - t o explore and experim ent , uneart h short cut s, creat e useful t ools- - t his book is a m ust - have.

-1-

CREDITS .................................................................................................................................................. - 5 PREFACE............................................................................................................................................... - 10 CHAPTER 1. CUSTOMIZING THE USER ENVIRONMENT.......................................................... - 14 HACK 0 INTRODUCTION .......................................................................................................................- 15 HACK 1 GET THE MOST OUT OF THE DEFAULT SHELL ......................................................................- 16 HACK 2 USEFUL TCSH SHELL CONFIGURATION FILE OPTIONS ..........................................................- 21 HACK 3 CREATE SHELL BINDINGS ......................................................................................................- 25 HACK 4 USE TERMINAL AND X BINDINGS ...........................................................................................- 29 HACK 5 USE THE MOUSE AT A TERMINAL ...........................................................................................- 33 HACK 6 GET YOUR DAILY DOSE OF TRIVIA ........................................................................................- 35 HACK 7 LOCK THE SCREEN .................................................................................................................- 39 HACK 8 CREATE A TRASH DIRECTORY ...............................................................................................- 42 HACK 9 CUSTOMIZE USER CONFIGURATIONS ....................................................................................- 46 HACK 10 MAINTAIN YOUR ENVIRONMENT ON MULTIPLE SYSTEMS ...................................................- 56 HACK 11 USE AN INTERACTIVE SHELL................................................................................................- 60 HACK 12 USE MULTIPLE SCREENS ON ONE TERMINAL .....................................................................- 64 CHAPTER 2. DEALING WITH FILES AND FILESYSTEMS ......................................................... - 69 INTRODUCTION ....................................................................................................................................- 70 HACK 13 FIND THINGS ........................................................................................................................- 71 HACK 14 GET THE MOST OUT OF GREP .............................................................................................- 76 HACK 15 MANIPULATE FILES WITH SED ..............................................................................................- 81 HACK 16 FORMAT TEXT AT THE COMMAND LINE ...............................................................................- 84 HACK 17 DELIMITER DILEMMA ............................................................................................................- 91 HACK 18 DOS FLOPPY MANIPULATION..............................................................................................- 94 HACK 19 ACCESS WINDOWS SHARES WITHOUT A SERVER ............................................................- 102 HACK 20 DEAL WITH DISK HOGS ......................................................................................................- 105 HACK 21 MANAGE TEMPORARY FILES AND SWAP SPACE ...............................................................- 111 HACK 22 RECREATE A DIRECTORY STRUCTURE USING MTREE ......................................................- 115 HACK 23 GHOSTING SYSTEMS .........................................................................................................- 121 HACK 24 CUSTOMIZE THE DEFAULT BOOT MENU ............................................................................- 128 HACK 25 PROTECT THE BOOT PROCESS .........................................................................................- 134 HACK 26 RUN A HEADLESS SYSTEM ................................................................................................- 137 HACK 27 LOG A HEADLESS SERVER REMOTELY..............................................................................- 141 HACK 28 REMOVE THE TERMINAL LOGIN BANNER ...........................................................................- 145 HACK 29 PROTECTING PASSWORDS WITH BLOWFISH HASHES ......................................................- 149 HACK 30 MONITOR PASSWORD POLICY COMPLIANCE ....................................................................- 152 HACK 31 CREATE AN EFFECTIVE, REUSABLE PASSWORD POLICY .................................................- 161 HACK 32 AUTOMATE MEMORABLE PASSWORD GENERATION .........................................................- 167 HACK 33 USE ONE TIME PASSWORDS .............................................................................................- 172 HACK 34 RESTRICT LOGINS ..............................................................................................................- 176 -

-2-

CHAPTER 4. BACKING UP.............................................................................................................. - 180 HACK 35 BACK UP FREEBSD WITH SMBFS .................................................................................. - 182 HACK 36 CREATE PORTABLE POSIX ARCHIVES ............................................................................ - 186 HACK 37 INTERACTIVE COPY ........................................................................................................... - 191 HACK 38 SECURE BACKUPS OVER A NETWORK ............................................................................. - 195 HACK 39 AUTOMATE REMOTE BACKUPS ......................................................................................... - 198 HACK 40 AUTOMATE DATA DUMPS FOR POSTGRESQL DATABASES............................................. - 204 HACK 41 PERFORM CLIENT-SERVER CROSS-PLATFORM BACKUPS WITH BACULA ....................... - 208 CHAPTER 5. NETWORKING HACKS ............................................................................................ - 216 HACK 42 SEE CONSOLE MESSAGES OVER A REMOTE LOGIN........................................................ - 218 HACK 43 SPOOF A MAC ADDRESS ................................................................................................. - 221 HACK 44 USE MULTIPLE WIRELESS NIC CONFIGURATIONS .......................................................... - 225 HACK 45 SURVIVE CATASTROPHIC INTERNET LOSS ....................................................................... - 230 HACK 46 HUMANIZE TCPDUMP OUTPUT .......................................................................................... - 233 HACK 47 UNDERSTAND DNS RECORDS AND TOOLS ...................................................................... - 240 HACK 48 SEND AND RECEIVE EMAIL WITHOUT A MAIL CLIENT ...................................................... - 246 HACK 49 WHY DO I NEED SENDMAIL?............................................................................................. - 251 HACK 50 HOLD EMAIL FOR LATER DELIVERY .................................................................................. - 255 HACK 51 GET THE MOST OUT OF FTP............................................................................................ - 258 HACK 52 DISTRIBUTED COMMAND EXECUTION ............................................................................... - 262 HACK 53 INTERACTIVE REMOTE ADMINISTRATION .......................................................................... - 265 CHAPTER 6. SECURING THE SYSTEM........................................................................................ - 269 HACK 54 STRIP THE KERNEL ........................................................................................................... - 271 HACK 55 FREEBSD ACCESS CONTROL LISTS ................................................................................ - 282 HACK 56 PROTECT FILES WITH FLAGS ............................................................................................ - 289 HACK 57 TIGHTEN SECURITY WITH MANDATORY ACCESS CONTROL ............................................ - 295 HACK 58 USE MTREE AS A BUILT-IN TRIPWIRE................................................................................ - 299 HACK 59 INTRUSION DETECTION WITH SNORT, ACID, MYSQL, AND FREEBSD .......................... - 305 HACK 60 ENCRYPT YOUR HARD DISK ............................................................................................. - 317 HACK 61 SUDO GOTCHAS ................................................................................................................ - 322 HACK 62 SUDOSCRIPT ...................................................................................................................... - 326 HACK 63 RESTRICT AN SSH SERVER .............................................................................................. - 332 HACK 64 SCRIPT IP FILTER RULESETS ........................................................................................... - 336 HACK 65 SECURE A WIRELESS NETWORK USING PF..................................................................... - 339 HACK 66 AUTOMATICALLY GENERATE FIREWALL RULES ............................................................... - 344 HACK 67 AUTOMATE SECURITY PATCHES....................................................................................... - 350 HACK 68 SCAN A NETWORK OF WINDOWS COMPUTERS FOR VIRUSES ......................................... - 355 CHAPTER 7. GOING BEYOND THE BASICS............................................................................... - 359 HACK 69 TUNE FREEBSD FOR DIFFERENT APPLICATIONS ............................................................ - 361 -

-3-

HACK 70 TRAFFIC SHAPING ON FREEBSD ......................................................................................- 366 HACK 71 CREATE AN EMERGENCY REPAIR KIT ...............................................................................- 372 HACK 72 USE THE FREEBSD RECOVERY PROCESS .......................................................................- 376 HACK 73 USE THE GNU DEBUGGER TO ANALYZE A BUFFER OVERFLOW ......................................- 381 HACK 74 CONSOLIDATE WEB SERVER LOGS ...................................................................................- 385 HACK 75 SCRIPT USER INTERACTION ..............................................................................................- 391 HACK 76 CREATE A TRADE SHOW DEMO.........................................................................................- 396 CHAPTER 8. KEEPING UP-TO-DATE ........................................................................................... - 402 HACK 77 AUTOMATED INSTALL .........................................................................................................- 404 HACK 78 FREEBSD FROM SCRATCH ...............................................................................................- 409 HACK 79 SAFELY MERGE CHANGES TO /ETC ...................................................................................- 415 HACK 80 AUTOMATE UPDATES .........................................................................................................- 419 HACK 81 CREATE A PACKAGE REPOSITORY ....................................................................................- 425 HACK 82 BUILD A PORT WITHOUT THE PORTS TREE ......................................................................- 429 HACK 83 KEEP PORTS UP-TO-DATE WITH CTM..............................................................................- 433 HACK 84 NAVIGATE THE PORTS SYSTEM .........................................................................................- 436 HACK 85 DOWNGRADE A PORT ........................................................................................................- 441 HACK 86 CREATE YOUR OWN STARTUP SCRIPTS ...........................................................................- 445 HACK 87 AUTOMATE NETBSD PACKAGE BUILDS ............................................................................- 449 HACK 88 EASILY INSTALL UNIX APPLICATIONS ON MAC OS X........................................................- 453 CHAPTER 9. GROKKING BSD........................................................................................................ - 457 HACK 89 HOW'D HE KNOW THAT? ...................................................................................................- 459 HACK 90 CREATE YOUR OWN MANPAGES .......................................................................................- 462 HACK 91 GET THE MOST OUT OF MANPAGES .................................................................................- 466 HACK 92 APPLY, UNDERSTAND, AND CREATE PATCHES .................................................................- 470 HACK 93 DISPLAY HARDWARE INFORMATION ..................................................................................- 476 HACK 94 DETERMINE WHO IS ON THE SYSTEM ...............................................................................- 481 HACK 95 SPELLING BEE ....................................................................................................................- 485 HACK 96 LEAVE ON TIME ..................................................................................................................- 489 HACK 97 RUN NATIVE JAVA APPLICATIONS .....................................................................................- 492 HACK 98 ROTATE YOUR SIGNATURE................................................................................................- 495 HACK 100 FUN WITH X......................................................................................................................- 501 -

-4-

Credits About t he Aut hor Cont ribut ors Acknowledgm ent s

About the Author Dru Lavigne is t he aut hor of ONLam p.com 's FreeBSD Basics colum n and has been an avid BSD user since FreeBSD 2.2.1. As an I T inst ruct or, she specializes in net working, rout ing, and securit y. She is also responsible for I SECOM's Prot ocol Dat abase, which can be found at ht t p: / / www.isecom .org.

Contributors The following people cont ribut ed t heir hacks, writ ing, and inspirat ion t o t his book: •

John Richard, known locally as JR, is a syst em adm inist rat or in Kingst on, Ont ario, Canada. His t radem ark in t he field is his insist ence on a FreeBSD box as t he prim ary firewall on a net work. He has enj oyed working wit h t he aut hor in t he past at a privat e college in Kingst on. I n his spare t im e, he experim ent s wit h FreeBSD and rides his Harley- Davidson. [ H a ck # 6 4 ]

•

Joe Warner is a Technical Analyst for Siem ens Medical Solut ions Healt h Services Corporat ion and has been using FreeBSD as a server and deskt op since Oct ober of 2000. Joe has lived in Salt Lake Cit y, Ut ah for m ost of his life and enj oys * BSD, com put ing, hist ory, and The Mat rix. [ H a ck s # 3 5 a nd # 5 9 ]

•

Dan Langille ( ht t p: / / www.langille.org/ ) runs a consult ing group in Ot t awa, Canada. He has fond m em ories of his years in New Zealand, where t he clim at e is m uch m ore conducive t o year- round m ount ain biking. He lives in a house ruled by felines. [ H a ck # 4 1 ]

•

Robert Bernier's professional career has included engineering, accident invest igat ion, and Olym pic t rials. I n t he 1980s, his int erest ret urned t o I T when he realized he wouldn't have t o use a punch card anym ore. Event ually he discovered Linux and by t he m id- 1990s had developed a passion for all t hings open source. Today, Robert t eaches at t he local com m unit y college and writ es for a num ber of I T publicat ions based in Nort h Am erica and Europe. [ H a ck # 1 2 ]

•

Kirk Russell ( [email protected] ) is a kernel t est er at QNX Soft ware Syst em s ( ht t p: / / www.qnx.com / ) . [ H a ck # 3 6 ]

-5-

•

Karl Vogel is a syst em adm inist rat or for t he C- 17 Program Office. He's worked at Wright - Pat t erson Air Force Base for 22 years and has a BS in Mechanical & Aerospace Engineering from Cornell Universit y. [ H a ck # 3 2 ]

•

Howard Owen discovered com put ers by reading about Conway's " Life" in Life m agazine. I t t ook m any years from t hat discovery t o t he t im e he could act ually m ake a living wit h t he godforsaken t hings. Once t hat happened, however, Howard t urned int o a " m aj or geek." He has worked as a sysadm in, syst em s engineer, and syst em s archit ect . He is current ly em ployed by I BM in Silicon Valley support ing Linux, but he st ill runs FreeBSD and OpenBSD at hom e. [ H a ck s # 6 1 a nd # 6 2 ]

•

Daniel Harris is a st udent and occasional consult ant in West Virginia. He is int erest ed in com put er net working, docum ent at ion, and securit y; he also enj oys writ ing, arm chair polit ics, and am at eur radio. [ H a ck # 5 5 ]

•

Andrew Gould, CPA, perform s financial and clinical dat a analysis for a hospit al in Texas. His prim ary t ool for dat a int egrat ion is a Post greSQL dat abase server running on FreeBSD. Andrew has been using FreeBSD at bot h work and hom e for four years. Andrew has a BS in Educat ion and a BBA in Account ing from t he Universit y of Texas at Aust in. [ H a ck s # 1 7 2 .6 , # 4 0 , # 4 4 , a n d # 6 8 ]

•

Jim Mock is a FreeBSD adm in and developer t urned Mac OS X user and developer. He's a FreeBSD com m it t er, as well as an OpenDarwin com m it t er, and he current ly m aint ains 50+ DarwinPort s. Jim is also a m em ber of t he DarwinPort s Port Manager t eam . He can be reached at j im @bsdnews.org or t hrough his personal sit e at ht t p: / / soupnazi.org/ . [ H a ck # 8 8 ]

•

Avleen Vig is a syst em s adm inist rat or at Eart hLink ( ht t p: / / www.eart hlink.net / ) , where he m aint ains t he com pany's web, m ail, news, and ot her I nt ernet services for over 8 m illion users. He spends his spare t im e wit h his newborn son, cont ribut ing t o t he various I nt ernet and Unix com m unit ies, and enj oying life. Aft er seizing t he day in 2001 and m oving t o LA from London, he's wait ing t o see where life will t ake him next . [ H a ck # 6 9 ]

•

Alexandru Popa is a CCNA st udying for a CCNP, and is act ively involved in t he FreeBSD com m unit y in his spare t im e. At t he t im e of t his writ ing, he was st udying Com put er Science at t he Polit echnica Universit y of Bucharest . He also m aint ains cvsup.ro.freebsd.org out of a basem ent in a desert ed building, using a large ham st er array for power. He can be cont act ed at [email protected]. [ H a ck # 7 0 ]

•

Jens Schweikhardt is a Germ an soft ware engineer and I nt ernet wizard who is const ant ly looking for int erest ing t hings t o do. As a seven- t im e I OCCC winner, he is well- known for t aking C com pilers t o t heir lim it s. He cont ribut es t o Unix

-6-

st andardizat ion and, of course, t o God's Own Operat ing Syst em . When not hacking, Jens has been caught writ ing rom ant ic poet ry and riding his I t alian Mot o Guzzi around t he Swabian hills and valleys. I f he were given one m odest wish, it would be clear skies when he goes st argazing wit h his t elescope. [ H a ck # 7 8 ] •

Mat t hew Seam an is 38 years old and a form er scient ist and academ ic ( Oxford Universit y post graduat e) . He is now a specialist in com put er syst em adm inist rat ion, net work archit ect ure, and infrast ruct ure design. [ H a ck s # 4 9 , # 5 0 , a nd # 9 7 ]

•

Nat han Rosenquist first t ried FreeBSD in 1996, and has been using Unix ever since. During t he day, he can be found developing Perl- based web applicat ions and business aut om at ion soft ware. He lives in Shadow Hills, California wit h his girlfriend Carrie and t heir dog Nut m eg. [ H a ck # 3 9 ]

•

Adrian Mayo ( ht t p: / / unix.1dot 1.com / ) has worked wit h com put ers for 20 years, specializing in t he design of safet y and m ission- crit ical soft ware for t he aerospace and m edical indust ries. He has gained exposure t o BSD Unix t hrough Apple's Mac OS X operat ing syst em . He is Edit or for t he news and support sit e ht t p: / / www.osxfaq.com , writ ing m ost of t he t echnical cont ent , including t he Unix t ut orials and Daily Unix t ips. [ H a ck s # 1 4 , # 1 5 , a nd # 1 6 ]

•

Sebast ian St ark ( [email protected]) works as a syst em adm inist rat or at t he Max Planck I nst it ut e for Biological Cybernet ics in Germ any. He m anages a bunch of workst at ions, as well as a com put er clust er t hat is used for m achine- learning research. [ H a ck # 5 2 ]

•

Marlon Berlin ( m [email protected]) st udies linguist ics, com parat ive lit erat ure, and m at hem at ics in Berlin. He works for DNS: NET, a Germ an I SP, as a syst em s developer. [ H a ck # 5 2 ]

•

David Maxwell ( david@net bsd.org) is a Net BSD Developer and m em ber of t he Net BSD Securit y- Officer t eam . He at t ended Unix Unanim ous in Toront o since t he first m eet ing in t he early ` 80s, and st ill visit s when he can. He was an avid Am iga user, and relishes a good ( or bad) pun when he can m ust er one. David current ly works at I nt egrat ed Device Technology, I nc. ( I DT) . [ H a ck s # 1 0 , # 5 3 , # 7 3 , # 7 5 , a nd # 7 6 ]

-7-

•

Julio Merino Vidal is st udying I nform at ics Engineering at t he UPC Universit y of Barcelona, Spain. He has been a Net BSD developer since Novem ber 2002, working on t he Net BSD Packages Collect ion ( ht t p: / / www.pkgsrc.org/ ) and t ranslat ing t he web sit e t o Spanish. He also m aint ains his own free soft ware proj ect s, including Buildt ool ( ht t p: / / buildt ool.sourceforge.net / ) . You can cont act him at j m m v@Net BSD.org. [ H a ck s # 2 7 a nd # 8 7 ]

•

Jan L. Pet erson ( j lp@pet erson.at h.cx) is a professional syst em adm inist rat or wit h 16 years of experience working wit h m ult iple Unix versions ( and t he occasional Windows m achine) . Laid off from his last j ob when t he com pany was acquired by a direct com pet it or, he has spent t he last couple of years as a consult ant . More about Jan can be found at ht t p: / / www.pet erson.at h.cx/ ~ j lp/ . [ H a ck # 7 4 ]

•

Michael Vince was born in 1977. His init ial int erest in com put ers was video gam es, but he soon vent ured int o m any ot her areas, such as program m ing, Unix, t he Web, and net works. Having com plet ed a Diplom a in Com put er Syst em s and a CCNA, he is an I T adm inist rat or for soft ware com panies and has been involved in large soft ware proj ect s t hat put his developm ent skills t o good use. A t ech news j unkie, he is always int erest ed in t he fut ure of com put ing. He also enj oys st aying up lat e solving difficult problem s t hat require com plex regular expressions in Perl, going t o t he gym , and hanging out in cafes. He is current ly working on a soft ware product called Ezm in. [ H a ck # 6 4 ]

•

Daniel Carosone has been involved wit h Net BSD as a user, advocat e, and developer for over 10 years. He is a m em ber of t he Net BSD Securit y Officer t eam , which provides leadership for securit y m at t ers wit hin t he proj ect and coordinat es responses t o public incident s and vulnerabilit ies. He is Chief Technologist for e- Secure, specializing in securit y consult ing and m anagem ent services t o financial, governm ent , and t elecom m unicat ions organizat ions. He prom ot es securit y awareness t hrough conference present at ions and universit y lect ures. He lives in Melbourne, Aust ralia, and—when not working t oo hard—enj oys hiking, driving, and ast ronom y. [ H a ck # 6 0 ]

•

Aaron Crandall, BSEE, has used OpenBSD since 2.7. He current ly works for t he Oregon Graduat e I nst it ut e running com put ers as a part - t im e Mast er's st udent . He's built and given away m ore OpenBSD firewalls t han he can count . Cont act him at [email protected]. [ H a ck # 4 5 ]

•

chrom at ic is t he Technical Edit or of t he O'Reilly Net work. I n pract ice, t hat m eans he edit s ONLam p.com ( open source adm inist rat ion and developm ent ) and, occasionally, books like t his one. Out side of work, he enj oys cooking and som ehow produces a whole slew of weird soft ware hacks like SDL Parrot , t iny m ail t ools, and t hat Perl 6 t hing. Wade t hrough t he disarray of his web sit e at ht t p: / / wgz.org/ chrom at ic/ . [ H a ck # 9 2 ]

-8-

•

Bret t Warden, BSEE, specializes in Perl program m ing and em bedded syst em s. He lives in t he Nort hwest wit h his wife, son, and t wo ant isocial cat s. He's current ly keeping an eye out for cont ract ing and perm anent posit ions. You can find a collect ion of odd proj ect s at ht t p: / / www.wgz.org/ bwarden/ . [ H a ck # 6 5 ]

Acknowledgments I would like t o t hank t he m any BSD and open source users who so willingly shared t heir experiences, ideas, and support . You serve as a const ant rem inder t hat BSD is m ore t han an operat ing syst em —it is a com m unit y. I would also like t o t hank all of m y st udent s and t he readers of t he FreeBSD Basics colum n. Your quest ions and feedback fuel m y curiosit y; m ay t his book ret urn t hat favor. Thanks t o David Lent s and Rob Flickenger for reviews and advice. Special t hanks t o Jacek Art ym iak for his invaluable input from t he OpenBSD and Net BSD perspect ives. And finally, special t hanks t o chrom at ic. A writ er couldn't have asked for a bet t er edit or.

-9-

Preface " What was it about UNI X t hat won m y heart ? . . . UNI X is m yst erious when you first approach. A lit t le int im idat ing, t oo. But despit e an unadorned and oft en plain present at ion, t he discerning suit or can t ell t here's lot going on under t he surface." —Thom as Scoville, ht t p: / / unix.oreilly.com / news/ unix_love_0299.ht m l When t he above- m ent ioned art icle was first published, I was st ill very m uch a BSD newbie. My spare hours were spent st ruggling wit h kernel recom piles, PPP connect ivit y ( or lack t hereof) , rm and chmod disast ers, and reading and rereading every bit of t he t hen available docum ent at ion. Yet , t hat art icle gave voice t o m y experience, for, like t he quot ed aut hor, I had st um bled upon operat ing syst em love. I n ot her words, I was discovering how t o hack on BSD. Since t hen, I 've learned t hat t here is an unspoken com m onalit y bet ween t he novice Unix user and t he seasoned guru. I t doesn't m at t er whet her you've j ust survived your first successful inst allat ion or you've j ust execut ed a com plex script t hat will save your com pany t im e and m oney, t he feeling is t he sam e. I t 's t he excit em ent of vent uring int o unknown t errit ory and discovering som et hing new and wonderful. I t 's t hat sense of accom plishm ent t hat com es wit h figuring som et hing out for yourself, wit h finding your own solut ion t o t he problem at hand. This book cont ains 100 hacks writ t en by users who love hacking wit h BSD. You'll find hacks suit ed t o bot h t he novice user and t he seasoned vet eran, as well as everyone in bet ween. Read t hem in any order t hat suit s your purpose, but keep t he " onion principle" in m ind. While each hack does present at least one pract ical solut ion t o a problem , t hat 's j ust t he out er layer. Use your im aginat ion t o peel away deeper layers, exposing new solut ions as you do so.

Why BSD Hacks? The t erm hacking has an unfort unat e reput at ion in t he popular press, where it oft en refers t o som eone who breaks int o syst em s or wreaks havoc wit h com put ers. Am ong ent husiast s, on t he ot her hand, t he t erm hack refers t o a " quick- n- dirt y" solut ion t o a problem or a clever way t o do som et hing. The t erm hacker is very m uch a com plim ent , praising som eone for being creat ive and having t he t echnical chops t o get t hings done. O'Reilly's Hacks series is an at t em pt t o reclaim t he word, docum ent t he ways people are hacking ( in a good way) , and pass t he hacker et hic of creat ive part icipat ion on t o a new generat ion of hackers. Seeing how ot hers approach syst em s and problem s is oft en t he quickest way t o learn about a new t echnology. BSD Hacks is all about m aking t he m ost of your BSD syst em . The BSDs of t oday have a proud lineage, t racing back t o som e of t he original hackers—people who built Unix and t he I nt ernet as we know it t oday. As you'd expect , t hey faced m any problem s and solved problem s bot h quickly and elegant ly. We've collect ed som e of t hat wisdom , bot h classic and m odern, about using t he com m and line, securing syst em s, keeping t rack of your files, m aking backups, and, m ost im port ant ly, how t o becom e your own BSD guru along t he way.

How to Use this Book One of t he beaut ies of Unix is t hat you can be very product ive wit h surprisingly lit t le knowledge. Even bet t er, each new t rick you learn can shave m inut es off of your day. We've arranged t he chapt ers in t his book by subj ect area, not by any suggest ed order of learning.

- 10 -

Skip around t o what int erest s you m ost or solves your current problem . I f t he current hack depends on inform at ion in anot her hack, we'll include a link for you t o follow. Furt herm ore, t he " See Also" sect ions at t he end of individual hacks oft en include references such as man fortune. These refer t o t he m anual pages installed on your m achine. I f you're not fam iliar wit h t hese m anpages, st art wit h [ H a ck # 8 9 ] .

How This Book Is Organized To m ast er BSD, you'll have t o underst and several t opics. We've arranged t he hacks loosely int o chapt ers. They are:

Chapt er 1Cust om izing t he User Environm ent Though m odern BSDs have m yriad graphical applicat ions and ut ilit ies, t he com bined wisdom of 35 years of com m and- line program s is j ust a shell away. This chapt er dem onst rat es how t o m ake t he m ost of t he com m and line, cust om izing it t o your needs and preferences.

Chapt er 2Dealing wit h Files and Filesyst em s What good is knowing Unix com m ands if you have no files? You have t o slice, dice, and st ore dat a som ewhere. This chapt er explains t echniques for finding and processing inform at ion, whet her it 's on your m achine or on a server elsewhere.

Chapt er 3The Boot and Login Environm ent s The best - laid securit y plans of adm inist rat ors oft en go out t he window when users ent er t he pict ure. Keeping t he bad guys off of sensit ive m achines requires a t wopronged approach: prot ect ing norm al user account s t hrough good password policies and prot ect ing t he boxes physically. This chapt er explores several opt ions for cust om izing and securing t he boot and login processes.

Chapt er 4Backing Up Aft er you st art creat ing files, you're bound t o run across dat a you can't afford t o lose. That 's where backups com e in. This chapt er offers several ideas for various m et hods of ensuring t hat your precious dat a will persist in t he face of t ragedy.

Chapt er 5Net working Hacks Unless you're a die- hard individualist , you're likely connect ed t o a net work. That fact present s several new opport unit ies for clever hacks as well as m yst ifying failures. This chapt er illum inat es ways t o t ake advant age of your net work connect ion.

Chapt er 6Securing t he Syst em Securit y is as m uch a m indset as it is a process. Knowing t he t ools at your disposal will help. This chapt er delves int o m ult iple t ools and ideas for increasing t he securit y of your syst em s, whet her keeping out t he bad guys or st aying on t op of updat es.

- 11 -

Chapt er 7Going Beyond t he Basics Wit h years and years of refinem ent , t he BSDs provide powerful and m aint ainable environm ent s. Are you t aking full advant age of everyt hing your syst em has t o offer? This chapt er pushes t he envelope of what you can accom plish.

Chapt er 8Keeping Up- t o- Dat e No bragging about BSD is com plet e wit hout m ent ioning t he port s or packages syst em t hat keeps t housands of applicat ions right at your fingert ips. Keeping up- t odat e could never be easier, could it ? This chapt er t ackles t he subj ect of inst alling and updat ing soft ware, including t he core syst em .

Chapt er 9Grokking BSD You cannot be a t rue BSD m ast er unt il you grok t he Unix m indset . How did t he gurus becom e gurus? I s t he t rue pat h st ill open? This chapt er reveals som e secret s of t he m ast ers and has a lit t le fun along t he way.

Conventions Used in This Book This book uses t he following t ypographical convent ions:

I t alic I ndicat es new t erm s, URLs, em ail addresses, filenam es, pat hnam es, and direct ories.

Constant width I ndicat es com m ands, opt ions, swit ches, variables, at t ribut es, funct ions, user and group nam es, t he cont ent s of files, and t he out put from com m ands.

Constant width bold I n code exam ples, shows com m ands or ot her t ext t hat should be t yped lit erally by t he user.

Constant width italic Shows t ext t hat should be replaced wit h user- supplied values.

Color The second color is used t o indicat e a cross- reference wit hin t he t ext .

- 12 -

This icon signifies a t ip, suggest ion, or general not e.

This icon indicat es a warning or caut ion.

The t herm om et er icons, found next t o each hack, indicat e t he relat ive com plexit y of t he hack:

beginner

m oderat e

expert

Using Code Examples This book is here t o help you get your j ob done. I n general, you m ay use t he code in t his book in your program s and docum ent at ion. You do not need t o cont act us for perm ission unless you're reproducing a significant port ion of t he code. For exam ple, writ ing a program t hat uses several chunks of code from t his book does not require perm ission. Selling or dist ribut ing a CD- ROM of exam ples from O'Reilly books does require perm ission. Answering a quest ion by cit ing t his book and quot ing exam ple code does not require perm ission. I ncorporat ing a significant am ount of exam ple code from t his book int o your product 's docum ent at ion does require perm ission. We appreciat e, but do not require, at t ribut ion. An at t ribut ion usually includes t he t it le, aut hor, publisher, and I SBN, for exam ple: "BSD Hacks by Dru Lavigne. Copyright 2004 O'Reilly Media, I nc., 0- 596- 00679- 9." I f you feel your use of code exam ples falls out side fair use or t he perm ission given here, feel free t o cont act us at perm [email protected] .

- 13 -

Chapter 1. Customizing the User Environment Sect ion 0. I nt roduct ion Sect ion 1. Get t he Most Out of t he Default Shell Sect ion 2. Useful t csh Shell Configurat ion File Opt ions Sect ion 3. Creat e Shell Bindings Sect ion 4. Use Term inal and X Bindings Sect ion 5. Use t he Mouse at a Term inal Sect ion 6. Get Your Daily Dose of Trivia Sect ion 7. Lock t he Screen Sect ion 8. Creat e a Trash Direct ory Sect ion 9. Cust om ize User Configurat ions Sect ion 10. Maint ain Your Environm ent on Mult iple Syst em s Sect ion 11. Use an I nt eract ive Shell Sect ion 12. Use Mult iple Screens on One Term inal

- 14 -

Hack 0 Introduction Users of open source ( ht t p: / / opensource.org) Unix operat ing syst em s are an int erest ing breed. They like t o poke under t he surface of t hings, t o find out how t hings work, and t o figure out new and int erest ing ways of accom plishing com m on com put ing t asks. I n short , t hey like t o " hack." While t his book concent rat es on t he BSDs, m any of t he hacks apply t o any open source operat ing syst em . Each hack is sim ply a dem onst rat ion of how t o exam ine a com m on problem from a slight ly different angle. Feel free t o use any of t hese hacks as a springboard t o your own cust om ized solut ion. I f your part icular operat ing syst em doesn't cont ain t he t ool used in t he solut ion, use a t ool t hat does exist , or invent your own! This chapt er provides m any t ools for get t ing t he m ost out of your working environm ent . You'll learn how t o m ake friends wit h your shell and how t o perform your m ost com m on t asks wit h j ust a few keyst rokes or m ouse clicks. You'll also uncover t ricks t hat can help prevent com m and- line disast ers. And, above all, you'll discover t hat hacking BSD is fun. So, pull your chair up t o your operat ing syst em of choice and let 's st art hacking.

- 15 -

Hack 1 Get the Most Out of the Default Shell

Be com e a spe e d da e m on a t t h e com m a n d lin e . For bet t er or for worse, you spend a lot of t im e at t he com m and line. I f you're used t o adm inist ering a Linux syst em , you m ay be dism ayed t o learn t hat bash is not t he default shell on a BSD syst em , for eit her t he superuser or regular user account s. Take heart ; t he FreeBSD superuser's default tcsh shell is also brim m ing wit h short cut s and lit t le t ricks designed t o let you breeze t hrough even t he m ost t edious of t asks. Spend a few m om ent s learning t hese t ricks and you'll feel right at hom e. I f you're new t o t he com m and line or consider yourself a t errible t ypist , read on. Unix m ight be a whole lot easier t han you t hink. Net BSD and OpenBSD also ship wit h t he C shell as t heir default shell. However, it is not always t he sam e tcsh, but oft en it s sim pler variant , csh, which doesn't support all of t he t ricks provided in t his hack. However, bot h Net BSD and OpenBSD provide a tcsh package in t heir respect ive package collect ions.

1.2.1 History and Auto-Completion I hat e t o live wit hout t hree keys: up arrow, down arrow, and Tab. I n fact , you can recognize m e in a crowd, as I 'm t he one m ut t ering loudly t o m yself if I 'm on a syst em t hat doesn't t reat t hese keys t he way I expect t o use t hem . tcsh uses t he up and down arrow keys t o scroll t hrough your com m and hist ory. I f t here is a golden rule t o com put ing, it should be: " You should never have t o t ype a com m and m ore t han once." When you need t o repeat a com m and, sim ply press your up arrow unt il you find t he desired com m and. Then, press Ent er and t hink of all t he keyst rokes you j ust saved yourself. I f your fingers fly fast er t han your eyes can read and you whiz past t he right com m and, sim ply use t he down arrow t o go in t he ot her direct ion. The Tab key was specifically designed for bot h t he lazy t ypist and t he t errible speller. I t can be painful wat ching som e people t ype out a long com m and only t o have it fail because of a t ypo. I t 's even worse if t hey haven't heard about hist ory, as t hey t hink t heir only choice is t o t ry t yping out t he whole t hing all over again. No wonder som e people hat e t he com m and line! Tab act ivat es aut o- com plet ion. This m eans t hat if you t ype enough let t ers of a recognizable com m and or file, tcsh will fill in t he rest of t he word for you. However, if you inst ead hear a beep when you press t he Tab key, it m eans t hat your shell isn't sure what you want . For exam ple, if I want t o run sockstat and t ype: % so

t hen press m y Tab key, t he syst em will beep because m ult iple com m ands st art wit h so. However, if I add one m ore let t er:

- 16 -

% soc

and t ry again, t he syst em will fill in t he com m and for m e: % sockstat

1.2.2 Editing and Navigating the Command Line There are m any m ore short cut s t hat can save you keyst rokes. Suppose I 've j ust finished edit ing a docum ent . I f I press m y up arrow, m y last com m and will be displayed at t he prom pt : % vi mydocs/today/verylongfilename

I 'd now like t o double- check how m any words and lines are in t hat file by running t his com m and: % wc mydocs/today/verylongfilename

I could pound on t he backspace key unt il I get t o t he vi port ion of t he com m and, but it would be m uch easier t o hold down t he Ct rl key and press a. That would bring m e t o t he very beginning of t hat com m and so I could replace t he vi wit h wc. For a m nem onic device, rem em ber t hat j ust as a is t he first let t er of t he alphabet , it also represent s t he first let t er of t he com m and at a tcsh prom pt . I don't have t o use m y right arrow t o go t o t he end of t he com m and in order t o press Ent er and execut e t he com m and. Once your com m and looks like it should, you can press Ent er. I t doesn't m at t er where your cursor happens t o be. Som et im es you would like your cursor t o go t o t he end of t he com m and. Let 's say I want t o run t he word count com m and on t wo files, and right now m y cursor is at t he first c in t his com m and: % wc mydocs/today/verylongfilename

I f I hold down Ct rl and press e, t he cursor will j um p t o t he end of t he com m and, so I can t ype in t he rest of t he desired com m and. Rem em ber t hat e is for end. Finally, what if you're in t he m iddle of a long com m and and decide you'd like t o st art from scrat ch, erase what you've t yped, and j ust get your prom pt back? Sim ply hold down Ct rl and press u for undo. I f you work in t he Cisco or PI X I OS syst em s, all of t he previous t ricks work at t he I OS com m and line.

Did you know t hat t he cd com m and also includes som e built - in short cut s? You m ay have heard of t his one: t o ret urn t o your hom e direct ory quickly, sim ply t ype:

- 17 -

% cd

That 's very convenient , but what if you want t o change t o a different previous direct ory? Let 's say t hat you st art out in t he / usr/ share/ doc/ en_US.I SO8859- 1/ books/ handbook direct ory, t hen use cd t o change t o t he / usr/ X11R6/ et c/ X11 direct ory. Now you want t o go back t o t hat first direct ory. I f you're anyt hing like m e, you really don't want t o t ype out t hat long direct ory pat h again. Sure, you could pick it out of your hist ory, but chances are you originally navigat ed int o t hat deep direct ory st ruct ure one direct ory at a t im e. I f t hat 's t he case, it would probably t ake you longer t o pick each piece out of t he hist ory t han it would be t o j ust t ype t he com m and m anually. Fort unat ely, t here is a very quick solut ion. Sim ply t ype: % cd -

Repeat t hat com m and and wat ch as your prom pt changes bet ween t he first and t he second direct ory. What , your prom pt isn't changing t o indicat e your current working direct ory? Don't worry, [ H a ck # 2 ] will t ake care of t hat .

1.2.3 Learning from Your Command History Now t hat you can m ove around fairly quickly, let 's fine- t une som e of t hese hacks. How m any t im es have you found yourself repeat ing com m ands j ust t o alt er t hem slight ly? The following scenario is one exam ple. Rem em ber t hat docum ent I creat ed? I nst ead of using t he hist ory t o bring up m y previous com m and so I could edit it , I m ight have found it quicker t o t ype t his: % wc !$ wc mydocs/today/verylongfilename 19

97

620 mydocs/today/verylongfilename

The !$ t ells t he shell t o t ake t he last param et er from t he previous com m and. Since t hat com m and was: % vi mydocs/today/verylongfilename

it replaced t he !$ in m y new com m and wit h t he very long filenam e from m y previous com m and. The ! ( or bang! ) charact er has several ot her useful applicat ions for dealing wit h previously issued com m ands. Suppose you've been ext rem ely busy and have issued several dozen com m ands in t he last hour or so. You now want t o repeat som et hing you did half an hour ago. You could keep t apping your up arrow unt il you com e across t he com m and. But why search yourself when ! can search for you? For exam ple, if I 'd like t o repeat t he com m and mailstats, I could give ! enough let t ers t o figure out which com m and t o pick out from m y hist ory: $ !ma

- 18 -

! will pick out t he m ost recent ly issued com m and t hat begins wit h ma. I f I had issued a man com m and som et im e aft er mailstats com m and, tcsh would find t hat inst ead. This would fix it t hough: % !mai

I f you're not int o t rial and error, you can view your hist ory by sim ply t yping: % history

I f you're really lazy, t his com m and will do t he sam e t hing: % h

Each com m and in t his hist ory will have a num ber. You can specify a com m and by giving ! t he associat ed num ber. I n t his exam ple, I 'll ask tcsh t o reissue t he mailstats com m and: % h 165

16:51

mailstats

166

16:51

sockstat

167

16:52

telnet localhost 25

168

16:54

man sendmail

% !165

1.2.4 Silencing Auto-Complete The last t ip I 'll m ent ion is for t hose of you who find t he syst em bell irrit at ing. Or perhaps you j ust find it frust rat ing t yping one let t er, t abbing, t yping anot her let t er, t abbing, and so on unt il aut o- com plet e works. I f I t ype: % ls -l b

t hen hold down t he Ct rl key while I press d: backups/

bin/

book/

boring.jpg

ls -l b

I 'll be shown all of t he b possibilit ies in m y current direct ory, and t hen m y prom pt will ret urn m y cursor t o what I 've already t yped. I n t his exam ple, if I want t o view t he size and perm issions of boring.j pg, I 'll need t o t ype up t o here:

- 19 -

% ls -l bor

before I press t he Tab key. I 'll leave it up t o your own im aginat ion t o decide what t he d st ands for.

1.2.5 See Also - man tcsh

- 20 -

Hack 2 Useful tcsh Shell Configuration File Options

M a k e t h e sh e ll a fr ie ndly pla ce t o w or k in . Now t hat you've had a chance t o m ake friends wit h t he shell, let 's use it s configurat ion file t o creat e an environm ent you'll enj oy working in. Your prom pt is an excellent place t o st art .

1.3.1 Making Your Prompt More Useful The default tcsh prom pt displays % when you're logged in as a regular user and hostname# when you're logged in as t he superuser. That 's a fairly useful way t o figure out who you're logged in as, but we can do m uch bet t er t han t hat . Each user on t he syst em , including t he superuser, has a .cshrc file in his hom e direct ory. Here are m y current prom pt set t ings: dru@~:grep prompt ~/.cshrc if ($?prompt) then set prompt = "%B%n@%~%b: "

That isn't t he default tcsh prom pt , as I 've been using m y favorit e cust om ized prom pt for t he past few years. The possible prom pt form at t ing sequences are easy t o underst and if you have a list of possibilit ies in front of you. That list is buried deeply wit hin man cshrc, so here's a quick way t o zero in on it : dru@~:man cshrc /prompt may include

Here I 've used t he / t o invoke t he m anpage search ut ilit y. The search st ring prompt may include brings you t o t he right sect ion, and is int uit ive enough t hat even m y rust y old brain can rem em ber it . I f you com pare t he form at t ing sequences shown in t he m anpage t o m y prom pt st ring, it reads as follows: set prompt = "%B%n@%~%b: "

That 's a lit t le dense. Table 1- 1 dissect s t he opt ions.

- 21 -

Ta ble 1 - 1 . Pr om pt ch a r a ct e r s Ch a r a ct e r

Ex pla na t ion

"

St art s t he prom pt st ring.

%B

Turns on bold.

%n

Shows t he login nam e in t he prom pt .

@

I use t his as a separat or t o m ake m y prom pt m ore visually appealing.

%~

Shows t he current working direct ory. I t result s in a short er prom pt t han %/, as m y hom e direct ory is short ened from /usr/home/myusername t o ~

%b

Turns off bold.

:

Again, t his is an ext ra charact er I use t o separat e m y prom pt from t he cursor.

"

Ends t he prom pt st ring.

Wit h t his prom pt , I always know who I am and where I am . I f I also needed t o know what m achine I was logged int o ( useful for rem ot e adm inist rat ion) , I could also include %M or %m som ewhere wit hin t he prom pt st ring.

Switching to the Superuser The superuser's .cshrc file ( in / root , t he superuser's hom e direct ory) has an ident ical prom pt st ring. This is very fort unat e, as it reveals som et hing you m ight not know about t he su com m and, which is used t o swit ch users. Right now I 'm logged in as t he user dru and m y prom pt looks like t his: dru@/usr/ports/net/ethereal:

Wat ch t he shell out put carefully aft er I use su t o swit ch t o t he root user: dru@/usr/ports/net/ethereal: su Password: dru@/usr/ports/net/ethereal:

Things seem even m ore confusing if I use t he whoami com m and: dru@/usr/ports/net/ethereal: whoami dru

However, t he id com m and doesn't lie: dru@/usr/ports/net/ethereal: id uid=0(root) gid=0(wheel) groups=0(wheel), 5(operator)

- 22 -

I t t urns out t hat t he default invocat ion of su doesn't act ually log you in as t he superuser. I t sim ply gives you superuser privileges while ret aining your original login shell. I f you really want t o log in as t he superuser, include t he login ( -l) swit ch: dru@/usr/ports/net/ethereal: su -l Password: root@~: whoami root root@~: id uid=0(root) gid=0(wheel) groups=0(wheel), 5(operator)

I highly recom m end you t ake som e t im e t o experim ent wit h t he various form at t ing sequences and hack a prom pt t hat best m eet s your needs. You can add ot her feat ures, including cust om ized t im e and dat e st rings and com m and hist ory num bers [ H a ck # 1 ] , as well as flashing or underlining t he prom pt .

1.3.2 Setting Shell Variables Your prom pt is an exam ple of a shell variable. There are dozens of ot her shell variables you can set in .cshrc. My t rick for finding t he shell variables sect ion in t he m anpage is: dru@~:man cshrc /variables described

As t he nam e im plies, shell variables affect only t he com m ands t hat are built int o t he shell it self. Don't confuse t hese wit h environm ent variables, which affect your ent ire working environm ent and every com m and you invoke. I f you t ake a look at your ~ / .cshrc file, environm ent variables are t he ones writ t en in uppercase and are preceded wit h t he setenv com m and. Shell variables are writ t en in lowercase and are preceded wit h t he set com m and. You can also enable a shell variable by using t he set com m and at your com m and prom pt . ( Use unset t o disable it .) Since t he variable affect s only your current login session and it s children, you can experim ent wit h set t ing and unset t ing variables t o your heart 's cont ent . I f you get int o t rouble, log out of t hat session and log in again. I f you find a variable you want t o keep perm anent ly, add it t o your ~ / .cshrc file in t he sect ion t hat cont ains t he default set com m ands. Let 's t ake a look at som e of t he m ost useful ones. I f you enj oyed Ct rl- d from [ H a ck # 1 ] , you'll like t his even bet t er: set autolist

- 23 -

Now whenever you use t he Tab key and t he shell isn't sure what you want , it won't beep at you. I nst ead, t he shell will show you t he applicable possibilit ies. You don't even have t o press Ct rl- d first ! The next variable m ight save you from possible fut ure peril: set rmstar

I 'll t est t his variable by quickly m aking a t est direct ory and som e files: dru@~:mkdir test dru@~:cd test dru@~/test:touch a b c d e

Then, I 'll t ry t o rem ove t he files from t hat t est direct ory: dru@~/test:rm * Do you really want to delete all files? [n/y]

Since m y prom pt t ells m e what direct ory I 'm in, t his t rick gives m e one last chance t o double- check t hat I really am delet ing t he files I want t o delet e. I f you're prone t o t ypos, consider t his one: set correct=all

This is how t he shell will respond t o t ypos at t he com m and line: dru@~:cd /urs/ports CORRECT>cd /usr/ports (y|n|e|a)?

Pressing y will correct t he spelling and execut e t he com m and. Pressing n will execut e t he m isspelled com m and, result ing in an error m essage. I f I press e, I can edit m y com m and ( alt hough, in t his case, it would be m uch quicker for t he shell t o go wit h it s correct spelling) . And if I com plet ely panic at t he t hought of all of t hese choices, I can always press a t o abort and j ust get m y prom pt back. I f you like t o save keyst rokes, t ry: set implicitcd

You'll never have t o t ype cd again. I nst ead, sim ply t ype t he nam e of t he direct ory and t he shell will assum e you want t o go t here.

- 24 -

Hack 3 Create Shell Bindings

Tr a in you r sh e ll t o r un a com m a n d for you w h e n e ve r you pr e ss a m a ppe d k e y. Have you ever list ened t o a Windows power user expound on t he j oys of hot keys? Perhaps you yourself have been known t o gaze wist fully at t he ext ra but t ons found on a Microsoft keyboard. Did you know t hat it 's easy t o configure your keyboard t o launch your m ost com m only used applicat ions wit h a keyst roke or t wo? One way t o do t his is wit h t he bindkey com m and, which is built int o t he tcsh shell. As t he nam e suggest s, t his com m and binds cert ain act ions t o cert ain keys. To see your current m appings, sim ply t ype bindkey. The out put is several pages long, so I 've included only a short sam ple. However, you'll recognize som e of t hese short cut s from [ H a ck # 1 ] . Standard key bindings "^A"

->

beginning-of-line

"^B"

->

backward-char

"^E"

->

end-of-line

"^F"

->

forward-char

"^L"

->

clear-screen

"^N"

->

down-history

"^P"

->

up-history

"^U"

->

kill-whole-line

Arrow key bindings down

-> history-search-forward

up

-> history-search-backward

left

-> backward-char

right

-> forward-char

home

-> beginning-of-line

end

-> end-of-line

The ^ m eans hold down your Ct rl key. For exam ple, press Ct rl and t hen l, and you'll clear your screen m ore quickly t han by t yping clear. Not ice t hat it doesn't m at t er if you use t he uppercase or lowercase let t er.

- 25 -

1.4.1 Creating a Binding One of m y favorit e short cut s isn't bound t o a key by default : complete-word-fwd. Before I do t he act ual binding, I 'll first check which keys are available: dru@~:bindkey | grep undefined

"^G"

->

is undefined

"\305"

->

is undefined

"\307"

->

is undefined

Alt hough it is possible t o bind keys t o num erical escape sequences, I don't find t hat very convenient . However, I can very easily use t hat available Ct rl- g. Let 's see what happens when I bind it : dru@~:bindkey "^G" complete-word-fwd

When I t yped in t hat com m and, I knew som et hing worked because m y prom pt ret urned silent ly. Here's what happens if I now t ype ls -l /etc/, hold down t he Ct rl key, and repeat edly press g: ls -l /etc/COPYRIGHT ls -l /etc/X11 ls -l /etc/aliases ls -l /etc/amd.map

I now have a quick way of cycling t hrough t he files in a direct ory unt il I find t he exact one I want . Even bet t er, if I know what let t er t he file st art s wit h, I can specify it . Here I 'll cycle t hrough t he files t hat st art wit h a: ls -l /etc/a ls -l /etc/aliases ls -l /etc/amd.map ls -l /etc/apmd.conf ls -l /etc/auth.conf ls -l /etc/a

Once I 've cycled t hrough, t he shell will bring m e back t o t he let t er a and beep.

- 26 -

I f you prefer t o cycle backward, st art ing wit h words t hat begin wit h z inst ead of a, bind your key t o complete-word-back inst ead. When you use bindkey, you can bind any com m and t he shell underst ands t o any underst ood key binding. Here's m y t rick t o list t he com m ands t hat tcsh underst ands: dru@~ man csh /command is bound

And, of course, use bindkey alone t o see t he underst ood key bindings. I f you j ust want t o see t he binding for a part icular key, specify it . Here's how t o see t he current binding for Ct rl- g: dru@~:bindkey "^G" "^G"

->

complete-word-fwd

1.4.2 Specifying Strings What 's really cool is t hat you're not lim it ed t o j ust t he com m ands found in man csh. The s swit ch t o bindkey allow s you t o specify any st ring. I like t o bind t he lynx web browser t o Ct rl- w: dru@~:bindkey -s "^W" "lynx\n"

I chose w because it rem inds m e of t he World Wide Web. But why did I put \n aft er t he lynx? Because t hat t ells t he shell t o press Ent er for m e. That m eans by sim ply pressing Ct rl- w, I have inst ant access t o t he Web. Not e t hat I overwrit e t he default binding for Ct rl- w. This perm it s you t o m ake bindings t hat are m ore int uit ive and useful for your own purposes. For exam ple, if you never plan on doing what ever ^J does by default , sim ply bind your desired com m and t o it . There are m any pot ent ial key bindings, so scrolling t hrough t he out put of bindkeys can be t edious. I f you only st ick wit h " Ct rl let t er" bindings, t hough, it 's easy t o view your cust om izat ions wit h t he following com m and: dru@~:bindkey | head -n 28

As wit h all shell m odificat ions, experim ent wit h your bindings first by using bindkey at t he com m and prom pt . I f you get int o real t rouble, you can always log out t o go back t o t he default s. However, if you find som e bindings you want t o keep, m ake t hem perm anent by adding your bindkey st at em ent s t o your .cshrc file. Here is an exam ple: dru@~:cp ~/.cshrc ~/.cshrc.orig dru@~:echo 'bindkey "^G" complete-word-fwd' >> ~/.cshrc

Not ice t hat I backed up m y original .cshrc file first , j ust in case m y fingers slip on t he next part . I t hen used >> t o append t he echoed t ext t o t he end of .cshrc. I f I 'd used > inst ead, it

- 27 -

would have replaced m y ent ire .cshrc file wit h j ust t hat one line. I don't recom m end t est ing t his on any file you want t o keep. Along t hose lines, set t ing: set noclobber

will prevent t he shell from clobbering an exist ing file if you forget t hat ext ra > in your redirect or. You'll know you j ust prevent ed a nast y accident if you get t his error m essage aft er t rying t o redirect out put t o a file: .cshrc: File exists.

1.4.3 See Also • •

man tcsh [ H a ck # 2 ]

- 28 -

Hack 4 Use Terminal and X Bindings

Ta k e a dva n t a ge of you r t e r m ina l's ca pa bilit ie s. I t 's not j ust t he tcsh shell t hat is capable of underst anding bindings. Your FreeBSD t erm inal provides t he kbdcontrol com m and t o m ap com m ands t o your keyboard. Unfort unat ely, neit her Net BSD nor OpenBSD offer t his feat ure. You can, however, rem ap your keyboard under X, as described lat er.

1.5.1 Creating Temporary Mappings Let 's st art by experim ent ing wit h som e t em porary m appings. The synt ax for m apping a com m and wit h kbdcontrol is as follows: kbdcontrol -f number "command"

Table 1- 2 list s t he possible num bers, each wit h it s associat ed key com binat ion.

Ta ble 1 - 2 . Ke y n u m be r s N u m be r

Ke y com bin a t ion

1, 2, . . . 12

F1, F2, . . . F12

13, 14, . . . 24

Shift + F1, Shift + F2, . . . Shift + F12

25, 26, . . . 36

Ct rl+ F1, Ct rl+ F2, . . . Ct rl+ F12

37, 38, . . . 48

Shift + Ct rl+ F1, Shift + Ct rl+ F2, . . . Shift + Ct rl+ F12

49

Hom e

50

Up arrow

51

Page Up

52

Num pad - ( Num Lock off)

53

Left arrow ( also works in edit or)

54

Num pad 5 ( wit hout Num Lock)

55

Right arrow

56

Num pad + ( wit hout Num Lock)

57

End

58

Down arrow ( affect s c hist ory)

59

Page Down

60

I ns

61

Del

62

Left GUI key ( Windows icon next t o left Ct rl)

- 29 -

Ta ble 1 - 2 . Ke y n u m be r s N u m be r

Ke y com bin a t ion

63

Right GUI key ( Windows icon next t o right Alt )

64

Menu ( m enu icon next t o right Ct rl)

Those last t hree key com binat ions m ay or m ay not be present , depending upon your keyboard. My Logit ech keyboard has a key wit h a Windows icon next t o t he left Ct rl key; t hat is t he left GUI key. There's anot her key wit h a Windows icon next t o m y right Alt key; t his is t he right GUI key. The next key t o t he right has an icon of a cursor point ing at a square cont aining lines; t hat is t he Menu key. Now t hat we know t he possible num bers, let 's m ap lynx t o t he Menu key: % kbdcontrol -f 64 "lynx"

Not e t hat t he com m and m ust be cont ained wit hin quot es and be in your pat h. ( You could give an absolut e pat h, but t here's a nast y lim it at ion com ing up soon.) I f I now press t he Menu key, lynx is t yped t o t he t erm inal for m e. I j ust need t o press Ent er t o launch t he browser. This m ay seem a bit t edious at first , but it is act ually quit e handy. I t can save you from inadvert ent ly launching t he wrong applicat ion if you're anyt hing like m e and t end t o forget which com m ands you've m apped t o which keys. Let 's see what happens if I m odify t hat original m apping som ewhat : % kbdcontrol -f 64 "lynx www.google.ca" kbdcontrol: function key string too long (18 > 16)

When doing your own m appings, beware t hat t he com m and and it s argum ent s can't exceed 16 charact ers. Ot her t han t hat , you can pret t y well m ap any com m and t hat st rikes your fancy.

1.5.2 Shell Bindings Versus Terminal Bindings Before going any furt her, I 'd like t o pause a bit and com pare shell- specific bindings, which we saw in [ H a ck # 3 ] , and t he t erm inal- specific bindings we're running across here. One advant age of using kbdcontrol is t hat your cust om bindings work in any t erm inal, regardless of t he shell you happen t o be using. A second advant age is t hat you can easily m ap t o any key on your keyboard. Shell m appings can be com plicat ed if you want t o m ap t hem t o anyt hing ot her t han " Ct rl let t er" . However, t he t erm inal m appings have som e rest rict ions t hat don't apply t o t he tcsh m appings. For exam ple, shell m appings don't have a 16 charact er rest rict ion, allowing for full pat hnam es. Also, it was relat ively easy t o ask t he shell t o press Ent er t o launch t he desired com m and. Term inal bindings affect only t he current user's t erm inal. Any ot her users who are logged in on different t erm inals are not affect ed. However, if t he m appings are added t o rc.conf ( which only t he superuser can do) , t hey will affect all t erm inals. Since bindings are t erm inal

- 30 -

specific, even invoking su won't change t he behavior, as t he user is st ill st uck at t he sam e t erm inal.

1.5.3 More Mapping Caveats There are som e ot her caveat s t o consider when choosing which key t o m ap. I f you use t he t csh shell and enj oy viewing your hist ory [ H a ck # 1 ] , you'll be disappoint ed if you rem ap your up and down arrows. The right and left arrows can also be problem at ic if you use t hem for navigat ion, say, in a t ext edit or. Finally, if you're physically sit t ing at your FreeBSD syst em , F1 t hrough F8 are already m apped t o virt ual t erm inals and F9 is m apped t o your GUI t erm inal. By default , F10 t o F12 are unm apped. I f you st art experim ent ing wit h m appings and find you're st uck wit h one you don't like, you can quickly ret urn all of your keys t o t heir default m appings wit h t his com m and: % kbdcontrol -F

On t he ot her hand, if you find som e new m appings you absolut ely can't live wit hout , m ake t hem perm anent . I f you have superuser privileges on a FreeBSD syst em you physically sit at , you can carefully add t he m appings t o / et c/ rc.conf. Here, I 've added t wo m appings. One m aps lynx t o t he Menu key and t he ot her m aps startx t o t he left GUI key: keychange="64 lynx" keychange="62 startx"

Since t he superuser will be set t ing t hese m appings, t he m apped keys will affect all users on t hat syst em . I f you want t o save your own personal m appings, add your specific kbdcontrol com m ands t o t he end of your shell configurat ion file. For exam ple, I 've added t hese t o t he very end of m y ~ / .cshrc file, j ust before t he last line which says endif: % kbdcontrol -f 64 "lynx" % kbdcontrol -f 62 "startx"

1.5.4 Making Mappings Work with X This is all ext rem ely handy, but what will happen if you t ry one of your newly m apped keys from an X Window session? You can press t hat key all you want , but not hing will happen. You won't even hear t he sound of t he syst em bell beeping at you in prot est . This is because t he X prot ocol handles all input and out put during an X session. You have a few opt ions if you want t o t ake advant age of keyboard bindings while in an X GUI . One is t o read t he docum ent at ion for your part icular window m anager. Most of t he newer window m anagers provide a point and click int erface t o m anage keyboard bindings. My favorit e alt ernat ive is t o t ry t he xbindkeys_config applicat ion, which is available in t he port s collect ion [ H a ck # 8 4 ] : # cd /usr/ports/x11/xbindkeys_config # make install clean

- 31 -

This port also requires xbindkeys: # cd /usr/ports/x11/xbindkeys # make install clean Rat her t han building bot h port s, you could inst ead add t his line t o / usr/ port s/ x11/ xbindkeys_config/ Makefile: BUILD_DEPENDS=

xbindkeys:${PORTSDIR}/x11/xbindkeys

This will ask t he xbindkeys_config build t o inst all bot h port s.

Once your builds are com plet e, open an xterm and t ype: % xbindkeys --defaults

~/.xbindkeysrc

% xbindkeys_config

The GUI in Figure 1- 1 will appear.

Figu r e 1 - 1 . Th e x bin dk e ys_ con fig pr ogr a m

Creat ing a key binding is a sim ple m at t er of pressing t he New but t on and t yping a useful nam e int o t he Nam e: sect ion. Then, press Get Key and a lit t le window will appear. Press t he desired key com binat ion, and voilà, t he correct m apping required by X will aut ofill for you. Associat e your desired Act ion: , t hen press t he Save & Apply & Exit but t on. Any keyboard m appings you creat e using t his ut ilit y will be saved t o a file called ~ / .xbindkeysrc.

1.5.5 See Also man kbdcontrol, man atkbd , The xbindkeys web sit e ( ht t p: / / hocwp.free.fr/ xbindkeys/ xbindkeys.ht m l)

- 32 -

Hack 5 Use the Mouse at a Terminal

Use you r m ou se t o copy a n d pa st e a t a t e r m ina l. I f you're used t o a GUI environm ent , you m ight feel a bit out of your elem ent while working at t he t erm inal. Sure, you can learn t o m ap hot keys and t o use navigat ional t ricks, but darn it all, som et im es it 's j ust nice t o be able t o copy and past e! Don't fret ; your m ouse doesn't have t o go t o wast e. I n fact , depending upon how you have configured your syst em , t he m ouse daem on moused m ay already be enabled. The j ob of t his daem on is t o list en for m ouse dat a in order t o pass it t o your console driver. Of course, if you're using screen [ H a ck # 1 2 ] , you can also t ake advant age of it s copy and past e m echanism .

1.6.1 If X Is Already Installed I f you inst alled and configured X when you inst alled your syst em , moused is m ost likely st art ed for you when you boot up. You can check wit h t his: % grep moused /etc/rc.conf moused_port="/dev/psm0" moused_type="auto" moused_enable="YES"

Very good. moused needs t o know t hree t hings: • • •

The m ouse port ( in t his exam ple, / dev/ psm 0, t he PS/ 2 port ) The t ype of prot ocol ( in t his exam ple, auto) Whet her t o st art at boot t im e

I f you receive sim ilar out put , you're ready t o copy and past e. To copy t ext , sim ply select it by clicking t he left m ouse but t on and dragging. Then, place t he m ouse where you'd like t o past e t he t ext and click t he m iddle but t on. That 's it . To select an ent ire word, double- click anywhere on t hat word. To select an ent ire line, t riple- click anywhere on t hat line.

- 33 -

1 .6 .1 .1 Con figu r ing a t w o- bu t t on m ouse What if you don't have t hree m ouse but t ons? As t he superuser, add t he following line t o / et c/ rc.conf ( assum ing it 's not already t here) : moused_flags="-m 2=3"

This flag t ells moused t o t reat t he second, or right , m ouse but t on as if it were t he t hird, or m iddle, m ouse but t on. Now you can use t he right m ouse but t on t o past e your copied t ext . To apply t hat change, rest art moused: # /etc/rc.d/moused restart Stopping moused. Starting moused:.

Test your change by copying som e t ext wit h t he left m ouse but t on and past ing wit h t he right m ouse but t on.

1.6.2 If X Is Not Installed You can achieve t he sam e result s on a syst em wit hout X inst alled. You'll have t o add t he lines t o / et c/ rc.conf m anually, t hough. The exam ple I 've given you is for a PS/ 2 m ouse. I f you're using anot her t ype of m ouse, read t he " Configuring Mouse Daem on" sect ion of man moused. I t gives explicit det ails on figuring out what t ype of m ouse you have and what t ype of prot ocol it underst ands. I t even includes a sect ion on configuring a lapt op syst em for m ult iple m ice: one for when on t he road and one for when t he lapt op is at t ached t o t he docking st at ion. For exam ple, if you're using a USB m ouse, t he only difference is t hat t he port is /dev/usm0 inst ead of /dev/psm0. A serial m ouse physically plugged int o COM1 would be /dev/cuaa0. You m ay have t o experim ent wit h t he t ype, as auto doesn't work wit h all serial m ice. Again, t he m anpage is your best reference.

1.6.3 See Also • • •

man moused Docum ent at ion on enabling m ouse support in Net BSD at ht t p: / / www.net bsd.org/ Docum ent at ion/ wscons/ Docum ent at ion on enabling m ouse support in OpenBSD at ht t p: / / www.openbsd.org/ faq/ faq7.ht m l)

- 34 -

Hack 6 Get Your Daily Dose of Trivia

Br igh t e n you r da y w it h som e t e r m ina l e ye ca n dy. As t he saying goes, all work and no play m akes Jack a dull boy. But what 's a poor Jack or Jill t o do if your days include spending inordinat e am ount s of t im e in front of a com put er screen? Well, you could head over t o ht t p: / / www.t hinkgeek.net / t o st ock up on cube goodies and caffeine. Or, you could t ake advant age of som e of t he ent ert ainm ent s built int o your operat ing syst em .

1.7.1 A Fortune a Day Let 's st art by configuring som e t erm inal eye candy. Does your syst em quot e you a cheery, wit t y, or downright st range bit of wisdom every t im e you log int o your t erm inal? I f so, you're receiving a fort une: login: dru Password: Last login: Thu Nov 27 10:10:16 on ttyv7

"You can't have everything. Where would you put it?" -- Steven Wright

I f you're not receiving a fort une, as t he superuser t ype /stand/sysinstall. Choose Configure, t hen Distributions, and select games wit h your spacebar. Press Tab t o select OK, t hen exit out of sysinstall when it is finished. Then, look for t he line t hat runs / usr/ gam es/ fort une in your ~ / .cshrc file: % grep fortune ~/.cshrc /usr/games/fortune

I f for som e reason it isn't t here, add it : % echo '/usr/games/fortune' >> ~/.cshrc

Don't forget t o use bot h great er- t han signs; you don't want t o erase t he cont ent s of your .cshrc file! To t est your change, use t he source shell com m and, which re- execut es t he cont ent s of t he file. This can com e in handy if you've updat ed an alias and want t o t ake advant age of it im m ediat ely: % source ~/.cshrc Indifference will be the downfall of mankind, but who cares?

- 35 -

I f you'd also like t o receive a fort une when you log out of your t erm inal, add t his line t o t he end of your .logout file. I f you don't have one, and t here isn't one by default , you can creat e it and add t his line in one st ep: % echo '/usr/games/fortune' > ~/.logout

Not e t hat t his t im e I used only one great er- t han sign, as I was creat ing t he file from scrat ch. I f t he file already exist s, use t wo great er- t han signs t o append your new line t o t he end of t he exist ing file. Believe it or not , fortune com es wit h swit ches, som e of which are m ore am using t han ot hers. I 'll leave it t o you t o peruse man fortune.

1.7.2 Pursuing Trivia I 'm a t rivia buff, so I love using t he calendar com m and. Cont rary t o logic, t yping calendar won't show m e t his m ont h's calendar ( t hat 's t he j ob of cal) . However, I will get an inst ant dose of t rivia, relat ed t o t he current dat e: % calendar Nov 27

Alfred Nobel establishes Nobel Prize, 1895

Nov 27

Friction match invented, England, 1826

Nov 27

Hoosac Railroad Tunnel completed, 1873, in NW Massachusetts

Nov 28

Independence Day in Albania and Mauritania

Nov 28

Independence from Spain in Panama

Nov 28

Proclamation of the Republic in Chad

Nov 27

Jimi Hendrix (Johnny Allen Hendrix) is born in Seattle, 1942

Cool. I had forgot t en it was t he anniversary of t he Hoosac t unnel, an event t hat put m y hom et own on t he m ap. I t 's an easy m at t er t o aut om at e t he out put provided by calendar. I f you want t o see your t rivia when you log in or log out , sim ply add a line t o your .cshrc or .logout file. Because t he line you add is really j ust a pat h t o t he program , use t he out put of t he which com m and t o add t hat line for you: % echo `which calendar` >> .cshrc

Again, don't forget t o append wit h >>, or have noclobber set in your .cshrc file [ H a ck # 2 ] .

1.7.3 Sundry Amusements Of course, t here are several ot her dat e and t im e relat ed m ini- hacks at your disposal. Here are t wo you m ight enj oy.

- 36 -

1 .7 .3 .1 The cu r r e nt t im e Ever wonder what t im e it is while you're working on t he t erm inal? Sure, you could use date, but t he out put is so sm all and boring. Try t his t he next t im e you want t o know what t im e it is: % grdc

Whoa, you can see t hat one from across t he room . That 's not a bad idea if you want t o send your cubicle buddy a hint . I 've been known t o add / usr/ gam es/ grdc t o m y ~ / .logout . When I log out , m y t erm inal displays t he t im e unt il I press Ct rl- c and log in again. That 's sort of a built - in password prot ect ed screen saver for t he t erm inal. 1 .7 .3 .2 The pha se of t h e m oon Have you ever read man pom? I t has one of t he m ore useful descript ions I 've seen: The pom ut ilit y displays t he current phase of t he m oon. Useful for select ing soft ware com plet ion t arget dat es and predict ing m anagerial behavior. Sounds like Dilbert had a hand in t hat one. I f I add t he line / usr/ gam es/ pom t o m y ~ / .cshrc, I 'll learn a bit about ast ronom y when I log in: % pom The Moon is Waxing Gibbous (53% of Full)

There's a one- liner t o prom ot e wat er cooler conversat ion.

1.7.4 Adding Some Color to Your Terminal Have you ever t ried t his com m and? % vidcontrol show

0

8 grey

1 blue

9 lightblue

2 green

10 lightgreen

3 cyan

11 lightcyan

4 red

12 lightred

5 magenta

13 lightmagenta

6 brown

14 yellow

7 white

15 lightwhite

- 37 -

Gee, t hat rem inds m e of m y old DOS days when I discovered ansi.sys. Yes, your t erm inal is capable of color and you're looking at your possible color schem es! ( I t likely looks m uch m ore excit ing on your t erm inal, since it 's not in color in t his book.) I f you see som e colors t hat appeal t o you, add t hem t o your t erm inal. For exam ple, t his com m and will set t he foreground color t o yellow and t he background color as blue: % vidcontrol yellow blue

Not e t hat you can use only colors 1 t hrough 7 as background colors; you'll receive a synt ax error if you t ry t o use colors 8- 15 in your background. Try out t he various com binat ions unt il you find one t hat appeals t o your sense of t ast e. You can even add a border if you like: % vidcontrol -b red

These set t ings affect only your own t erm inal. I f you want , add t he desired vidcontrol lines t o your ~ / .cshrc file so your set t ings are available when you log int o your t erm inal. I f you have problem s finding your cursor, t ry: % vidcontrol -c blink

or: % vidcontrol -c destructive

Changing t he cursor affect s all virt ual t erm inals on t he syst em . I f ot her users com plain about your im provem ent , t his will bring t hings back t o norm al: % vidcontrol -c normal

1.7.5 See Also • • • •

man man man The

fortune calendar vidcontrol gam es packages, in Net BSD and OpenBSD

- 38 -

Hack 7 Lock the Screen

Se cu r e you r u na t t e nde d t e r m ina l fr om pr yin g e ye s. I f you work in a net worked environm ent , t he im port ance of locking your screen before leaving your workst at ion has probably been st ressed t o you. Aft er all, your brilliant password becom es m oot if anyone can walk up t o your logged in st at ion and st art poking about t he cont ent s of your hom e direct ory. I f you use a GUI on your workst at ion, your Window Manager probably includes a locking feat ure. However, if you use a t erm inal, you m ay not be aware of t he m echanism s available for locking your t erm inal. As an adm inist rat or, you m ay want t o aut om at e t hese m echanism s as part of your securit y policy. Fort unat ely, FreeBSD's screen locking m echanism is cust om izable.

1.8.1 Using lock FreeBSD com es wit h lock ( and it 's available for Net BSD and OpenBSD) . I t s default invocat ion is sim ple: % lock Key: 1234 Again: 1234 lock /dev/ttyv6 on genisis. timeout in 15 minutes. time now is Fri Jan 2 12:45:02 EST 2004 Key:

Wit hout any swit ches, lock will request t hat t he user input a key which will be used t o unlock t he t erm inal. This is a good t hing, as it gives t he user an opport unit y t o use som et hing ot her t han her login password. I f t he user t ries t o be sm art and presses Ent er ( for an em pt y password) , t he lock program will abort . Once a key is set , it is required t o unlock t he screen. I f a user inst ead t ypes Ct rl- c, she won't t erm inat e t he program . I nst ead, she'll receive t his m essage: Key: lock: type in the unlock key. timeout in 10:59 minutes

Did you not ice t hat t im eout value of 15 m inut es? At t hat t im e, t he screen will unlock it self, which sort a dim inishes t he usefulness of locking your screen. Aft er all, if you run int o your boss in t he hall, your 5- m inut e coffee break m ight t urn int o a 25- m inut e im prom pt u brainst orm ing session.

- 39 -

To lock t he t erm inal forever, or at least unt il som eone t ypes t he correct key, use t he -n swit ch. I f t he syst em is a personal workst at ion, -v is also handy; t his locks all of t he virt ual t erm inals on t he syst em , m eaning a passerby can't use Alt - Fn t o swit ch t o anot her t erm inal. As an adm inist rat or, you can assist users in using t he desired swit ches by adding an alias t o / usr/ share/ skel/ dot .cshrc [ H a ck # 9 ] . This alias rem oves t he t im eout and locks all t erm inals: alias lock

/usr/bin/lock -nv

1.8.2 Using autologout I f you use t he tcsh shell, you also have t he abilit y eit her t o lock your session or t o be logged out of your session aut om at ically aft er a set period of inact ivit y. As an adm inist rat or, you can set your policy by adding a line t o / usr/ share/ skel/ dot .cshrc. Do be aware, t hough, t hat a user can edit her own ~ / .cshrc file, which will negat e your cust om ized set t ing.

The autologout variable can accept t wo num bers. The first num ber represent s t he num ber of m inut es of inact ivit y before logging out t he user. The second num ber represent s t he num ber of m inut es of inact ivit y before locking t he user's screen. Once t he screen is locked, t he user m ust input t he password t o unlock it . I f t he screen is not unlocked in t im e, t he user will be logged out once t he shell has been idle for t he logout period of m inut es. The m anpage is pret t y vague on how t o set t hose t wo num bers. For exam ple, if you t ry: set autologout = 30 15

users will receive t his error m essage when t hey t ry t o log in: set: Variable name must begin with a letter.

That 's a decept ive error m essage, as t his variable does accept num erals. The correct invocat ion is t o enclose t he t wo num bers bet ween parent heses: set autologout = (30 15)

This part icular set t ing will log out a user aft er 15 m inut es of inact ivit y. The user will know t his happened as t he t erm inal will resem ble: % Password:

Aft er 30 m inut es of inact ivit y ( or 15 m inut es aft er t he screen was locked) , t he user will be logged out and see t his:

- 40 -

% Password:auto-logout Consider whet her or not your users t end t o run background j obs before globally im plem ent ing autologout. Also see [ H a ck # 1 1 ] , which allow s users t o reat t ach t o t heir t erm inals.

1.8.3 Enforcing Logout What if you do want t o enforce a logout policy t hat users can't change in t heir shell configurat ion files? Consider using idled, which can be inst alled from / usr/ port s/ sysut ils/ idled or built from source. This ut ilit y was designed t o log out users eit her aft er a configured period of inact ivit y or aft er t hey've been logged in for a cert ain am ount of t im e. Once you've inst alled idled, copy t he t em plat e configurat ion file: # cd /usr/local/etc/ # cp idled.cf.template idled.cf

Open / usr/ local/ et c/ idled.cf using your favorit e edit or. You'll find t his file t o be well com m ent ed and quit e st raight forward. You'll be able t o configure t he t im e before logout as well as when t he user will receive a warning m essage. I n addit ion, you can refuse logins, set session t im eout s, and provide for exem pt ions.

1.8.4 See Also • • • •

man man man The

lock tcsh man idled idled.cf idled web sit e ( ht t p: / / www.darkwing.com / idled/ )

- 41 -

Hack 8 Create a Trash Directory

Sa ve " de le t e d" file s u n t il you 'r e r e a lly r e a dy t o se nd t h e m t o t h e bit bu ck e t . One of t he first t hings Unix users learn is t hat delet ed files are really, really gone. This is especially t rue at t he com m and line where t here isn't any Windows- st yle recycling bin t o rum m age t hrough should you have a change of heart regarding t he fat e of a rem oved file. I t 's off t o t he backups! ( You do have backups, don't you?) Fort unat ely, it is very sim ple t o hack a sm all script t hat will send rem oved files t o a cust om t rash direct ory. I f you've never writ t en a script before, t his is an excellent exercise in how easy and useful script ing can be.

1.9.1 Shell Scripting for the Impatient Since a script is an execut able file, you should place your script s in a direct ory t hat is in your pat h. Rem em ber, your pat h is j ust a list of direct ories where t he shell will look for com m ands if you don't give t hem full pat hnam es. To see your pat h: % echo $PATH PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/ local/bin:/usr/X11R6/bin:/home/dru/bin

I n t his out put , t he shell will look for execut ables in t he bin subdirect ory of dru's hom e direct ory. However, it won't look for execut ables placed direct ly in m y hom e direct ory, or / hom e/ dru. Since bin isn't creat ed by default , I should do t hat first : % cd % mkdir bin

As I creat e script s, I 'll st ore t hem in / hom e/ dru/ bin, since I don't have perm ission t o st ore t hem anywhere else. Fort unat ely, no one else has perm ission t o st ore t hem in m y bin direct ory, so it 's a good m at ch. The script s t hem selves cont ain at least t hree lines: #!/bin/sh # a comment explaining what the script does the command to be executed

The first line indicat es t he t ype of script by specifying t he program t o use t o execut e t he script . I 've chosen t o use a Bourne script because t hat shell is available on all Unix syst em s.

- 42 -

Your script should also have com m ent s, which st art wit h t he # charact er. I t 's surprising how forget ful you can be six m ont hs down t he road, especially if you creat e a lot of script s. For t his reason, you should also give t he script a nam e t hat rem inds you of what it does. The t hird and subsequent lines cont ain t he m eat of t he script : t he act ual com m and( s) t o execut e. This can range from a sim ple one- liner t o a m ore com plex set of com m ands, variables, and condit ions. Fort unat ely, we can m ake a t rash script in a sim ple one- liner.

1.9.2 The Code Let 's st art wit h t his variant , which I found as t he result of a Google search: % more ~/bin/trash #!/bin/sh # script to send removed files to trash directory mv $1 ~/.trash/

You should recognize t he pat h t o t he Bourne shell, t he com m ent , and t he mv com m and. Let 's t ake a look at t hat $1. This is known as a posit ional param et er and specifically refers t o t he first param et er of t he trash com m and. Since t he mv com m ands t akes filenam es as param et ers, t he com m and: mv $1 ~/.trash/

is really saying, mv t he first filenam e, what ever it happens t o be, t o a direct ory called .t rash in t he user's hom e direct ory ( represent ed by t he shell short cut of ~) . This m ove operat ion is our cust om " recycle." Before t his script can do anyt hing, it m ust be set as execut able: % chmod +x ~/bin/trash

And I m ust creat e t hat t rash direct ory for it t o use: % mkdir ~/.trash

Not e t hat I 've chosen t o creat e a hidden t rash direct ory; any file or direct ory t hat begins wit h t he . charact er is hidden from norm al list ings. This really only reduces clut t er, t hough, as you can see t hese files by passing t he -a swit ch t o ls. I f you also include t he F swit ch, direct ory nam es will end wit h a / : % ls -aF ~ .cshrc

.history

.trash/

bin/

images/

myfile

- 43 -

1.9.3 Replacing rm with ~/bin/trash Now com es t he neat part of t he hack. I want t his script t o kick in every t im e I use rm. Since it is t he shell t hat execut es com m ands, I sim ply need t o m ake m y shell use t he trash com m and inst ead. I do t hat by adding t his line t o ~ / .cshrc: alias rm

trash

That line basically says: when I t ype rm, execut e trash inst ead. I t doesn't m at t er which direct ory I am in. As long as I st ay in m y shell, it will mv any files I t ry t o rm t o m y hidden t rash direct ory.

1.9.4 Running the Code Safely Whenever you creat e a script , always t est it first . I 'll st art by t elling m y shell t o reread it s configurat ion file: % source ~/.cshrc

Then, I 'll m ake som e t est files t o rem ove: % cd % mkdir test % cd test % touch test1 % rm test1

% ls ~/.trash test1

Looks like t he script is working. However, it has a flaw. Have you spot t ed it yet ? I f not , t ry t his: % touch a aa aaa aaaa % rm a*

% ls ~/.trash test1

a

% ls test aa

aaa

aaaa

- 44 -

What happened here? I passed t he shell m ore t han one param et er. The a* was expanded t o a, aa, aaa, and aaaa before trash could execut e. Those four param et ers were t hen passed on t o t he mv com m and in m y script . However, trash passes only t he first param et er t o t he mv com m and, ignoring t he rem aining param et ers. Fort unat ely, t hey weren't rem oved, but t he script st ill didn't achieve what I want ed. You can act ually have up t o nine param et ers, nam ed $1 t o $9. However, our goal is t o cat ch all param et ers, regardless of t he am ount . To do t hat , we use $@: mv $@ ~/.trash/

Make t hat change t o your script , t hen t est it by rem oving m ult iple files. You should now have a script t hat works every t im e.

1.9.5 Taking Out the Trash You should occasionally go t hrough your t rash direct ory and really rem ove t he files you no longer want . I f you're really on your t oes you m ay be t hinking, " But how do I em pt y t he t rash direct ory?" I f you do t his: % rm ~/.trash/*

your t rash direct ory won't lose any files! This t im e you really do want t o use rm, not trash. To t ell your shell t o use t he real rm com m and, sim ply put a \ in front of it like so: % \rm /trash/*

Voila, em pt y recycling bin.

1.9.6 Hacking the Hack One obvious ext ension is t o keep versioned backups. Use t he date com m and t o find t he t im e of delet ion and append t hat t o t he nam e of t he file in t he trash com m and. You could get infinit ely m ore com plicat ed by st oring a lim it ed num ber of versions or delet ing all versions older t han a week or a m ont h. Of course, you could also keep your im port ant files under version cont rol and leave t he com plexit y t o som eone else!

- 45 -

Hack 9 Customize User Configurations

N ow t h a t you k now how t o se t u p a u se ful e n vir onm e n t for you r se lf, it 's t im e t o sh a r e t h e w e a lt h. I t 's very easy for a syst em adm inist rat or t o ensure t hat each newly creat ed user st art s out wit h t he sam e configurat ion files. For exam ple, every user can receive t he sam e cust om ized prom pt , shell variables, or hot keys. Whenever you creat e a new user, several default ( and hidden, or dot , files) are copied int o t he new user's hom e direct ory. I n FreeBSD, t he source of t hese files is / usr/ share/ skel/ . Any cust om izat ions you m ake t o t hese files will be seen by all subsequent ly creat ed users. Do not e t hat you'll have t o m anually copy over any m odified files t o exist ing users. I t 's useful t o underst and t hese files, as t hey apply t o every user you creat e. Depending upon your needs, you'll probably end up rem oving som e of t he default s, cust om izing ot hers, and even adding a few of your own.

1.10.1 Default Files Let 's t ake a quick t our of t he default files: % ls -l /usr/share/skel total 24 drwxr-xr-x

2 root

wheel

512 Jul 28 16:09 ./

drwxr-xr-x

27 root

wheel

512 Jul 28 16:06 ../

-rw-r--r--

1 root

wheel

921 Jul 28 16:09 dot.cshrc

-rw-r--r--

1 root

wheel

248 Jul 28 16:09 dot.login

-rw-r--r--

1 root

wheel

158 Jul 28 16:09 dot.login_conf

-rw-------

1 root

wheel

371 Jul 28 16:09 dot.mail_aliases

-rw-r--r--

1 root

wheel

331 Jul 28 16:09 dot.mailrc

-rw-r--r--

1 root

wheel

797 Jul 28 16:09 dot.profile

-rw-------

1 root

wheel

276 Jul 28 16:09 dot.rhosts

-rw-r--r--

1 root

wheel

975 Jul 28 16:09 dot.shrc

Not e t hat each st art s wit h t he word dot. However, when t he files are copied int o a user's hom e direct ory, t he dots t urn int o lit eral dot s ( .) . Also, t he files in t his direct ory are owned by root, but when a new user is creat ed, t he copied over files will change ownership as t hey are placed in t hat user's hom e direct ory.

- 46 -

1 .1 0 .1 .1 dot .csh r c Let 's exam ine each default file, st art ing wit h dot .cshrc. ( [ H a ck # 2 ] int roduced several .cshrc hacks.) I f you'd like new users t o receive your cust om izat ions, sim ply replace / usr/ share/ skel/ dot .cshrc wit h your hacked version of .cshrc. Don't forget t o renam e t he file as you copy it : # cp /root/.cshrc /usr/share/skel/dot.cshrc

Here, I overwrot e t he default dot .cshrc by copying over t he superuser's cust om ized version of .cshrc. Alt hough you could edit / usr/ share/ skel/ dot .cshrc direct ly, you m ay find it m ore convenient t o have a cust om ized copy st ored elsewhere. All isn't lost if you already have exist ing users whom you'd like t o receive t his file. First , find out what users already exist and have hom e direct ories. This is a quick way t o do so: # ls /usr/home dru

test

Since t his syst em has only t wo exist ing users, it 's an easy m at t er t o copy over m y cust om ized .cshrc. I 'm also a lazy t ypist , so I use ~ inst ead of t yping out /usr/home. Also not e t hat I have t o rem em ber t o m anually change ownership: # cp /root/.cshrc ~dru/ # chown dru ~dru/.cshrc # cp /root/.cshrc ~test/ # chown test ~test/.cshrc

I f your syst em already cont ains m any users, you'll probably prefer t o writ e a script . Here is an exam ple: #!/usr/bin/perl -w

# copydotfiles.pl #

- copy default files to user directories

#

- change ownership of those files

# You may wish to change these constants for your system:

use constant HOMEDIR => '/usr/home'; use constant SKELDIR => '/usr/share/skel'; use constant PREFIX

=> 'dot';

- 47 -

use strict;

use File::Copy; use File::Spec::Functions;

die "Usage: $0 \n" unless @ARGV;

for my $user ( get_users( ) ) { for my $dotfile (@ARGV) { my $source = catfile( SKELDIR( ), my $dest

PREFIX( ) . $dotfile );

= catfile( $user->{homedir},

if (-e $dest) { warn "Skipping existing dotfile $dest...\n"; next; }

copy(

$source,

$dest )

or die "Cannot copy $source to $dest: $!\n"; chown( $user->{uid}, $dest ); } }

sub get_users { local *DIRHANDLE; opendir( DIRHANDLE, HOMEDIR( ) ) or die "Cannot open home directory: $!\n";

- 48 -

$dotfile );

my @users;

while (my $directory = readdir( DIRHANDLE )) { next if $directory =~ /^\./;

my $path = File::Spec->catdir( HOMEDIR( ), $directory ); my $uid

= getpwnam( $directory );

next unless -d $path; next unless $uid;

push @users, { homedir => $path, uid => $uid }; }

return @users; }

This script first exam ines all of t he users wit h hom e direct ories, ret urning a list of t hose direct ories and t he user I Ds. I t loops t hrough t hat list , copying each dot file you provided on t he com m and line t o t hat user's hom e direct ory and changing t he ownership t o t he user. I f you run it as: # copydotfiles.pl .cshrc

all users will receive a new .cshrc file, unless one already exist s. 1 .1 0 .1 .2 dot .login The next file, dot .login, is used only by t he csh and tcsh shells. I f your users don't plan on using t hese shells, you can safely rem ove t his file from / usr/ share/ skel. I f your users do use t hose shells, consider whet her t here are any com m ands you would like t o run when users log in. Not e t hat t his file is read aft er .cshrc. By default , t he only uncom m ent ed line in t his file is: % grep -v '#' /usr/share/skel/dot.login

- 49 -

[ -x /usr/games/fortune ] && /usr/games/fortune freebsd-tips

Here, I used t he reverse filt er swit ch -v t o t he grep search ut ilit y t o look for all t he lines t hat do not begin wit h t he # com m ent sym bol. The result ing line t ells t he shell t o run t he fortune program . I f you chose t o inst all t he gam es dist ribut ion when you inst alled FreeBSD, your fortune appears j ust before t he MOTD whenever you login. Have you ever not iced t hat you don't receive a fort une when you use su? That 's because .login is only read when you log in, and t he default invocat ion of su does not act ually log you in. I nst ead, it opens what is known as a nonlogin shell. You also get one of t hose every t im e you open an xterm. Basically, t he only t im e you get a real login shell is when you t ype in your usernam e and password at a login prom pt . Herein lies t he difference bet ween .cshrc and .login. Place what you would like t o happen only when you log in int o .login, and place what you would like t o happen whenever you use t he csh shell, even if it isn't a login shell, int o .cshrc. I f you don't see t he need for a difference, you don't need / usr/ share/ skel/ dot .login. 1 .1 0 .1 .3 dot .login _ con f Reading t he default cont ent s of dot .login_conf will give you an idea of it s purpose and where t o go for addit ional inform at ion: % more /usr/share/skel/dot.login_conf # $FreeBSD: src/share/skel/dot.login_conf,v 1.3 2001/06/10 17:08:53 ache Exp $ # # see login.conf(5) # #me:\ #

:charset=iso-8859-1:\

#

:lang=de_DE.ISO8859-1:

Not e t hat t his file is com m ent ed by default , but shows t he synt ax a user can use t o creat e a cust om ized .login.conf. Usually such set t ings are set in t he globally adm inist rat ed / et c/ login.conf file, and individual users can override only som e of t hose set t ings. I f your users don't have a need or t he know- how t o configure t hose set t ings, you can safely rem ove t his file from / usr/ share/ skel. 1 .1 0 .1 .4 dot .m a il_ a lia se s a nd dot .m a ilr c The next t wo files work hand in hand and cust om ize t he behavior of man mail. Since it is quit e rare t o find users who st ill rely on t he original mail program , you can safely rem ove t hose files.

- 50 -

1 .1 0 .1 .5 dot .pr ofile The dot .profile file is read by t he Bourne, bash, and Korn shells. I t is t he only file read when a user logs int o a Bourne shell, t he first file read when a user logs int o t he Korn shell, and is opt ional for bash users. I f your users don't use t he Bourne or Korn shells, t here's not m uch sense populat ing t heir hom e direct ories wit h t his file. Depending upon your slant , you m ay wish t o keep t his file in order t o place pat h st at em ent s and environm ent variables for use wit h Bourne shell script s. However, m ost users t end t o place t hose direct ly int o t he script it self t o allow for port abilit y. I f your users wish t o use t he bash shell, which isn't inst alled by default , keep in m ind t hat .profile allows a user t o override t he set t ings found in t he global / et c/ profile file. You m ay find it easier t o m ake your edit s t o t he global file and t hen rem ove / usr/ share/ skel/ dot .profile. More sophist icat ed users can always creat e t heir own ~ / .profile. However, m ost bash users t end t o m ake t heir m odificat ions t o ~ / .bash_profile. 1 .1 0 .1 .6 dot .r h ost s Did you happen t o not ice in t he earlier long list ing t hat t his file has different perm issions from m ost of t he ot her files? I f you read man rhosts, you'll see t hat t his file is ignored if it is writ able by any user ot her t han t he owner of t he file. So, when is t his file used? I t 's used when a user t ypes one of t he r* com m ands: rsh, rcp, or rlogin. I won't show you how t o set up t his file or use t hose com m ands, as t hey were designed for use back in t he days when net works were considered t rust ed. They've pret t y well been replaced by ssh and scp, which provide a m uch safer way t o log int o rem ot e syst em s and t o t ransfer files. For t his reason, I always rem ove / usr/ share/ skel/ dot .rhost s from m y syst em s. 1 .1 0 .1 .7 dot .sh r c The last default file is dot .shrc. As you m ay have guessed, it is t he rc file for sh, t he Bourne shell. Again, if your users don't log int o t hat shell, t hey won't m iss t his file.

1.10.2 Missing (but Useful) Dot Files Now t hat we've had t he opport unit y t o look at t he default files, it 's t im e t o consider any useful m issing files. 1 .1 0 .2 .1 dot .logou t We've already seen t hat ~ / .login is read when a user logs int o t he csh or tcsh shells. Not surprisingly, ~ / .logout is read when a user logs out of t heir login shell. This is an excellent place t o put com m ands you would like t o execut e as a user logs out . I t could be som et hing as sim ple as: # more dot.logout # this line clears your screen when you logout clear # add your own commands or scripts, one line at a time, # which you would like to execute

- 51 -

# whenever you logout and leave your terminal

This dot .logout will clear t he user's t erm inal, m aking it m uch neat er for t he next person who logs in. Not ice t hat I com m ent ed t his file, so t he user is aware of it s use. When creat ing your own dot files, use lot s of com m ent s. I f you int end for your users t o cust om ize t heir own dot files, use com m ent s t hat explain t he synt ax t hey can use when t hey do t heir m odificat ions. dot .logout can run any com m and or script t hat suit s a user's needs. Here are som e ideas t o get your im aginat ion rolling: • • •

A script t hat backs up t he user's hom e direct ory A script t hat shows how m uch t im e t he user spent online A script t hat displays ot her st at ist ics, such as available disk space

1 .1 0 .2 .2 dot .x in it r c I also find it very useful t o creat e a cust om dot .xinit rc. By default , users receive t he ext rem ely light weight twm window m anager. Since I usually inst all KDE, t his line ensures t hat each user will receive t hat window m anager inst ead: # more dot.xinitrc exec startkde

You can also specify which program s you would like t o launch when a user t ypes startx and t heir ~ / .xinit rc file kicks in. For exam ple, t his is a popular line t o add: # more dot.xinitrc exec xterm & exec startkde

This st art s an xterm in t he background. Not ice t he & at t he end of it s line—t his is t o ensure t hat once xterm loads, it doesn't int erfere wit h any ot her program s t hat are st ill loading. When you're creat ing your own dot .xinit rc, you can st art any program you like. However, st art your window m anager last . St art your ot her program s, one line at a t im e, put t ing an & at t he end of each line. The only line t hat does not have an & will be t he very last line, t he one t hat loads your window m anager. Since I prefer t o st art m y browser inst ead of an xterm, here is m y cust om ized dot .xinit rc: #to start another program when you "startx", type: #exec path_to_program & #before these lines exec /usr/X11R6/bin/mozilla & exec startkde

- 52 -

There are dozens of possibilit ies for cust om ized dot files. Take st ock of your own syst em s, and ask yourself: " What program s do m y users use?" For exam ple, if your users use bash, vim, screen, procmail, or fetchmail, why not st art t hem off wit h a cust om ized configurat ion file t hat cont ains com m ent s on how t o add t heir own cust om izat ions and URLs of where t o go for furt her ideas? A lit t le hom ework and creat ivit y on your part can help your users get t he m ost out of t he ut ilit ies t hey use on a daily basis.

1.10.3 Editing /usr/src/share/skel/Makefile Let 's end t his hack by exam ining where t he default dot files in / usr/ share/ skel cam e from in t he first place. You'll find t he answer here: % ls /usr/src/share/skel ./

dot.login

dot.profile

../

dot.login_conf

dot.rhosts

Makefile

dot.mail_aliases

dot.shrc

dot.cshrc

dot.mailrc

That Makefile cont rols t he inst allat ion of t hose files: # more /usr/src/share/skel/Makefile

#

@(#)Makefile

8.1 (Berkeley) 6/8/93

# $FreeBSD: src/share/skel/Makefile,v 1.8 2002/07/29 09:40:13 ru Exp $

FILES1= dot.cshrc dot.login dot.login_conf dot.mailrc dot.profile dot.shrc FILES2=

dot.mail_aliases dot.rhosts

MODE1=

0644

MODE2=

0600

NOOBJ=

noobj

all clean cleandir depend lint tags:

install: ${INSTALL} -o ${BINOWN} -g ${BINGRP} -m ${MODE1} ${FILES1} \ ${DESTDIR}${BINDIR}/skel ${INSTALL} -o ${BINOWN} -g ${BINGRP} -m ${MODE2} ${FILES2} \

- 53 -

${DESTDIR}${BINDIR}/skel

.include

Even if you've never read a Makefile before, you'll find it 's not t oo hard t o figure out what 's going on if you already know which result s t o expect . I n t his Makefile, FILES=1 is sim ply a list of files t o inst all. Take a look at MODE1; it t ells t he chmod com m and what perm issions t o set on t hose files. Sim ilarly, FILES=2 is anot her list of files. Those t wo files had different perm issions, which were defined by MODE2. Move down t o t he install sect ion. Don't worry so m uch about t he synt ax; rat her, not ice t he pat t ern. The first set of files are inst alled and t heir m ode is applied. Then t he second set of files are inst alled wit h t heir m ode. I t 's an easy m at t er t o cust om ize t his file t o reflect t he dot files you'd like t o see inst alled. I n t his exam ple, I only want t o inst all m y cust om versions of dot .cshrc, dot .login, and dot .xinit rc. Since t hey all require t he first m ode, I 'll rem ove any references t o t he second set of files: # cd /usr/src/share/skel # cp Makefile Makefile.orig # vi Makefile

#

@(#)Makefile

8.1 (Berkeley) 6/8/93

# my customized dot files to be installed into /usr/share/skel

FILES1= dot.cshrc dot.login dot.xinitrc MODE1=

0644

NOOBJ=

noobj

all clean cleandir depend lint tags:

install: ${INSTALL} -o ${BINOWN} -g ${BINGRP} -m ${MODE1} ${FILES1} \ ${DESTDIR}${BINDIR}/skel

.include

- 54 -

Now let 's t ry a t est run. I 'll replace t he default dot files found in / usr/ src/ share/ skel wit h m y cust om ized versions. I 'll t hen rem ove t he cont ent s of / usr/ share/ skel and see what happens when I run m y cust om ized Makefile: # cd /usr/src/share/skel # rm dot.* # cp ~/mystuff/dot.* .

# rm /usr/share/skel/* # ls /usr/share/skel

# make install install -o root -g wheel -m 0644 dot.cshrc dot.login dot.xinitrc /usr/share/skel # ls /usr/share/skel dot.cshrc

dot.login

dot.xinitrc

I find it very handy t o keep a copy of m y cust om ized Makefile and dot files in a separat e direct ory, in t his case ~ / m yst uff. This ensures t hey are backed up. I t 's easy for m e t o grab t hose files whenever I want t o cust om ize a part icular syst em . I t 's especially im port ant t o use a separat e locat ion if you use cvsup t o keep your syst em up- t o- dat e. Ot herwise, your next updat e will not ice your m odified src and happily replace t hose m issing original source files. But don't worry; it won't t ouch your new / usr/ share/ skel. Of course, som et im es t his is a very useful t rick in it self. I f you ever m ess up a file locat ed som ewhere wit hin /usr/src, a quick cvsup will put everyt hing back t he way it was. See [ H a ck # 8 0 ] for det ails on aut om at ing cvsup.

1.10.4 The Other BSDs •

The preceding discussion is based on FreeBSD, but it also applies t o Net BSD and OpenBSD syst em s, save for a few t iny differences out lined here. See t he m anpages ret urned by apropos user

1 .1 0 .4 .1 N e t BSD Net BSD adm inist rat ors will find t he skelet on hom e direct ory in / et c/ skel. Specify a different locat ion by passing t he -k opt ion t o useradd. 1 .1 0 .4 .2 Ope n BSD OpenBSD syst em s st ore t he skelet on hom e direct ory in / et c/ skel. Specify a different skelet on direct ory locat ion by passing t he -dotdir opt ion t o adduser.

- 55 -

Hack 10 Maintain Your Environment on Multiple Systems

Th e sign of a t r u e Unix gu r u is t h e a bilit y t o pe r for m a t a sk qu ick ly w h e n con fr on t e d w it h a n un fa m ilia r sh e ll, k e yboa r d, t e r m in a l, w in dow m a na ge r , or ope r a t in g syst e m . A large part of using Unix syst em s effect ively involves configuring a com fort able environm ent using fam iliar t ools available from t he Unix shell prom pt . I t 's m uch easier t o perform a t ask quickly when all of t he short cut s your fingers have learned work on t he first t ry. Even som et hing as sim ple as set t ing up your prom pt t he way you like it can st eal significant t im e from your product ivit y if you need t o do it on several host s. I f you're going t o spend significant t im e in a Unix shell, it 's wort h get t ing organized. A bit of onet im e effort will reward you lat er, every t im e you sit down at t he keyboard.

1.11.1 Enter unison unison is a t ool for m aint aining synchronized copies of direct ories. I 've used it t o m aint ain a cent ral reposit ory of all of m y dot files, shell script s, signat ures file, Spam Assassin configurat ion—basically any file I 'd like t o have available, regardless of which host I happen t o be logged int o. You can inst all unison from t he Net BSD pkgsrc collect ion: # cd /usr/pkgsrc/net/unison # make install clean

FreeBSD and OpenBSD port s also include net / unison. Even bet t er, t his ut ilit y is available for m ost Unix and Windows plat form s. See t he m ain unison web sit e for det ails.

1.11.2 Using unison Whenever I configure a new Unix host or get a shell on anot her syst em , I inst all unison. Then, I creat e a direct ory t o receive t he files I 've st ored in t he / usr/ work/ sync direct ory at host .exam ple.com . I call t he local direct ory ~ / sync. To synchronize t hose t wo direct ories: % unison ~/sync ssh://[email protected]://usr/work/sync p = /home/david/.unison; bn = .unison Contacting server...

- 56 -

p = /home/david/sync; bn = sync [email protected]'s password:

Aft er ssh prom pt s for a password or pass phrase, t he unison exchange begins. On a first t im e synchronizat ion, unison will ask only one quest ion: whet her you wish t o copy t he rem ot e direct ory t o t he local host . Looking for changes Warning: No archive files were found for these roots.

This can happen

either because this is the first time you have synchronized these roots, or because you have upgraded Unison to a new version with a different archive format.

Updat e det ect ion m ay t ake a while on t his run if t he replicas are large. unison will assum e t hat t he last synchronized st at e of bot h replicas was com plet ely em pt y. This m eans t hat any files t hat are different will be report ed as conflict s, and any files t hat exist only on one replica will be j udged as new and propagat ed t o t he ot her replica. I f t he t wo replicas are ident ical, t hen unison will report no changes: Press return to continue. Waiting for changes from server Reconciling changes

local

host.example.com > somefile

Rem em ber t o use t he backt icks ( `) , oft en found on t he far left of t he keyboard on t he sam e key as t he t ilde ( ~) . I f you inst ead use t he single quot e ( ') charact er, usually locat ed on t he right side of t he keyboard on t he sam e key as t he double quot e ( " ) , your file will cont ain t he echoed st ring which xmms inst ead of t he desired pat h. The user's current shell will affect how which's swit ches work. Here is an exam ple from t he C shell: % which -a xmms -a: Command not found. /usr/X11R6/bin/xmms

% which which which: shell built-in command.

- 71 -

This is a m at t er of which which t he user is using. Here, t he user used t he which which is built int o t he C shell and doesn't support t he opt ions used by t he which ut ilit y. Where t hen is t hat which? Try t he whereis com m and: % whereis -b which which: /usr/bin/which

Here, I used -b t o search only for t he binary. Wit hout any swit ches, whereis will display t he binary, t he m anpage pat h, and t he pat h t o t he original sources. I f your users prefer t o use t he real which com m and inst ead of t he shell version and if t hey are only int erest ed in seeing binary pat hs, consider adding t hese lines t o / usr/ share/ skel/ dot .cshrc [ H a ck # 9 ] : alias which

/usr/bin/which -a

alias whereis

whereis -b

The -a swit ch will list all binaries wit h t hat nam e, not j ust t he first binary found.

2.2.2 Finding Commands How do you proceed when you know what it is t hat you want t o do, but have no clue which com m ands are available t o do it ? I know I clung t o t he whatis com m and like a life preserver when I was first int roduced t o Unix. For exam ple, when I needed t o know how t o set up PPP: % whatis ppp i4bisppp(4) network driver

- isdn4bsd synchronous PPP over ISDN B-channel

ng_ppp(4)

- PPP protocol netgraph node type

ppp(4)

- point to point protocol network interface

ppp(8)

- Point to Point Protocol (a.k.a. user-ppp)

pppctl(8)

- PPP control program

pppoed(8)

- handle incoming PPP over Ethernet connections

pppstats(8)

- print PPP statistics

On t he days I had t im e t o sat isfy m y curiosit y, I t ried t his variat ion: % whatis "(1)"

That will show all of t he com m ands t hat have a m anpage in sect ion 1. I f you're rust y on your m anpage sect ions, whatis intro should refresh your m em ory.

- 72 -

2.2.3 Finding Words The previous com m ands are great for finding binaries and m anpages, but what if you want t o find a part icular word in one of your own t ext files? That requires t he not oriously userunfriendly find com m and. Let 's be realist ic. Even wit h all of your Unix experience, you st ill have t o dig int o eit her t he m anpage or a good book whenever you need t o find som et hing. Can you really expect novice users t o figure it out ? To st art wit h, t he regular old invocat ion of find will find filenam es, but not t he words wit hin t hose files. We need a j udicious use of grep t o accom plish t hat . Fort unat ely, find's -exec swit ch allows it t o use ot her ut ilit ies, such as grep, wit hout forking anot her process. St art off wit h a find com m and t hat looks like t his: % find . -type f -exec grep "word" {

} \;

This invocat ion says t o st art in t he current direct ory ( .) , look t hrough files, not direct ories ( type f) , while running t he grep com m and ( -exec grep) in order t o search for t he word word. Not e t hat t he synt ax of t he -exec swit ch always resem bles: -exec command with_its_parameters {

} \;

What happens if I search t he files in m y hom e direct ory for t he word alias? % find . -type f -exec grep "alias" { alias h

history 25

alias j

jobs -l

} \;

Antialiasing=true Antialiasing arguments=-sDEVICE=x11 -dTextAlphaBits=4 -dGraphicsAlphaBits=2 -dMaxBitmap=10000000 (proc-arg 0 "antialiasing" "Apply antialiasing (TRUE/FALSE)") (proc-arg 0 "antialiasing" "Apply antialiasing (TRUE/FALSE)")

While it 's nice t o see t hat find successfully found t he word alias in m y hom e direct ory, t here's one slight problem . I have no idea which file or files cont ained m y search expression! However, adding / dev/ null t o t hat com m and will fix t hat : # find . -type f -exec grep "alias" /dev/null { ./.cshrc:alias h

history 25

./.cshrc:alias j

jobs -l

} \;

./.kde/share/config/kghostviewrc:Antialiasing=true ./.kde/share/config/kghostviewrc:Antialiasing arguments=-sDEVICE=x11 -dTextAlphaBits=4 -dGraphicsAlphaBits=2 -dMaxBitmap=10000000

- 73 -

./.gimp-1.3/pluginrc: (TRUE/FALSE)")

(proc-arg 0 "antialiasing" "Apply antialiasing

./.gimp-1.3/pluginrc: (TRUE/FALSE)")

(proc-arg 0 "antialiasing" "Apply antialiasing

Why did adding not hing, / dev/ null, aut om agically cause t he nam e of t he file t o appear next t o t he line t hat cont ains t he search expression? I s it because Unix is t ruly am azing? Aft er all, it does allow even t he st at e of not hingness t o be expressed as a filenam e. Act ually, it works because grep will list t he filenam e whenever it searches m ult iple files. When you j ust use { }, find will pass each filenam e it finds one at a t im e t o grep. Since grep is searching only one filenam e, it assum es you already know t he nam e of t hat file. When you use /dev/null { }, find act ually passes grep t wo files, / dev/ null along wit h whichever file find happens t o be working on. Since grep is now com paring t wo files, it 's nice enough t o t ell you which of t he files cont ained t he search st ring. We already know / dev/ null won't cont ain anyt hing, so we j ust convinced grep t o give us t he nam e of t he ot her file. That 's pret t y handy. Now let 's m ake it friendly. Here's a very sim ple script called fstring: % more ~/bin/fstring #!/bin/sh # script to find a string # replaces $1 with user's search string find . -type f -exec grep "$1" /dev/null {

} \;

That $1 is a posit ional param et er. This script expect s t he user t o give one param et er: t he word t he user is searching for. When t he script execut es, t he shell will replace " $1" wit h t he user's search st ring. So, t he script is m eant t o be run like t his: % fstring word_to_search

I f you're planning on using t his script yourself, you'll probably rem em ber t o include a search st ring. I f you want ot her users t o benefit from t he script , you m ay want t o include an if st at em ent t o generat e an error m essage if t he user forget s t he search st ring: #!/bin/sh # script to find a string # replaces $1 with user's search string # or gives error message if user forgets to include search string if test $1 then

- 74 -

find . -type f -exec grep "$1" /dev/null {

} \;

else echo "Don't forget to include the word you would like to search for" exit 1 fi

Don't forget t o m ake your script execut able wit h chmod +x and t o place it in t he user's pat h. / usr/ local/ bin is a good locat ion for ot her users t o benefit .

2.2.4 See Also • • • • •

man man man man man

which whereis whatis find grep

- 75 -

Hack 14 Get the Most Out of grep

You m a y not k now w h e r e it s odd n a m e or igin a t e d, bu t you ca n't a r gue t he u se fu ln e ss of gr e p. Have you ever needed t o find a part icular file and t hought , " I don't recall t he filenam e, but I rem em ber som e of it s cont ent s" ? The oddly nam ed grep com m and does j ust t hat , searching inside files and report ing on t hose t hat cont ain a given piece of t ext .

2.3.1 Finding Text Suppose you wish t o search your shell script s for t he t ext $USER. Try t his: % grep -s '$USER' * add-user:if [ "$USER" != "root" ]; then bu-user:

echo "

[-u user] - override $USER as the user to backup"

bu-user:if [ "$user" = "" ]; then user="$USER"; fi del-user:if [ "$USER" != "root" ]; then mount-host:mounted=$(df | grep "$ALM_AFP_MOUNT/$USER") ..... mount-user:

echo "

[-u user] - override $USER as the user to backup"

mount-user:if [ "$user" = "" ]; then user="$USER"; fi

I n t his exam ple, grep has searched t hrough all files in t he current direct ory, displaying each line t hat cont ained t he t ext $USER. Use single quot es around t he t ext t o prevent t he shell from int erpret ing special charact ers. The -s opt ion suppresses error m essages when grep encount ers a direct ory. Perhaps you only want t o know t he nam e of each file cont aining t he t ext $USER. Use t he -l opt ion t o creat e t hat list for you: % grep -ls '$USER' * add-user bu-user del-user mount-host mount-user

- 76 -

2.3.2 Searching by Relevance What if you're m ore concerned about how m any t im es a part icular st ring occurs wit hin a file? That 's known as a relevance search . Use a com m and sim ilar t o: % grep -sc '$USER' * | grep -v ':0' | sort

-k 2 -t : -r

mount-host:6 mount-user:2 bu-user:2 del-user:1 add-user:1

How does t his m agic work? The -c flag list s each file wit h a count of m at ching lines, but it unfort unat ely includes files wit h zero m at ches. To count er t his, I piped t he out put from grep int o a second grep, t his t im e searching for ':0' and using a second opt ion, -v, t o reverse t he sense of t he search by displaying lines t hat don't m at ch. The second grep reads from t he pipe inst ead of a file, searching t he out put of t he first grep. For a lit t le ext ra flair, I sort ed t he subsequent out put by t he second field of each line wit h sort -k 2, assum ing a field separat or of colon ( -t : ) and using -r t o reverse t he sort int o descending order.

2.3.3 Document Extracts Suppose you wish t o search a set of docum ent s and ext ract a few lines of t ext cent ered on each occurrence of a keyword. This t im e we are int erest ed in t he m at ching lines and t heir surrounding cont ext , but not in t he filenam es. Use a com m and som et hing like t his: % grep -rhiw -A4 -B4 'preferences' *.txt > research.txt % more research.txt

This grep com m and searches all files wit h t he .txt ext ension for t he word preferences. I t perform s a recursive search ( -r) t o include all subdirect ories, hides ( -h) t he filenam e in t he out put , m at ches in a case- insensit ive ( -i) m anner, and m at ches preferences as a com plet e word but not as part of anot her word ( -w) . The -A4 and -B4 opt ions display t he four lines im m ediat ely aft er and before t he m at ched line, t o give t he desired cont ext . Finally, I 've redirect ed t he out put t o t he file research.t xt . You could also send t he out put st raight t o t he vim t ext edit or wit h: % grep -rhiw -A4 -B4 'preferences' *.txt | vim Vim: Reading from stdin... vim can be inst alled from / usr/ port s/ edit ors/ vim .

- 77 -

Specifying vim - t ells vim t o read st din ( in t his case t he piped out put from grep) inst ead of a file. Type :q! t o exit vim. To search files for several alt ernat ives, use t he -e opt ion t o int roduce ext ra search pat t erns: % grep -e 'text1' -e 'text2' * Q. How did grep get it s odd nam e? A. grep was writ t en as a st andalone program t o sim ulat e a com m only perform ed com m and available in t he ancient Unix edit or ex. The com m and in quest ion searched an ent ire file for lines cont aining a regular expression and displayed t hose lines. The com m and was g/re/p: globally search for a regular expression and print t he line.

2.3.4 Using Regular Expressions To search for t ext t hat is m ore vaguely specified, use a regular expression. grep underst ands bot h basic and ext ended regular expressions, t hough it m ust be invoked as eit her egrep or grep -E when given an ext ended regular expression. The t ext or regular expression t o be m at ched is usually called t he pat t ern. Suppose you need t o search for lines t hat end in a space or t ab charact er. Try t his com m and ( t o insert a t ab, press Ct rl- V and t hen Ct rl- I , shown as in t he exam ple) : % grep -n '[ ]$' test-file 2:ends in space 3:ends in tab

I used t he [...] const ruct t o form a regular expression list ing t he charact ers t o m at ch: space and t ab. The expression m at ches exact ly one space or one t ab charact er. $ anchors t he m at ch t o t he end of a line. The -n flag t ells grep t o include t he line num ber in it s out put . Alt ernat ively, use: % grep -n '[[:blank:]]$' test-file 2:ends is space 3:ends in tab

Regular expressions provide m any preform ed charact er groups of t he form [[: descript ion: ] ] . Exam ple groups include all cont rol charact ers, all digit s, or all alphanum eric charact ers. See man re_format for det ails. We can m odify a previous exam ple t o search for eit her " preferences" or " preference" as a com plet e word, using an ext ended regular expression such as t his: % egrep -rhiw -A4 -B4 'preferences?' *.txt > research.txt

- 78 -

The ? sym bol specifies zero or one of t he preceding charact er, m aking t he s of preferences opt ional. Not e t hat I use egrep because ? is available only in ext ended regular expressions. I f you wish t o search for t he ? charact er it self, escape it wit h a backslash, as in \?. An alt ernat ive m et hod uses an expression of t he form (string1|string2), which m at ches eit her one st ring or t he ot her: % egrep -rhiw -A4 -B4 'preference(s|)' *.txt > research.txt

As a final exam ple, use t his t o seek out all bash, tcsh, or sh shell script s: % egrep '^#\!/bin/(ba|tc|)sh[[:blank:]]*$' *

The caret ( ^) charact er at t he st art of a regular expression anchors it t o t he st art of t he line ( m uch as $ at t he end anchors it t o t he end) . (ba|tc|) m at ches ba, t c, or not hing. The * charact er specifies zero or m ore of [[:blank:]], allowing t railing whit espace but not hing else. Not e t hat t he ! charact er m ust be escaped as \! t o avoid shell int erpret at ion in tcsh ( but not in bash) . Here's a handy t ip for debugging regular expressions: if you don't pass a filenam e t o grep, it will read st andard input , allowing you t o ent er lines of t ext t o see which m at ch. grep will echo back only m at ching lines.

2.3.5 Combining grep with Other Commands grep works well wit h ot her com m ands. For exam ple, t o display all tcsh processes: % ps axww | grep -w 'tcsh' saruman 10329

0.0

0.2

6416

1196

p1

Ss

Sat01PM

0:00.68 -tcsh (tcsh)

saruman 11351

0.0

0.2

6416

1300 std

Ss

Sat07PM

0:02.54 -tcsh (tcsh)

saruman 13360

0.0

0.0

1116

4 std

R+

10:57PM

0:00.00 grep -w tcsh

%

Not ice t hat t he grep com m and it self appears in t he out put . To prevent t his, use: % ps axww | grep -w '[t]csh' saruman 10329

0.0

0.2

6416

1196

p1

Ss

Sat01PM

0:00.68 -tcsh (tcsh)

saruman 11351

0.0

0.2

6416

1300 std

Ss

Sat07PM

0:02.54 -tcsh (tcsh)

%

- 79 -

I 'll let you figure out how t his works.

2.3.6 See Also • •

man grep man re_format ( regular expressions)

- 80 -

Hack 15 Manipulate Files with sed

I f you 've e ve r ha d t o ch a nge t he for m a t t in g of a file , you k n ow t ha t it ca n be a t im e - con su m ing pr oce ss. Why wast e your t im e m aking m anual changes t o files when Unix syst em s com e wit h m any t ools t hat can very quickly m ake t he changes for you?

2.4.1 Removing Blank Lines Suppose you need t o rem ove t he blank lines from a file. This invocat ion of grep will do t he j ob: % grep -v '^$' letter1.txt > tmp ; mv tmp letter1.txt

The pat t ern ^$ anchors t o bot h t he st art and t he end of a line wit h no int ervening charact ers—t he regexp definit ion of a blank line. The -v opt ion reverses t he search, print ing all nonblank lines, which are t hen writ t en t o a t em porary file, and t he t em porary file is m oved back t o t he original. grep m ust never out put t o t he sam e file it is reading, or t he file will end up em pt y.

You can rewrit e t he preceding exam ple in sed as: % sed '/^$/d' letter1.txt > tmp ; mv tmp letter1.txt

'/^$/d' is act ually a sed script . sed's norm al m ode of operat ion is t o read each line of input , process it according t o t he script , and t hen writ e t he processed line t o st andard out put . I n t his exam ple, t he expression '/^$/ is a regular expression m at ching a blank line, and t he t railing d' is a sed funct ion t hat delet es t he line. Blank lines are delet ed and all ot her lines are print ed. Again, t he result s are redirect ed t o a t em porary file, which is t hen copied back t o t he original file.

2.4.2 Searching with sed sed can also do t he work of grep: % sed -n '/$USER/p' *

This com m and will yield t he sam e result s as: % grep '$USER' *

- 81 -

The -n ( no- print , perhaps) opt ion prevent s sed from out put t ing each line. The pat t ern /$USER/ m at ches lines cont aining $USER, and t he p funct ion print s m at ched lines t o st andard out put , overriding -n.

2.4.3 Replacing Existing Text One of t he m ost com m on uses for sed is t o perform a search and replace on a given st ring. For exam ple, t o change all occurrences of 2003 int o 2004 in a file called dat e, include t he t wo search st rings in t he form at 's/oldstring/newstring/', like so: % sed 's/2003/2004/' date Copyright 2004 ... This was written in 2004, but it is no longer 2003. ...

Alm ost ! Not iced t hat t hat last 2003 rem ains unchanged. This is because wit hout t he g ( global) flag, sed will change only t he first occurrence on each line. This com m and will give t he desired result : % sed 's/2003/2004/g' date

Search and replace t akes ot her flags t oo. To out put only changed lines, use: % sed -n 's/2003/2004/gp' date

Not e t he use of t he -n flag t o suppress norm al out put and t he p flag t o print changed lines.

2.4.4 Multiple Transformations Perhaps you need t o perform t wo or m ore t ransform at ions on a file. You can do t his in a single run by specifying a script wit h m ult iple com m ands: % sed 's/2003/2004/g;/^$/d' date

This perform s bot h subst it ut ion and blank line delet ion. Use a sem icolon t o separat e t he t wo com m ands. Here is a m ore com plex exam ple t hat t ranslat es HTML t ags of t he form int o PHP bullet in board t ags of t he form [font]: % cat index.html hello

- 82 -

% sed 's//[\1]/g' index.html [title]hello [/title]

How did t his work? The script searched for an HTML t ag using t he pat t ern ''. Angle bracket s m at ch lit erally. I n a regular expression, a dot ( .) represent s any charact er and an ast erisk ( *) m eans zero or m ore of t he previous it em . Escaped parent heses, \( and \), capt ure t he m at ched pat t ern laying bet ween t hem and place it in a num bered buffer. I n t he replace st ring, \1 refers t o t he cont ent s of t he first buffer. Thus t he t ext bet ween t he angle bracket s in t he search st ring is capt ured int o t he first buffer and writ t en back inside square bracket s in t he replace st ring. sed t akes full advant age of t he power of regular expressions t o copy t ext from t he pat t ern t o it s replacem ent . % cat index1.html hello

% sed 's//[\1]/g' index1.html [title>hello. To prevent t his behavior, we need t o m at ch zero or m ore of any charact er except q_host, a->q_user, p); bc-start if (bitset(EF_VRFYONLY, e->e_flags)) { a->q_state = QS_VERIFIED; return; } bc-end message("aliased to %s", shortenstring(p, MAXSHORTSTR));

and t hen apply a sed script such as: % sed '/bc-start/,/bc-end/s/^/\/\//' source.c

t o get : if (tTd(27, 1)) sm_dprintf("%s (%s, %s) aliased to %s\n",

- 84 -

a->q_paddr, a->q_host, a->q_user, p); //bc-start //

if (bitset(EF_VRFYONLY, e->e_flags))

//

{

//

a->q_state = QS_VERIFIED;

//

return;

//

}

//bc-end message("aliased to %s", shortenstring(p, MAXSHORTSTR));

The script used search and replace t o add // t o t he st art of all lines ( s/^/\/\//) t hat lie bet ween t he t wo m arkers ( /bc-start/,/bc-end/) . This will apply t o every block in t he file bet ween t he m arker pairs. Not e t hat in t he sed script , t he / charact er has t o be escaped as \/ so it is not m ist aken for a delim it er.

2.5.2 Removing Comments When we need t o delet e t he com m ent s and t he t wo bc- lines ( let 's assum e t hat t he edit ed cont ent s were copied back t o source.c) , we can use a script such as: % sed '/bc-start/d;/bc-end/d;/bc-start/,/bc-end/s/^\/\///' source.c

Oops! My first at t em pt won't work. The bc- lines m ust be delet ed aft er t hey have been used as address ranges. Trying again we get : % sed '/bc-start/,/bc-end/s/^\/\///;/bc-start/d;/bc-end/d' source.c

I f you want t o leave t he t wo bc- m arker lines in but com m ent t hem out , use t his piece of t rickery: % sed '/bc-start/,/bc-end/{/^\/\/bc-/\!s/\/\///;}' source.c

t o get : if (tTd(27, 1)) sm_dprintf("%s (%s, %s) aliased to %s\n", a->q_paddr, a->q_host, a->q_user, p); //bc-start if (bitset(EF_VRFYONLY, e->e_flags)) {

- 85 -

a->q_state = QS_VERIFIED; return;

} //bc-end message("aliased to %s", shortenstring(p, MAXSHORTSTR));

Not e t hat in t he bash shell you m ust use: % sed '/bc-start/,/bc-end/{/^\/\/bc-/!s/\/\///;}' source.c

because t he bang charact er ( !) does not need t o be escaped as it does in tcsh. What 's wit h t he curly braces? They prevent a com m on m ist ake. You m ay im agine t hat t his exam ple: % sed -n '/$USER/p;p' *

print s each line cont aining $USER t wice because of t he p;p com m ands. I t doesn't , t hough, because t he second p is not rest rained by t he /$USER/ line address and t herefore applies t o every line. To print t wice j ust t hose lines cont aining $USER, use: % sed -n '/$USER/p;/$USER/p' *

or: % sed -n '/$USER/{p;p;}' *

The const ruct {...} int roduces a funct ion list t hat applies t o t he preceding line address or range. A line address followed by ! ( or \! in t he tcsh shell) reverses t he address range, and so t he funct ion ( list ) t hat follows is applied t o all lines not m at ching. The net effect is t o rem ove // from all lines t hat don't st art wit h //bc- but t hat do lie wit hin t he bc- m arkers.

2.5.3 Using the Holding Space to Mark Text sed reads input int o t he pat t ern space, but it also provides a buffer ( called t he holding space) and funct ions t o m ove t ext from one space t o t he ot her. All ot her funct ions ( such as s and d) operat e on t he pat t ern space, not t he holding space. Check out t his sed script :

- 86 -

% cat case.script # Sed script for case insensitive search # # copy pattern space to hold space to preserve it h y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/ # use a regular expression address to search for lines containing: /test/ { i\ vvvv a\ ^^^^ } # restore the original pattern space from the hold space x;p

First , I have writ t en t he script t o a file inst ead of t yping it in on t he com m and line. Lines st art ing wit h # are com m ent s and are ignored. Ot her lines specify a sed com m and, and com m ands are separat ed by eit her a newline or ; charact er. sed reads one line of input at a t im e and applies t he whole script file t o each line. The following funct ions are applied t o each line as it is read:

h Copies t he pat t ern space ( t he line j ust read) int o t he holding space.

y/ABC/abc/ Operat es on t he pat t ern space, t ranslat ing A t o a, B t o b, and C t o c and so on, ensuring t he line is all lowercase.

/test/ {...} Mat ches t he line j ust read if it includes t he t ext test ( what ever t he original case, because t he line is now all lowercase) and t hen applies t he list of funct ions t hat follow. This exam ple appends t ext before ( i\) and aft er ( a\) t he m at ched line t o highlight it .

x

- 87 -

Exchanges t he pat t ern and hold space, t hus rest oring t he original cont ent s of t he pat t ern space.

p Print s t he pat t ern space. Here is t he t est file: % cat case This contains text

Hello

that we want to

TeSt

search for, but in

test

a case insensitive

XXXX

manner using the sed

TEST

editor.

Bye bye.

%

Here are t he result s of running our sed script on it : % sed -n -f case.script case This contains text

Hello

vvvv that we want to

TeSt

^^^^ vvvv search for, but in

test

^^^^ a case insensitive

XXXX

vvvv manner using the sed

TEST

^^^^ editor.

Bye bye.

Not ice t he vvv ^^^ m arkers around lines t hat cont ain test.

- 88 -

2.5.4 Translating Case The tr com m and can t ranslat e one charact er t o anot her. To change t he cont ent s of case int o all lowercase and writ e t he result s t o file lower- case, we could use: % tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' \ < case > lower-case

tr works wit h st andard input and out put only, so t o read and writ e files we m ust use redirect ion.

2.5.5 Translating Characters To t ranslat e carriage ret urn charact ers int o newline charact ers, we could use: % tr \\r \\n < cr > lf

where cr is t he original file and lf is a new file cont aining line feeds in place of carriage ret urns. \n represent s a line feed charact er, but we m ust escape t he backslash charact er in t he shell, so we use \\n inst ead. Sim ilarly, a carriage ret urn is specified as \\r.

2.5.6 Removing Duplicate Line Feeds tr can also squeeze m ult iple consecut ive occurrences of a part icular charact er int o a single occurrence. For exam ple, t o rem ove duplicat e line feeds from t he lines file: % tr -s \\n < lines > tmp ; mv tmp lines

Here we use t he t m p file t rick again because tr, like grep and sed, will t rash t he input file if it is also t he out put file.

2.5.7 Deleting Characters tr can also delet e select ed charact ers. I f for inst ance if you hat e vowels, run your docum ent s t hrough t his: % tr -d aeiou < file

2.5.8 Translating Tabs to Spaces To t ranslat e t abs int o m ult iple spaces, use t he -x flag: % cat tabs

- 89 -

col

col

col

% od -x tabs 0000000

636f

6c09

636f

6c09

636f

6c0a

0a00

2020

636f

6c20

2020

0000015

% col -x < tabs > spaces % cat spaces col

col

col

% od -h spaces 0000000

636f

6c20

2020

0000020

636f

6c0a

0a00

2020

0000025

I n t his exam ple I have used od -x t o oct al dum p in hexadecim al t he cont ent s of t he before and aft er files, which shows m ore clearly t hat t he t ranslat ion has worked. ( 09 is t he code for Tab and 20 is t he code for Space.)

2.5.9 See Also • • • •

man man man man

sed tr col od

- 90 -

Hack 17 Delimiter Dilemma

D e a l w it h double quot a t ion m a r k s in de lim it e d file s. I m port ing dat a from a delim it ed t ext file int o an applicat ion is usually painless. Even if you need t o change t he delim it er from one charact er t o anot her ( from a com m a t o a colon, for exam ple) , you can choose from m any t ools t hat perform sim ple charact er subst it ut ion wit h great ease. However, one com m on sit uat ion is not solved as easily: m any business applicat ions export dat a int o a space- or com m a- delim it ed file, enclosing individual fields in double quot at ion m arks. These fields oft en cont ain t he delim it er charact er. I m port ing such a file int o an applicat ion t hat processes only one delim it er ( Post greSQL for exam ple) m ay result in an incorrect int erpret at ion of t he dat a. This is one of t hose sit uat ions where t he user should feel lucky if t he process fails. One solut ion is t o writ e a script t hat t racks t he use of double quot es t o det erm ine whet her it is working wit hin a t ext field. This is doable by creat ing a variable t hat act s as a t ext / nont ext swit ch for t he charact er subst it ut ion process. The script should change t he delim it er t o a m ore appropriat e charact er, leave t he delim it ers t hat were enclosed in double quot es unchanged, and rem ove t he double quot es. Rat her t han m ake t he changes t o t he original dat afile, it 's safer t o writ e t he edit ed dat a t o a new file.

2.6.1 Attacking the Problem The following algorit hm m eet s our needs: 1. Creat e t he swit ch variable and assign it t he value of 1, m eaning " nont ext " . We'll declare t he variable tswitch and define it as tswitch = 1. 2. Creat e a variable for t he delim it er and define it . We'll use t he variable delim wit h a space as t he delim it er, so delim = ' '. 3. Decide on a bet t er delim it er. We'll use t he t ab charact er, so new_delim = '\t'. 4. Open t he dat afile for reading. 5. Open a new file for writ ing. Now, for every charact er in t he dat afile: 1. Read a charact er from t he dat afile. 2. I f t he charact er is a double quot at ion m ark, tswitch = tswitch * -1. 3. I f t he charact er equals t he charact er in delim and tswitch equals 1, writ e new_delim t o t he new file. 4. I f t he charact er equals t hat in delim and tswitch equals - 1, writ e t he value of delim t o t he new file. 5. I f t he charact er is anyt hing else, writ e t he charact er t o t he new file.

2.6.2 The Code The Pyt hon script redelim .py im plem ent s t he preceding algorit hm . I t prom pt s t he user for t he original dat afile and a nam e for t he new dat afile. The delim and new_delim variables are hardcoded, but t hose are easily changed wit hin t he script .

- 91 -

This script copies a space- delim it ed t ext file wit h t ext values in double quot es t o a new, t abdelim it ed file wit hout t he double quot es. The advant age of using t his script is t hat it leaves spaces t hat were wit hin double quot es unchanged. There are no com m and- line argum ent s for t his script . The script will prom pt t he user for source and dest inat ion file inform at ion. You can redefine t he variables for t he original and new delim it ers, delim and new_delim, in t he script as needed. #!/usr/local/bin/python import os

print """ Change text file delimiters.

# Ask user for source and target files. sourcefile = raw_input('Please enter the path and name of the source file:') targetfile = raw_input('Please enter the path and name of the target file:')

# Open files for reading and writing. source = open(sourcefile,'r') dest

= open(targetfile,'w')

# The variable 'm' acts as a text/non-text switch that reminds python # whether it is working within a text or non-text data field. tswitch = 1

# If the source delimiter that you want to change is not a space, # redefine the variable delim in the next line. delim = ' '

# If the new delimiter that you want to change is not a tab, # redefine the variable new_delim in the next line. new_delim = '\t'

for charn in source.read( ):

- 92 -

if tswitch =

= 1:

if charn =

= delim:

dest.write(new_delim) elif charn =

= '\"':

tswitch = tswitch * -1 else: dest.write(charn) elif tswitch =

= -1:

if charn =

= '\"':

tswitch = tswitch * -1 else: dest.write(charn)

source.close( ) dest.close( )

Use of redelim .py assum es t hat you have inst alled Pyt hon, which is available t hrough t he port s collect ion or as a binary package. The Pyt hon m odule used in t his code is inst alled by default .

2.6.3 Hacking the Hack I f you prefer working wit h Perl, DBD: : AnyDat a is anot her good solut ion t o t his problem .

2.6.4 See Also •

The Pyt hon hom e page ( ht t p: / / www.pyt hon.org/ )

- 93 -

Hack 18 DOS Floppy Manipulation

Br in g sim plicit y ba ck t o u sing floppie s. I f you're like m any Unix users, you originally cam e from a Windows background. Rem em ber your init ial shock t he first t im e you t ried t o use a floppy on a Unix syst em ? Didn't Windows seem so m uch sim pler? Forever gone seem ed t he days when you could sim ply insert a floppy, copy som e files over, and rem ove t he disk from t he drive. I nst ead, you were expect ed t o plunge int o t he int ricacies of t he mount com m and, only t o discover t hat you didn't even have t he right t o use t he floppy drive in t he first place! There are several ways t o m ake using floppies m uch, m uch easier on your FreeBSD syst em . Let 's st art by t aking st ock of t he default m echanism s for m anaging floppies.

2.7.1 Mounting a Floppy Suppose I have form at t ed a floppy on a Windows syst em , copied som e files over, and now want t o t ransfer t hose files t o m y FreeBSD syst em . I n realit y, t hat floppy is a st orage m edia. Since it is st oring files, it needs a filesyst em in order t o keep t rack of t he locat ions of t hose files. Because t hat floppy was form at t ed on a Windows syst em , it uses a filesyst em called FAT12. I n Unix, a filesyst em can't be accessed unt il it has been m ount ed. This m eans you have t o use t he mount com m and before you can access t he content s of t hat floppy. While t his m ay seem st range at first , it act ually gives Unix m ore flexibilit y. An adm inist rat or can m ount and unm ount filesyst em s as t hey are needed. Not e t hat I used t he word adm inist rat or. Regular users don't have t his abilit y, by default . We'll change t hat short ly. Unix also has t he addit ional flexibilit y of being able t o mount different filesyst em s. I n Windows, a floppy will always cont ain t he FAT12 filesyst em . BSD underst ands floppies form at t ed wit h eit her FAT12 or UFS, t he Unix File Syst em . As you m ight expect from t he nam e, t he UFS filesyst em is assum ed unless you specify ot herwise. For now, becom e t he superuser and let 's pick apart t he default invocat ion of t he mount com m and: % su Password: # mount -t msdos /dev/fd0 /mnt #

I used t he t ype ( -t) swit ch t o indicat e t hat t his floppy was form at t ed from an msdos- based syst em . I could have used t he mount_msdosfs com m and inst ead: # mount_msdosfs /dev/fd0 /mnt

- 94 -

Bot h com m ands t ake t wo argum ent s. The first indicat es t he device t o be m ount ed. /dev/fd0 represent s t he first ( 0) floppy drive ( fd) device ( /dev) . The second argum ent represent s t he m ount point . A m ount point is sim ply an em pt y direct ory t hat act s as a point er t o t he m ount ed filesyst em . Your FreeBSD syst em com es wit h a default m ount point called / m nt . I f you prefer, creat e a different m ount point wit h a m ore useful nam e. Just rem em ber t o keep t hat direct ory em pt y so it will be available as a m ount point , because any files in your m ount point will becom e hidden and inaccessible when you m ount a device over it .

This can be a feat ure in it self if you have a filesyst em t hat should always be m ount ed. Place a README file in / m nt / im port ant _direct ory cont aining: " I f you can see t his file, cont act t he adm inist rat or at t his num ber . . . ."

I n t his exam ple, I 'll creat e a m ount point called / floppy, which I 'll use in t he rest of t he exam ples in t his hack: # mkdir /floppy

2.7.2 Common Error Messages This is a good place t o explain som e com m on error m essages. Trust m e, I experienced t hem all before I becam e proficient at t his whole mount business. At t he t im e, I wished for a list ing of error m essages so I could figure out what I had done wrong and how t o fix it . Let 's t ake a look at t he out put of t his com m and: # mount /dev/fd0 /mnt mount: /dev/fd0 on /mnt: incorrect super block

Rem em ber m y first mount com m and? I know it worked, as I j ust received m y prom pt back. I know t his com m and didn't work, because mount inst ead wrot e m e a m essage explaining why it did not do what I asked. That error m essage isn't act ually as bad as it sounds. I forgot t o include t he t ype swit ch, m eaning mount assum ed I was using UFS. Since t his is a FAT12 floppy, it sim ply didn't underst and t he filesyst em . This error m essage also looks part icularly nast y: fd0: hard error cmd=read fsbn 0 of 0-3 (No status) msdosfs: /dev/fd0: Input/output error

- 95 -

I f you get t hat one, quickly reach down and push in t he floppy before anyone else not ices. You forgot t o insert it int o t he bay. Here's anot her error m essage: msdosfs: /dev/fd0: Operation not permitted

Oops. Looks like I didn't becom e t he superuser before t rying t hat mount com m and. How about t his one: mount: /floppy: No such file or directory

Looks like I forgot t o m ake t hat m ount point first . A mkdir /floppy should fix t hat one. The one error m essage you do not want t o see is a syst em panic followed by a reboot . I t t ook m e a while t o break m yself of t he habit of j ust ej ect ing a floppy once I had copied over t he files I want ed. That 's som et hing you j ust don't do in Unix land. You m ust first warn your operat ing syst em t hat you have finished using a filesyst em before you physically rem ove it from t he com put er. Ot herwise, when it goes out looking for a file, it will panic when it realizes t hat it has j ust disappeared off of t he edge of t he universe! ( Well, t he com put er's universe anyway.) Put yourself in your operat ing syst em 's shoes for a m inut e. The user ent rust ed som et hing im port ant t o your care. You blinked for j ust a split second and it was gone, nowhere t o be found. You'd panic t oo!

2.7.3 Managing the Floppy How do you warn your operat ing syst em t hat t he universe has shrunk? You unm ount t he floppy before you ej ect it from t he floppy bay. Not e t hat t he act ual com m and used is m issing t he first n and is inst ead spelled umount: # umount /floppy

Also, t he only argum ent is t he nam e of your m ount point . I n t his exam ple, it 's / floppy. How can you t ell if a floppy is m ount ed? The disk free com m and will t ell you: # df Filesystem /dev/ad0s1a devfs /dev/ad0s1e /dev/ad0s1f /dev/ad0s1d /dev/fd0

1K-blocks

Used

Avail Capacity

257838

69838

167374

29%

1

1

0

100%

/dev

257838

616

236596

0%

/tmp

13360662 2882504 9409306

23%

/usr /var

257838

28368

208844

12%

1424

1

1423

0%

- 96 -

Mounted on /

/floppy

as will t he mount com m and wit h no argum ent s: # mount /dev/ad0s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/ad0s1e on /tmp (ufs, local, soft-updates) /dev/ad0s1f on /usr (ufs, local, soft-updates) /dev/ad0s1d on /var (ufs, local, soft-updates) /dev/fd0 on /floppy

(msdosfs, local)

This syst em current ly has a floppy / dev/ fd0 m ount ed on / floppy, m eaning you'll need t o issue t he umount com m and before ej ect ing t he floppy. Several ot her filesyst em s are also m ount ed, yet I only used t he mount com m and on m y floppy drive. When did t hey get m ount ed and how? The answer is in / et c/ fst ab , which cont rols which filesyst em s t o m ount at boot t im e. Here's m y / et c/ fst ab; it 's pret t y sim ilar t o t he earlier out put from df: # more /etc/fstab # Device

Mountpoint

FStype

Options

Dump

Pass#

/dev/ad0s1b

none

swap

sw

0

0

/dev/ad0s1a

/

ufs

rw

1

1

/dev/ad0s1e

/tmp

ufs

rw

2

2

/dev/ad0s1f

/usr

ufs

rw

2

2

/dev/ad0s1d

/var

ufs

rw

2

2

/dev/acd0

/cdrom

cd9660

ro,noauto

0

0

proc

/proc

procfs

rw

0

0

linproc

/compat/linux/proc

linprocfs

rw

0

0

Each m ount able filesyst em has it s own line in t his file. Each has it s own unique m ount point and it s filesyst em t ype list ed. See how t he / cdrom m ount point has t he opt ions ro,noauto inst ead of rw? The noauto t ells your syst em not t o m ount your CD- ROM at boot up. That is a good t hing—if t here's no CD in t he bay at boot t im e, t he kernel will eit her give an error m essage or pause for a few seconds, looking for t hat filesyst em . However, you can m ount a dat a CD- ROM at any t im e by sim ply t yping: # mount /cdrom

- 97 -

That com m and was short er t han t he usual mount com m and for one reason: t here was an ent ry for / cdrom in / et c/ fst ab. That m eans you can short en t he com m and t o m ount a floppy by creat ing a sim ilar ent ry for / floppy. Sim ply add t his line t o / et c/ fst ab: /dev/fd0

/floppy

msdos

rw,noauto

0

0

Test your change by insert ing a floppy and issuing t his com m and: # mount /floppy

I f you receive an error, check / et c/ fst ab for a t ypo and t ry again.

2.7.4 Allowing Regular Users to Mount Floppies Now t hat t he superuser can quickly m ount floppies, let 's give regular users t his abilit y. First , we have t o change t he default set t ing of t he vfs.usermount variable: # sysctl vfs.usermount=1 vfs.usermount: 0 -> 1

By changing t he default 0 t o a 1, we've j ust enabled users t o m ount virt ual filesyst em s. However, don't worry about your users running am ok wit h t his new freedom —t he devices t hem selves are st ill owned by root . Check out t he perm issions on t he floppy device: # ls -l /dev/fd0 crw-r-----

1 root

operator

9,

0 Nov 28 08:31 /dev/fd0

I f you'd like any user t o have t he right t o m ount a floppy, change t he perm issions so everyone has read and writ e access: # chmod 666 /dev/fd0

Now, if you don't want every user t o have t his right , you could creat e a group, add t he desired users t o t hat group, and assign t hat group perm issions t o / dev/ fd0.

You're alm ost t here. The only kicker is t hat t he user has t o own t he m ount point . The best place t o put a user's m ount point is in his hom e direct ory. So, logged in as your usual user account : % mkdir ~/floppy

- 98 -

Now, do you t hink t he mount com m and will recognize t hat new m ount point ? % mount ~/floppy mount: /home/dru/floppy: unknown special file or file system

Oh boy. Looks like we're back t o square one, doesn't it ? Rem em ber, t hat ent ry in / et c/ fst ab only refers t o root 's m ount point , so I can't use t hat short cut t o refer t o m y own m ount point . While it 's great t o have t he abilit y t o use t he mount com m and, I 'm t ruly t oo lazy t o have t o t ype out mount -t msdos /dev/fd0 ~/floppy, let alone rem em ber it . Thank goodness for aliases. Try adding t hese lines t o t he alias sect ion of your ~ .cshrc file: alias mf

mount -t msdos /dev/fd0 ~/floppy

alias uf

umount ~/floppy

Now you sim ply need t o t ype mf whenever you want t o m ount a floppy and uf when it 's t im e t o unm ount t he floppy. Or perhaps you'll prefer t o creat e a keyboard short cut [ H a ck # 4] .

2.7.5 Formatting Floppies Now t hat you can m ount and unm ount floppies wit h t he best of t hem , it 's t im e t o learn how t o form at t hem . Again, let 's st art wit h t he default invocat ions required t o form at a floppy, t hen m ove on t o som e ways t o sim plify t he process. When you form at a floppy on a Windows or DOS syst em , several event s occur: 1. The floppy is low- level form at t ed, m arking t he t racks and sect ors ont o t he disk. 2. A filesyst em is inst alled ont o t he floppy, along wit h t wo copies of it s FAT t able. 3. You are given t he opport unit y t o give t he floppy a volum e label. The sam e process also has t o occur when you form at a floppy on a FreeBSD syst em . On a 5.x syst em , t he order goes like t his: % fdformat -f 1440 /dev/fd0 Format 1440K floppy `/dev/fd0'? (y/n): y Processing ----------------------------------------

% bsdlabel -w /dev/fd0 fd1440

% newfs_msdos /dev/fd0 /dev/fd0: 2840 sectors in 355 FAT12 clusters (4096 bytes/cluster) bps=512 spc=8 res=1 nft=2 rde=512 sec=2880 mid=0xf0 spf=2 spt=18 hds=2 hid=0

- 99 -

First , not ice t hat we don't use t he mount com m and. You can't mount a filesyst em before you have a filesyst em ! ( You do have t o have t he floppy in t he drive, t hough.) Take a look at t he t hree st eps: 1. fdformat does t he low- level form at . 2. bsdlabel creat es t he volum e label. 3. newfs_msdos inst alls t he FAT12 filesyst em . I f I see t he following error m essage when I t ry t o mount t he floppy, I 'll realize t hat I forgot t hat t hird st ep: % mf msdosfs: /dev/fd0: Invalid argument

Because m y mf m ount floppy alias uses t he msdos filesyst em , it will com plain if t he floppy isn't form at t ed wit h FAT12.

2.7.6 Automating the Format Process Any t hree- st ep process is j ust begging t o be put int o a shell script . I like t o keep t hese script s under ~ / bin. I f you don't have t his direct ory yet , creat e it . Then creat e a script called ff ( for form at floppy) : % cd % mkdir bin % cd bin % vi ff #!/bin/sh #this script formats a floppy with FAT12 #that floppy can also be used on a Windows system

# first, remind the user to insert the floppy echo "Please insert the floppy and press enter" read pathname

# then, proceed with the three format steps

fdformat -f 1440 /dev/fd0 bsdlabel -w /dev/fd0 fd1440 newfs_msdos /dev/fd0 echo "Format complete."

- 100 -

Not e t hat t his script is basically t hose t hree com m ands, wit h com m ent s t hrown in so I rem em ber what t he script does. The only new part is t he read pathname line. I added it t o force t he user t o press Ent er before t he script proceeds. Rem em ber t o m ake t he script execut able: % chmod +x ff

I 'll t hen ret urn t o m y hom e direct ory and see how it works. Since I use t he C shell, I 'll use t he rehash com m and t o m ake t he shell aware t hat t here is a new execut able in m y pat h: % cd % rehash % ff Please insert the floppy and press enter

Format 1440K floppy `/dev/fd0'? (y/n): y Processing ---------------------------------------/dev/fd0: 2840 sectors in 355 FAT12 clusters (4096 bytes/cluster) bps=512 spc=8 res=1 nft=2 rde=512 sec=2880 mid=0xf0 spf=2 spt=18 hds=2 hid=0 Format complete.

Not t oo bad. I can now m anipulat e floppies wit h m y own cust om mf, uf, and ff com m ands.

2.7.7 See Also • • • • •

•

man fstab man fdformat man bsdlabel man newfs The Creat ing and Using Floppies sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ floppies.ht m l) The Mount ing and Unm ount ing File Syst em s sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ m ount unm ount .ht m l)

- 101 -

Hack 19 Access Windows Shares Without a Server

Sha r e file s be t w e e n W in dow s a n d Fr e e BSD w it h a m in im um of fu ss. You've probably heard of som e of t he Unix ut ilit ies available for accessing files residing on Microsoft syst em s. For exam ple, FreeBSD provides t he mount_smbfs and smbutil ut ilit ies t o m ount Windows shares and view or access resources on a Microsoft net work. However, bot h of t hose ut ilit ies have a caveat : t hey require an SMB server. The assum pt ion is t hat som ewhere in your net work t here is at least one NT or 2000 Server. Not all net works have t he budget or t he adm inist rat ive expert ise t o allow for com m ercial server operat ing syst em s. Sure, you can inst all and configure Sam ba, but isn't t hat overkill for, say, a hom e or very sm all office net work? Som et im es you j ust want t o share som e files bet ween a Windows 9x syst em and a Unix syst em . I t 's a m at t er of using t he right - sized t ool for t he j ob. You don't bring in a backhoe t o plant flowers in a window box.

2.8.1 Installing and Configuring Sharity-Light I f your sm all net work cont ains a m ix of Microsoft and Unix client s, consider inst alling Sharit y- Light on t he Unix syst em s. This applicat ion allows you t o m ount a Windows share from a Unix syst em . FreeBSD provides a port for t his purpose ( see t he Sharit y- Light web sit e for ot her support ed plat form s) : # cd /usr/ports/net/sharity-light # make install clean

Since Sharit y- Light is a com m and- line ut ilit y, you should be fam iliar wit h UNC or t he Universal Nam ing Convent ion. UNC is how you refer t o Microsoft shared resources from t he com m and line. A UNC looks like \\NetBIOSname\sharename. I t st art s wit h double backslashes, t hen cont ains t he Net BI OS nam e of t he com put er t o access and t he nam e of t he share on t hat com put er. Before using Sharit y- Light , you need t o know t he Net BI OS nam es of t he com put ers you wish t o access. I f you have m ult iple m achines running Microsoft operat ing syst em s, t he quickest way t o view each syst em 's nam e is wit h nbtstat. From one of t he Windows syst em s, open a com m and prom pt and t ype: C:> nbtstat -A 192.168.2.10

NETBIOS Remote Machine Name Table

Name

Type

Status

----------------------------------------LITTLE_WOLF

UNIQUE

Registered

- 102 -



Repeat for each I P address in your net work. Your out put will be several lines long, but t he ent ry ( usually t he first ) cont aining is t he one wit h t he nam e you're int erest ed in. I n t his exam ple, LITTLE_WOLF is t he Net BI OS nam e associat ed wit h 192.168.2.10. Even t hough nbtstat ? indicat es t hat -A is used t o view a rem ot e syst em , it also works wit h t he I P address of t he local syst em . This allows you t o check all of t he I P addresses in your net work from t he sam e syst em .

Once you know which I P addresses are associat ed wit h which Net BI OS nam es, you'll need t o add t hat inform at ion t o / et c/ host s on your Unix syst em s: # more /etc/hosts 127.0.0.1

localhost

192.168.2.95

genisis

#this system

192.168.2.10

little_wolf

#98 system sharing cygwin2

You'll also need t o know t he nam es of t he shares you wish t o access. Again, from a Microsoft com m and prom pt , repeat t his com m and for each Net BI OS nam e and m ake not e of your result s: C:> net view \\little_wolf Shared resources at \\LITTLE_WOLF

Sharename

Type

Comment

--------------------------------------CYGWIN2

Disk

The command was completed successfully.

Here t he com put er known as LITTLE_WOLF has only one share, t he CYGWI N2 direct ory. Finally, you'll need a m ount point on your Unix syst em , so you m ight as well give it a useful nam e. Since t he t ypical floppy m ount point is / floppy and t he t ypical CD m ount point is / cdrom , let 's use / windows: # mkdir /windows

- 103 -

2.8.2 Accessing Microsoft Shares Once you know t he nam es of your com put ers and shares, using Sharit y- Light is very easy. As t he superuser, m ount t he desired share: # shlight //little_wolf/cygwin2 /windows Password: Using port 49923 for NFS. Wat ch your slashes. Microsoft uses t he backslash ( \) at t he com m and line, whereas Unix and Sharit y- Light use t he forward slash ( /) .

Not e t hat I was prom pt ed for a password because Windows 9x and ME users have t he opt ion of password prot ect ing t heir shares. This part icular share did not have a password, so I sim ply pressed Ent er. Adding -n t o t he previous com m and will forego t he password prom pt . Type shlight -h t o see all available opt ions.

However, if t he share is on a Windows NT Workst at ion, 2000 Pro, or XP syst em , you m ust provide a usernam e and password valid on t hat syst em . The synt ax is: # shlight //2000pro/cdrom /windows -U username -P password

Once t he share is m ount ed, it works like any ot her m ount point . Depending on t he perm issions set on t he share, you should be able t o browse t hat shared direct ory, copy over or add files, and m odify files. When you're finished using t he share, unm ount it : $ unshlight /windows

2.8.3 See Also • • •

The Sharit y- Light README and FAQ ( / usr/ local/ share/ doc/ Sharit y- Light / ) The Sharit y- Light web sit e ( ht t p: / / www.obdev.at / product s/ sharit y- light / index.ht m l) The Sam ba web sit e ( ht t p: / / www.sam ba.org/ )

- 104 -

Hack 20 Deal with Disk Hogs

For t u na t e ly, you no lon ge r ha ve t o be a scr ipt gur u or a find w iza r d j u st t o k e e p up w it h w ha t is h a ppe n in g on you r disk s. Think for a m om ent . What t ypes of files are you always chasing aft er so t hey don't wast e resources? Your list probably includes t em p files, core files, and old logs t hat have already been archived. Did you know t hat your syst em already cont ains script s capable of cleaning out t hose files? Yes, I 'm t alking about your periodic script s.

2.9.1 Periodic Scripts You'll find t hese script s in t he following direct ory on a FreeBSD syst em : % ls /etc/periodic/daily | grep clean 100.clean-disks 110.clean-tmps 120.clean-preserve 130.clean-msgs 140.clean-rwho 150.clean-hoststat

Are you using t hese script s? To find out , look at your / et c/ periodic.conf file. What , you don't have one? That m eans you've never t weaked your default configurat ions. I f t hat 's t he case, copy over t he sam ple file and t ake a look at what 's available: # cp /etc/defaults/periodic.conf /etc/periodic.conf # more /etc/periodic.conf

2 .9 .1 .1 da ily_ cle a n_ disk s Let 's st art wit h daily_clean_disks. This script is ideal for finding and delet ing files wit h cert ain file ext ensions. You'll find it about t wo pages int o periodic.conf, in t he Daily options sect ion, where you m ay not e t hat it 's not enabled by default . Fort unat ely, configuring it is a heck of a lot easier t han using cron t o schedule a com plex find st at em ent . Before you enable any script , t est it first , especially if it 'll delet e files based on pat t ern- m at ching rules. Back up your syst em first ! For exam ple, suppose you want t o delet e old logs wit h t he .bz2

- 105 -

ext ension. I f you're not careful when you craft your daily_clean_disks_files line, you m ay end up inadvert ent ly delet ing all files wit h t hat ext ension. Any user who has j ust com pressed som e im port ant dat a will be very m iffed when she finds t hat her dat a has m yst eriously disappeared.

Let 's t est t his scenario. I 'd like t o prune all .core files and any logs older t han .0.bz2. I 'll edit t hat sect ion of / et c/ periodic.conf like so: # 100.clean-disks daily_clean_disks_enable="YES"

# Delete files daily

daily_clean_disks_files="*.[1-9].bz2 *.core"

# delete old logs, cores

daily_clean_disks_days=1

# on a daily basis

daily_clean_disks_verbose="YES"

# Mention files deleted

Not ice m y pat t ern- m at ching expression for t he .bz2 files. My expression m at ches any filenam e ( *) followed by a dot and a num ber from one t o nine ( .[1-9]) , followed by anot her dot and t he .bz2 ext ension. Now I 'll verify t hat m y syst em has been backed up, and t hen m anually run t hat script . As t his script is fairly resource- int ensive, I 'll do t his t est when t he syst em is under a light load: # /etc/periodic/daily/100.clean-disks

Cleaning disks: /usr/ports/distfiles/MPlayer-0.92.tar.bz2 /usr/ports/distfiles/gnome2/libxml2-2.6.2.tar.bz2 /usr/ports/distfiles/gnome2/libxslt-1.1.0.tar.bz2

Darn. Looks like I inadvert ent ly nuked som e of m y dist files. I 'd bet t er be a bit m ore explicit in m y m at ching pat t ern. I 'll t ry t his inst ead: # delete old logs, cores daily_clean_disks_files="messages.[1-9].bz2 *.core"

# /etc/periodic/daily/100.clean-disks

Cleaning disks: /var/log/messages.1.bz2 /var/log/messages.2.bz2

- 106 -

/var/log/messages.3.bz2 /var/log/messages.4.bz2

That 's a bit bet t er. I t didn't delet e / var/ log/ m essages or / var/ log/ m essages.1.bz2, which I like t o keep on disk. Rem em ber, always t est your pat t ern m at ching before scheduling a delet ion script . I f you keep t he verbose line at YES, t he script will report t he nam es of files it delet es. 2 .9 .1 .2 da ily_ cle a n_ t m ps The ot her cleaning script s are quit e st raight forward t o configure. Take daily_clean_tmps, for exam ple: # 110.clean-tmps daily_clean_tmps_enable="NO"

# Delete stuff daily

daily_clean_tmps_dirs="/tmp"

# Delete under here

daily_clean_tmps_days="3"

# If not accessed for

daily_clean_tmps_ignore=".X*-lock quota.user quota.group" # Don't delete # these daily_clean_tmps_verbose="YES"

# Mention files deleted

This is a quick way t o clean out any t em porary direct ories. Again, you get t o choose t he locat ions of t hose direct ories. Here is a quick way t o find out which direct ories nam ed t m p are on your syst em : # find / -type d -name tmp /tmp /usr/tmp /var/spool/cups/tmp /var/tmp

That com m and asks find t o st art at root ( /) and look for any direct ories ( -type d) nam ed t m p ( -name tmp) . I f I want ed t o clean t hose daily, I 'd configure t hat sect ion like so: # 110.clean-tmps

# Delete stuff daily daily_clean_tmps_enable="YES" daily_clean_tmps_dirs="/tmp /usr/tmp /var/spool/cups/tmp /var/tmp"

- 107 -

# If not accessed for daily_clean_tmps_days="1"

# Don't delete these daily_clean_tmps_ignore=".X*-lock quota.user quota.group"

# Mention files deleted daily_clean_tmps_verbose="YES"

Again, I im m ediat ely t est t hat script aft er saving m y changes: # /etc/periodic/daily/110.clean-tmps

Removing old temporary files: /var/tmp/gconfd-root

This script will not delet e any locked files or t em porary files current ly in use. This is an excellent feat ure and yet anot her reason t o run t his script on a daily basis, preferably at a t im e when few users are on t he syst em . 2 .9 .1 .3 da ily_ cle a n_ pr e se r ve Moving on, t he next script is daily_clean_preserve: # 120.clean-preserve daily_clean_preserve_enable="YES"

# Delete files daily

daily_clean_preserve_days=7

# If not modified for

daily_clean_preserve_verbose="YES"

# Mention files deleted

What exact ly is preserve? The answer is in man hier. Use t he m anpage search funct ion ( t he / key) t o search for t he word preserve: # man hier /preserve preserve/ temporary home of files preserved after an accidental death of an editor; see (ex)1

- 108 -

Now t hat you know what t he script does, see if t he default set t ings are suit ed for your environm ent . This script is run daily, but keeps preserved files unt il t hey are seven days old. The last t hree clean script s deal wit h cleaning out old files from msgs, rwho and sendmail's host st at cache. See man periodic.conf for m ore det ails. I ncident ally, you don't have t o wait unt il it is t im e for periodic t o do it s t hing; you can m anually run any periodic script at any t im e. You'll find t hem all in subdirect ories of / et c/ periodic/ .

2.9.2 Limiting Files I nst ead of wait ing for a daily process t o clean up any spills, you can t weak several knobs t o prevent t hese files from being creat ed in t he first place. For exam ple, t he C shell it self provides lim it s, any of which are excellent candidat es for a cust om ized dot .cshrc file [ H a ck # 9] . To see t he possible lim it s and t heir current values: % limit cputime

unlimited

filesize

unlimited

datasize

524288 kbytes

stacksize

65536 kbytes

coredumpsize

unlimited

memoryuse

unlimited

vmemoryuse

unlimited

descriptors

4557

memorylocked

unlimited

maxproc

2278

sbsize

unlimited

You can t est a lim it by t yping it at t he com m and line; it will rem ain for t he durat ion of your current shell. I f you like t he lim it , m ake it perm anent by adding it t o .cshrc. For exam ple: % limit filesize 2k % limit | grep filesize filesize

2 kbytes

will set t he m axim um file size t hat can be creat ed t o 2 KB. The limit com m and support s bot h k for kilobyt es and m for m egabyt es. Do not e t hat t his lim it does not affect t he t ot al size of t he area available t o st ore files, j ust t he size of a newly creat ed file. See t he Quot as sect ion of t he FreeBSD Handbook if you int end t o lim it disk space usage.

- 109 -

Having creat ed a file lim it , you'll occasionally want t o exceed it . For exam ple, consider decom pressing a file: % uncompress largefile.Z Filesize limit exceeded

% unlimit filesize % uncompress largefile.Z %

The unlimit com m and will allow m e t o override t he file- size lim it t em porarily ( for t he durat ion of t his shell) . I f you really do want t o force your users t o st ick t o lim it s, read man limits. Now back t o shell lim it s. I f you don't know what a core file is, you probably don't need t o collect t hem . Sure, periodic can clean t hose files out for you, but why m ake t hem in t he first place? Core files are large. You can lim it t heir size wit h: limit coredumpsize 1m

That com m and will lim it a core file t o 1 MB, or 1024 KB. To prevent core files com plet ely, set t he size t o 0: limit coredumpsize 0

I f you're int erest ed in t he rest of t he built - in lim it s, you'll find t hem in man tcsh . Searching for coredumpsize will t ake you t o t he right spot .

2.9.3 The Other BSDs The preceding discussion is based on FreeBSD. Ot her BSD syst em s ship wit h sim ilar script s t hat do ident ical t asks, but t hey are kept in a single file inst ead of in a separat e direct ory. 2 .9 .3 .1 N e t BSD For daily, weekly, and m ont hly t asks, Net BSD uses t he / et c/ daily, / et c/ weekly, and / et c/ m ont hly script s, whose behavior is cont rolled wit h t he / et c/ daily.conf, / et c/ weekly.conf, and / et c/ m ont hly.conf configurat ion files. For m ore inform at ion about t hem , read man daily.conf, man weekly.conf, and man monthly.conf. 2 .9 .3 .2 Ope n BSD OpenBSD uses t hree script s, / et c/ daily, / et c/ weekly, and / et c/ m ont hly. You can learn m ore about t hem by reading man daily.

2.9.4 See Also man

periodic.conf, man limits, man tcsh, The Quot as sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO88591/ books/ handbook/ quot as.ht m l)

- 110 -

Hack 21 Manage Temporary Files and Swap Space

Add m or e t e m por a r y or sw a p spa ce w it h ou t r e pa r t it ion ing. When you inst all any operat ing syst em , it 's im port ant t o allocat e sufficient disk space t o hold t em porary and swap files. I deally, you already know t he opt im um sizes for your syst em so you can part it ion your disk accordingly during t he inst all. However, if your needs change or you wish t o opt im ize your init ial choices, your solut ion doesn't have t o be as drast ic as a repart it ion—and reinst all—of t he syst em . man tuning has som e pract ical advice for guesst im at ing t he appropriat e size of swap and your ot her part it ions.

2.10.1 Clearing /tmp Unless you specifically chose ot herwise when you part it ioned your disk, t he inst aller creat ed a / t m p filesyst em for you: % grep tmp /etc/fstab /dev/ad0s1e

/tmp

ufs

rw

2

2

% df -h /tmp Filesystem

Size

Used

/dev/ad0s1e

252M

614K

Avail Capacity 231M

0%

Mounted on /tmp

Here I searched / et c/ fst ab for t he / t m p filesyst em . This part icular filesyst em is 256 MB in size. Only a sm all port ion cont ains t em porary files. The df ( disk free) com m and will always show you a num ber lower t han t he act ual part it ion size. This is because eight percent of t he filesyst em is reserved t o prevent users from inadvert ent ly overflowing a filesyst em . See man tunefs for det ails.

I t 's always a good idea t o clean out / t m p periodically so it doesn't overflow wit h t em porary files. Consider t aking advant age of t he built - in periodic script / et c/ periodic/ daily/ 110.cleant m ps [ H a ck # 2 0 ] . You can also clean out / t m p when t he syst em reboot s by adding t his line t o / et c/ rc.conf: clear_tmp_enable="YES"

- 111 -

2.10.2 Moving /tmp to RAM Anot her opt ion is t o m ove / t m p off of your hard disk and int o RAM. This has t he built - in advant age of aut om at ically clearing t he filesyst em when you reboot , since t he cont ent s of RAM are volat ile. I t also offers a perform ance boost , since RAM access t im e is m uch fast er t han disk access t im e. Before m oving / t m p, ensure you have enough RAM t o support your desired / t m p size. This com m and will show t he am ount of inst alled RAM: % dmesg | grep memory real memory

= 335462400 (319 MB)

avail memory = 320864256 (306 MB)

Also check t hat your kernel configurat ion file cont ains device md ( or m em ory disk) . The GENERIC kernel does; if you've cust om ized your kernel, double- check t hat you st ill have md support : % grep -w md /usr/src/sys/i386/conf/CUSTOM device

md

# Memory "disks"

Changing t he / t m p line in / et c/ fst ab as follows will m ount a 64 MB / t m p in RAM: md /tmp mfs rw,-s64m 2 0

Next , unm ount / t m p ( which is current ly m ount ed on your hard drive) and rem ount it using t he new ent ry in / et c/ fst ab: # umount /tmp # mount /tmp

# df -h /tmp Filesystem /dev/md0

Size

Used

63M

8.0K

Avail Capacity 58M

0%

Mounted on /tmp

Not ice t hat t he filesyst em is now md0, t he first m em ory disk, inst ead of ad0s1e, a part it ion on t he first I DE hard drive.

2.10.3 Creating a Swap File on Disk Swap is different t han / t m p. I t 's not a st orage area for t em porary files; inst ead, it is an area where t he filesyst em swaps dat a bet ween RAM and disk. A sufficient swap size can great ly

- 112 -

increase t he perform ance of your filesyst em . Also, if your syst em cont ains m ult iple drives, t his swapping process will be m uch m ore efficient if each drive has it s own swap part it ion. The init ial inst all creat ed a swap filesyst em for you: % grep swap /etc/fstab /dev/ad0s1b

none

swap

sw

0

0

% swapinfo Device

1K-blocks

Used

639688

68

/dev/ad0s1b

Avail Capacity 639620

0%

Type Interleaved

Not e t hat t he swapinfo com m and displays t he size of your swap files. I f you prefer t o see t hat out put in MB, t ry t he swapctl com m and wit h t he -lh flags ( which m ake t he list ing m ore hum an) : % swapctl -lh Device:

1048576-blocks

/dev/ad0s1b

624

Used: 0

To add a swap area, first det erm ine which area of disk space t o use. For exam ple, you m ay want t o place a 128 MB swapfile on / usr. You'll first need t o use dd t o creat e t his as a file full of null ( or zero) byt es. Here I 'll creat e a 128 MB swapfile as / usr/ swap0: # dd if=/dev/zero of=/usr/swap0 bs=1024k count=128 128+0 records in 128+0 records out 134217728 bytes transferred in 4.405036 secs (30469156 bytes/sec)

Next , change t he perm issions on t his file. Rem em ber, you don't want users st oring dat a here; t his file is for t he filesyst em : # chmod 600 /usr/swap0

Since t his is really a file on an exist ing filesyst em , you can't mount your swapfile in / et c/ fst ab. However, you can t ell t he syst em t o find it at boot t im e by adding t his line t o / et c/ rc.conf: swapfile="/usr/swap0"

To st art using t he swapfile now wit hout having t o reboot t he syst em , use mdconfig: # mdconfig -a -t vnode -f /usr/swap0 -u 1 && swapon /dev/md1

- 113 -

The -a flag at t aches t he m em ory disk. -t vnode m arks t hat t he t ype of swap is a file, not a filesyst em . The -f flag set s t he nam e of t hat file: / usr/ swap0. The unit num ber -u 1 m ust m at ch t he nam e of t he m em ory disk / dev/ m d1. Since t his syst em already has / t m p m ount ed on / dev/ m d0, I chose t o m ount swap on / dev/ m d1. && swapon t ells t he syst em t o enable t hat swap device, but only if t he mdconfig com m and succeeded. swapctl should now show t he new swap part it ion: % swapctl -lh Device:

1048576-blocks

Used:

/dev/ad0s1b

624

0

/dev/md1

128

0

2.10.4 Monitoring Swap Changes Whenever you m ake changes t o swap or are considering increasing swap, use systat t o m onit or how your swapfiles are being used in real t im e: % systat -swap

The out put will show t he nam es of your swap areas and how m uch of each is current ly in use. I t will also include a visual indicat ing what percent age of swap cont ains dat a.

2.10.5 OpenBSD Differences You can m ake t his hack work on OpenBSD, as long as you rem em ber t hat t he RAM disk device is rd and it s configurat ion t ool is rdconfig. Read t he relevant m anpages, and you'll be hacking away.

2.10.6 See Also • • • • • • •

man tuning (pract ical advice on / t m p and swap) man md man mdconfig man swapinfo man swapctl man systat The BSD Handbook ent ry on adding swap ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ adding- swapspace.ht m l)

- 114 -

Hack 22 Recreate a Directory Structure Using mtree

Pr e ve n t or r e cove r fr om r m disa st e r s. Som eday t he unt hinkable m ay happen. You're doing som e rout ine m aint enance and are dist ract ed by a phone call or perhaps anot her em ployee's quest ion. A m om ent lat er, you're faced wit h t he awful realizat ion t hat your fingers t yped eit her a rm * or a rm -R in t he wrong place, and now a port ion of your syst em has evaporat ed int o not hingness. Painful t hought , isn't it ? Let 's pause for a m om ent t o cat ch our breat h and exam ine a few ways t o prevent such a scenario from happening in t he first place. Close your eyes and t hink back t o when you were a fresh- faced newbie and were int roduced t o t he om nipot ent rm com m and. Ret urn t o t he t im e when you act ually read man rm and first discovered t he -i swit ch. " What a great idea," you t hought , " t o be prom pt ed for confirm at ion before irret rievably delet ing a file from disk." However, you soon discovered t hat t his swit ch can be a royal PI TA. Face it , it 's irrit at ing t o deal wit h t he const ant quest ion of whet her you're sure you want t o rem ove a file when you j ust issued t he com m and t o rem ove t hat file.

2.11.1 Necessary Interaction Fort unat ely, t here is a way t o request confirm at ion only when you're about t o do som et hing as rash as rm *. Sim ply m ake a file called - i. Well, act ually, it 's not quit e t hat sim ple. Your shell will com plain if you t ry t his: % touch -i touch: illegal option -- i usage: touch [-acfhm] [-r file] [-t [[CC]Y]MMDDhhmm[.SS]] file ...

You see, t o your shell, - i looks like t he -i swit ch, which touch doesn't have. That 's act ually part of t he m agic. The reason why we want t o m ake a file called - i in t he first place is t o fool your shell: when you t ype rm *, t he shell will expand * int o all of t he files in t he direct ory. One of t hose files will be nam ed - i, and, voila, you've j ust given t he int eract ive swit ch t o rm. So, how do we get past t he shell t o m ake t his file? Use t his com m and inst ead: % touch ./-i

The ./ act s as a sort of separat or inst ruct ion t o t he shell. To t he left of t he ./ go any opt ions t o t he com m and touch; in t his case, t here are none. To t he right of t he ./ is t he nam e of t he file t o touch in " t his direct ory." I n order for t his t o be effect ive, you need t o creat e a file called - i in every direct ory t hat you would like t o prot ect from an inadvert ent rm *.

- 115 -

An alt ernat ive m et hod is t o t ake advant age of t he rmstar shell variable found in t he tcsh shell. This m et hod will always prom pt for confirm at ion of a rm *, regardless of your current direct ory, as long as you always use tcsh. Since t he default shell for t he superuser is tcsh, add t his line t o / root / .cshrc: set rmstar This is also a good line t o add t o / usr/ share/ skel/ dot .cshrc [ H a ck # 9 ] .

I f you want t o t ake advant age of t he prot ect ion im m ediat ely, force t he shell t o reread it s configurat ion file: # source /root/.cshrc

2.11.2 Using mtree Now you know how t o prot ect yourself from rm *. Unfort unat ely, neit her m et hod will save you from a rm -R. I f you do m anage t o blow away a port ion of your direct ory st ruct ure, how do you fix t he m ess wit h a m inim um of fuss, fanfare, and years of t easing from your coworkers? Sure, you can always rest ore from backup, but t hat m eans filling in a form in t riplicat e, carrying it wit h you as you walk t o t he ot her side of t he building where backups are st ored, and sheepishly handing it over t o t he clerk in charge of t ape st orage. Fort unat ely for a hacker, t here is always m ore t han one way t o skin a cat , or in t his case, t o save your skin. That direct ory st ruct ure had t o be creat ed in t he first place, which m eans it can be recreat ed. When you inst alled FreeBSD, it creat ed a direct ory st ruct ure for you. The ut ilit y responsible for t his feat is called mtree. To see which direct ory st ruct ures were creat ed wit h mtree: % ls /etc/mtree/ ./

BSD.root.dist

BSD.x11-4.dist

../

BSD.sendmail.dist

BSD.x11.dist

BSD.include.dist

BSD.usr.dist

BSD.local.dist

BSD.var.dist

Each of t hese files is in ASCI I t ext , m eaning you can read, and m ore int erest ingly, edit t heir cont ent s. I f you're a hacker, I know what you're t hinking. Yes, you can edit a file t o rem ove t he direct ories you don't want and t o add ot her direct ories t hat you do. Let 's st art wit h a sim pler exam ple. Say you've m anaged t o blow away / var. To recreat e it : # mtree -deU -f /etc/mtree/BSD.var.dist -p /var

- 116 -

where:

-d I gnores everyt hing except direct ory files.

-e Doesn't com plain if t here are ext ra files.

-U Recreat es t he original ownerships and perm issions.

-f /etc/mtree/BSD.var.dist Specifies how t o creat e t he direct ory st ruct ure; t his is an ASCI I t ext file if you want t o read up ahead of t im e on what exact ly is going t o happen.

-p /var Specifies where t o creat e t he direct ory st ruct ure; if you don't specify, it will be placed in t he current direct ory. When you run t his com m and, t he recreat ed files will be echoed t o st andard out put so you can wat ch as t hey are creat ed for you. A few seconds lat er, you can: % ls /var ./

crash/

heimdal/

preserve/

../

cron/

lib/

run/

account/

db/

log/

rwho/

at/

empty/

mail/

spool/

backups/

games/

msgs/

yp/

That looks a lot bet t er, but don't breat he t hat sigh of relief quit e yet . You st ill have t o recreat e all of your log files. Yes, / var/ log is st ill glaringly em pt y. Rem em ber, mtree creat es a direct ory st ruct ure, not all of t he files wit hin t hat direct ory st ruct ure. I f you have a direct ory st ruct ure cont aining t housands of files, you're bet t er off grabbing your backup t ape. There is hope for / var/ log, t hough. Rat her t han racking your brain for t he nam es of all of t he m issing log files, do t his inst ead: % more /etc/newsyslog.conf # configuration file for newsyslog

- 117 -

# $FreeBSD: src/etc/newsyslog.conf,v 1.42 2002/09/21 12:07:35 markm Exp $ # # Note: some sites will want to select more restrictive protections than the # defaults.

In particular, it may be desirable to switch many of the 644

# entries to 640 or 600.

For example, some sites will consider the

# contents of maillog, messages, and lpd-errs to be confidential.

In the

# future, these defaults may change to more conservative ones. # # logfilename

[owner:group]

mode count size when

[ZJB]

[/pid_file] [sig_num] /var/log/cron

600

3

100

*

J

/var/log/amd.log

644

7

100

*

J

/var/log/auth.log

600

7

100

*

J

/var/log/kerberos.log

600

7

100

*

J

/var/log/lpd-errs

644

7

100

*

J

/var/log/xferlog

600

7

100

*

J

/var/log/maillog

640

7

*

@T00

J

/var/log/sendmail.st

640

10

*

168

B

/var/log/messages

644

5

100

*

J

/var/log/all.log

600

7

*

@T00

J

/var/log/slip.log

root:network

640

3

100

*

J

/var/log/ppp.log

root:network

640

3

100

*

J

/var/log/security

600

10

100

*

J

/var/log/wtmp

644

3

*

@01T05 B

/var/log/daily.log

640

7

*

@T00

J

/var/log/weekly.log

640

5

1

$W6D0

J

/var/log/monthly.log

640

12

*

$M1D0

J

/var/log/console.log

600

5

100

*

J

There you go, all of t he default log nam es and t heir perm issions. Sim ply touch t he required files and adj ust t heir perm issions accordingly wit h chmod.

- 118 -

2.11.3 Customizing mtree Let 's get a lit t le fancier and hack t he mtree hack. I f you want t o be able t o creat e a hom egrown direct ory st ruct ure, st art by perusing t he inst ruct ions in / usr/ src/ et c/ m t ree/ README. The one rule t o keep in m ind is don't use t abs. I nst ead, use four spaces for indent at ion. Here is a sim ple exam ple: % more MY.test.dist #home grown test directory structure /set type=dir uname=test gname=test mode=0755 . test1 .. test2 subdir2a .. subdir2b .. subsubdir2c

mode=01777

.. .. ..

Not e t hat you can specify different perm issions on different part s of t he direct ory st ruct ure. Next , I 'll apply t his file t o m y current direct ory: # mtree -deU -f MY.test.dist

and check out t he result s: # ls -F test1/ test2/ # ls -F test1 # # ls -F test2

- 119 -

subdir2a/ subdir2b/ # ls -F test2/subdir2b subsubdir2c/

As you can see, mtree can be a real t im esaver if you need t o creat e cust om direct ory st ruct ures when you do inst allat ions. Sim ply t ake a few m om ent s t o creat e a file cont aining t he direct ory st ruct ure and it s perm issions. You'll gain t he added bonus of having a record of t he required direct ory st ruct ure.

2.11.4 See Also • •

man mtree The Linux mtree port ( ht t p: / / www.wie- auch- im m er.de/ m t ree/ )

- 120 -

Hack 23 Ghosting Systems

Do you find yourself inst alling m ult iple syst em s, all cont aining t he sam e operat ing syst em and applicat ions? As an I T inst ruct or, I 'm const ant ly inst alling syst em s for m y next class or t rying t o fix t he ram ificat ions of a m isconfigurat ion from a previous class. As any syst em adm inist rat or can at t est t o, ghost ing or hard drive- cloning soft ware can be a real godsend. Backups are one t hing; t hey ret ain your dat a. However, an im age is a t rue t im esaver—it 's a copy of t he operat ing syst em it self, along wit h any inst alled soft ware and all of your configurat ions and cust om izat ions. I haven't always had t he luxury of a com m ercial ghost ing ut ilit y at hand. As you can well im agine, I 've t ried every hom egrown and open source ghost ing solut ion available. I st art ed wit h various invocat ions of dd, gzip, ssh, and dump, but kept running across t he sam e fundam ent al problem : it was easy enough t o creat e an im age, but inconvenient t o deploy t hat im age t o a fresh hard drive. I t was doable in t he labs t hat used rem ovable drives, but , ot herwise, I had t o open up a syst em , cable in t he drive t o be deployed, copy t he im age, and recable t he drive int o it s own syst em . Forget t he wear and t ear on t he equipm ent ; t hat solut ion wasn't working out t o be m uch of a t im esaver! What I really needed was a floppy t hat cont ained enough int elligence t o go out on t he net work and ret rieve and rest ore an im age. I t ried several open source applicat ions and found t hat Ghost For Unix, g4u, best fit t he bill.

2.12.1 Creating the Ghost Disk You're about t wo m inut es away from creat ing a boot able g4u floppy. Sim ply download g4u1.12fs from ht t p: / / t heat om icm oose.ca/ g4u/ and copy it t o a floppy: # cat g4u-1.12fs > /dev/fd0

Your only ot her requirem ent is a syst em wit h a drive capable of holding your im ages. I t can be any operat ing syst em , as long as it has an inst alled FTP server. I f it 's a FreeBSD syst em , you can configure an FTP server t hrough /stand/sysinstall. Choose Configure from t he m enu, t hen Networking. Use your spacebar t o choose Anon FTP. Choose Yes t o t he configurat ion m essage and accept t he default s by t abbing t o OK. The welcom e m essage is opt ional. Exit sysinstall once you're finished. You'll t hen need t o rem ove t he rem ark ( #) in front of t he FTP line in / et c/ inet d.conf, so it looks like t his: ftp

stream

tcp

nowait

root

/usr/libexec/ftpd

ftpd -l

I f inetd is already running, inform it of t he configurat ion change using killall -1 inetd. Ot herwise, st art inetd by sim ply t yping inetd. To ensure t he service is running:

- 121 -

# sockstat | grep 21 root

inetd

22433

4

tcp4

*:21

*:*

I n t his list ing, t he local syst em is list ening for request s on port 21, and t here aren't any current connect ions list ed in t he rem ot e address sect ion ( *:*) . g4u requires a usernam e and a password before it will creat e or ret rieve an im age. The default account is install, but you can specify anot her user account when you use g4u. To creat e t he install account on a FreeBSD FTP server: # pw useradd install -m -s /bin/csh Make sure t hat t he shell you give t his user is list ed in / et c/ shells or FTP aut hent icat ion will fail.

Then, use passwd install t o give t his account a password you will rem em ber.

2.12.2 Creating an Image Before you creat e an im age, fully configure a t est syst em . For exam ple, in m y securit y lab, I usually inst all t he lat est release of FreeBSD, add m y cust om ized / et c/ m ot d and shell prom pt , configure X, and inst all and configure t he applicat ions st udent s will use during t heir labs. I t 's a good idea t o know ahead of t im e how large t he hard drive is on t he t est syst em and how it has been part it ioned. There are several ways t o find out on a FreeBSD syst em , depending upon how good you are at m at h. One way is t o go back int o /stand/sysinstall and choose Configure t hen Fdisk. The first long line will give t he size of t he ent ire hard drive: Disk name:

ad0

DISK Geometry:

19885 cyls/16 heads/63 sectors = 20044080 sectors (9787MB)

Press q t o exit t his screen. I f you t hen t ype fdisk at t he com m and line, you'll see t he size of your part it ions: # fdisk

The data for partition 1 is: sysid 165 (0xa5), (FreeBSD/NetBSD/386BSD) start 63, size 4095441 (1999 Meg), flag 80 (active)

The data for partition 2 is:

- 122 -

The data for partition 3 is:

The data for partition 4 is:

This part icular syst em has a 9787 MB hard drive t hat has one 1999 MB part it ion cont aining FreeBSD. Whenever you're using any ghost ing ut ilit y, creat e an im age using t he sm allest hard drive size t hat you have available, but which is also large enough t o hold your desired dat a. This will reduce t he size of t he im age and prevent t he problem s associat ed wit h t rying t o rest ore an im age t o a sm aller hard drive.

Once you're sat isfied wit h your syst em , insert t he floppy and reboot . g4u will probe for hardware and configure t he NI C using DHCP. Once it 's finished, you'll be present ed wit h t his screen: Welcome to g4u Harddisk Image Cloning V1.12!

* To upload disk-image to FTP, type:

uploaddisk serverIP [image] [disk]

* To upload partition to FTP, type: [disk+part]

uploadpart serverIP [image]

* To install harddisk from FTP, type:

slurpdisk

serverIP [image] [disk]

* To install partition from FTP, type: [disk+part]

slurppart

serverIP [image]

* To copy disks locally, type:

copydisk disk0 disk1

[disk] defaults to wd0 for first IDE disk, [disk+part] defaults to wd0d for the whole first IDE disk. Use wd1 for second IDE disk, sd0 for first SCSI disk, etc. Default image for slurpdisk is 'rwd0d.gz'. Run 'dmesg' to see boot messages, 'disks' for recognized disks, 'parts ' for list of (BSD-type!) partitions on disk '" (wd0, ...), run any other commands without args to see usage message.

- 123 -

Creat ing t he im age is as sim ple as invoking uploaddisk wit h t he I P address of t he FTP server. I f you wish, include a useful nam e for t he im age; in t his exam ple, I 'll call t he im age securit ylab.gz: # uploaddisk 192.168.2.95 securitylab.gz

( cat $tmpfile ; dd progress=1 if=/dev/rwd0d bs=1m | gzip -9 ) | ftp -n tmpfile: open 192.168.2.95 user install bin put - securitylab.gz bye 5 4 3 2 1 working... Connected to 192.168.2.95. 220 genisis FTP server (Version 6.00LS) ready. 331 Password required for install. Password: type_password_here

230 User install logged in. Remote system type is UNIX. Using binary mode to transfer files. 200 Type set to I. remote: securitylab.gz 227 Entering Passive Mode (192,168,2,95,192,1) 150 Opening BINARY mode data connection for 'securitylab.gz'. ...................

- 124 -

This will t ake a while. How long depends upon t he size of t he drive and t he speed of your net work. When it is finished, you'll see a sum m ary: 9787+1 records in 9787+1 records out 10262568960 bytes transferred in 6033.533 secs (1700921 bytes/sec) 226 Transfer complete. 3936397936 bytes sent in 1:40:29 (637.58 KB/s) 221 Goodbye. #

You can also check out t he size of t he im age on t he FTP server: % du -h ~install/securitylab.gz 3.7G /home/install/securitylab.gz

That 's not t oo bad. I t t ook j ust over an hour and a half t o com press t hat 9 GB drive t o a 3.7 GB im age. The g4u web sit e also has som e hint s for furt her reducing t he size of t he im age or increasing t he speed of t he t ransfer. I f you use im ages on a regular basis, consider upgrading hubs or older swit ches t o 100 MB swit ches. This can speed up your t ransfer rat es significant ly.

I t 's also possible t o creat e an im age of each part icular filesyst em , but I find it easier j ust t o im age a fairly sm all drive. This is because an im age of t he ent ire drive includes t he m ast er boot record ( MBR) or t he desired part it ioning schem e.

2.12.3 Deploying the Image When you wish t o inst all t he im age, use t he floppy t o boot t he syst em t o receive t he im age. Once you receive t he prom pt , specify t he nam e of t he im age and t he I P address of t he FTP server: # slurpdisk 192.168.2.95 securitylab.gz I t doesn't m at t er what was previously on t hat drive. Since t he MBR is recreat ed, t he new drive will j ust cont ain t he im aged dat a. Once t he deploym ent is finished, sim ply reboot t he syst em wit hout t he floppy. I f t he new drive is bigger t han t he im age, you'll have free space left over on t he drive t hat you can part it ion wit h a part it ioning ut ilit y. Rem em ber, don't t ry t o deploy an im age t o a sm aller drive! •

See the Ghost For Unix web site (http://www.feyrer.de/g4u/ - 125 -

Chapter 3. The Boot and Login Environments I nt roduct ion Sect ion 24. Cust om ize t he Default Boot Menu Sect ion 25. Prot ect t he Boot Process Sect ion 26. Run a Headless Syst em Sect ion 27. Log a Headless Server Rem ot ely Sect ion 28. Rem ove t he Term inal Login Banner Sect ion 29. Prot ect ing Passwords Wit h Blowfish Hashes Sect ion 30. Monit or Password Policy Com pliance Sect ion 31. Creat e an Effect ive, Reusable Password Policy Sect ion 32. Aut om at e Mem orable Password Generat ion Sect ion 33. Use One Tim e Passwords Sect ion 34. Rest rict Logins

- 126 -

Introduction When it com es t o configuring syst em s, m any users are reluct ant t o change t he default boot process. Visions of unboot able syst em s, inaccessible dat a, and reinst alls dance in t heir heads. Yes, it is good t o be m indful of such t hings as t hey inst ill t he necessary at t ent ion t o det ail you'll need t o use when m aking changes. However, once you've t aken t he necessary precaut ions, do t ake advant age of t he hacks found in t his chapt er. Many of t hem will increase t he securit y of your syst em . This chapt er also includes several password hacks. You'll learn how t o creat e an effect ive password policy and m onit or com pliance t o t hat policy. You'll find t ools designed t o assist you and your users in m aking good password choices. You'll also learn how t o configure OTP, an excellent choice for when you're on t he road and wish t o access your net work's resources securely.

- 127 -

Hack 24 Customize the Default Boot Menu

Configu r e a spla sh scr e e n . You're not quit e sure what you did t o give t he im pression t hat you don't already have enough t o do. Som ehow, t hough, you were elect ed at t he lat est st aff m eet ing t o creat e a j azzy logo t hat will appear on every user's com put er when t hey boot up in t he m orning. While you m ay not be able t o t ell from first glance, t he FreeBSD boot m enu support s a surprising am ount of cust om izat ion. Let 's st art by exam ining your current m enu t o see which t ools you have t o work wit h.

3.2.1 The Default Boot Menu Your default boot m enu will vary slight ly depending upon your version of FreeBSD and whet her you chose t o inst all t he boot m enu when you inst alled t he syst em . Let 's st art wit h t he m ost vanilla boot prom pt and work our way up from t here. I n t his scenario, you'll see t his m essage as your syst em boot s: Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel] in 10 seconds...

FreeBSD 5.1 int roduced a quasi- graphical boot m enu t hat includes a pict ure of Beast ie and t he following opt ions: Welcome to FreeBSD!

1. Boot FreeBSD [default] 2. Boot FreeBSD with ACPI disabled 3. Boot FreeBSD in Safe Mode 4. Boot FreeBSD in single user mode 5. Boot FreeBSD with verbose logging 6. Escape to loader prompt 7. Reboot

Select option, [Enter] for default or [Space] to pause timer

10

- 128 -

I t is possible t o get t his m enu wit hout doing a full inst all of FreeBSD 5.1. I f you're like m e and use cvsup [ H a ck # 8 0 ] and buildworld t o keep up- t o- dat e, you already have t he necessary files but need t o do a bit of edit ing t o enable t his boot m enu. Even if you already have t he boot m enu, follow along because we're about t o discover som e of t he logic behind t he FreeBSD boot process. This will be excellent preparat ion for learning how t o hack in your own cust om izat ions. Let 's st art by t aking a look at t he direct ory t hat cont ains all of t he boot inform at ion. Not surprisingly, it 's called / boot : # ls /boot -F beastie.4th

cdboot*

kernel.old/

loader.rc

boot

defaults/

loader*

mbr

boot0

device.hints

loader.4th

modules/

boot1

frames.4th

loader.conf

pxeboot

boot2

kernel/

loader.help

screen.4th

support.4th

The act ual file cont aining t he new m enu is beast ie.4t h. I f your sources are out - of- dat e and you don't have t his file, you can download it from ht t p: / / www.freebsd.org/ cgi/ cvsweb.cgi/ src/ sys/ boot / fort h/ . Be sure t o download also t he lat est versions of fram es.4t h and screen.4t h. The / boot direct ory also cont ains t he loader execut able. This applicat ion is responsible for finishing t he boot process. To do so, it depends on t wo configurat ion files, loader.rc and loader.conf. Let 's t ake a peek at loader.rc: # more loader.rc \ Loader.rc \ $FreeBSD: src/sys/boot/forth/loader.rc,v 1.2 1999/11/24 17:59:37 dcs Exp $ \ \ Includes additional commands include /boot/loader.4th

\ Reads and processes loader.rc start

\ Tests for password -- executes autoboot first if a password was defined check-password

\ Unless set otherwise, autoboot is automatic at this point

- 129 -

We're aim ing t o be hackers here, not dest royers of syst em s. A syst em t hat refuses t o boot com plet ely is not a very fun syst em t o work on. So, before m ucking about wit h any of t he files in / boot , m ake sure you have your Em ergency Repair Kit ready ( see [ H a ck # 7 1 ] and [ H a ck # 7 2 ] for m ore inform at ion) . Also, t ake ext ra care in your edit ing and be especially alert for t ypos before saving your changes.

Lines t hat begin wit h a backslash ( \) are com m ent s. Addit ionally, you can add your own com m ent s t o lines cont aining a com m and by preceding your com m ent wit h a # like t his: include /boot/loader.4th

# do NOT remove this line!

start

# do NOT remove this line!

Those are good com m ent s t o add, as you want t o m ake sure you never rem ove t hose t wo lines—t hey are necessary t o t he workings of your boot loader. Before edit ing t his file, m ake a backup copy first : # cp loader.rc loader.rc.orig

Then, t o t ell your syst em t o use beast ie.4t h, carefully add t he following lines t o t he bot t om of / boot / loader.rc. \ Load in the boot menu include /boot/beastie.4th

\ Do the normal initialization and startup initialize drop

\ Start the boot menu beastie-start

Triple- check for t ypos. When you're ready, m ake sure t hat you've saved all of your work and check t hat no one else is connect ed t o t he syst em . I n order t o t est out t he change, you're going t o have t o reboot : # reboot

I f all went well, you now have a Beast ie m enu t o assist you in your boot up select ion. I f your boss had som et hing else in m ind ot her t han t he ult racool Beast ie m enu, let him know t hat have you not yet begun t o cust om ize!

- 130 -

3.2.2 Configuring the Splash Screen Rem em ber t he ot her file I m ent ioned, loader.conf? Well, you should act ually have t wo files wit h t hat nam e. / boot / default s/ loader.conf is t he syst em default , and you should never edit t his file. I nst ead, copy it over t o / boot / loader.conf and m ake your changes t here. That way, not only do you have a chance t o see what is available for cust om izat ion, you also reduce your risk of t ypos. Each line in t his file is com m ent ed and addit ional inform at ion can be gleaned from man loader.conf. Locat e t he Splash screen configuration sect ion so you can configure t hat com pany logo your boss keeps insist ing on. This is what it looks like by default : splash_bmp_load="NO"

# Set this to YES for bmp splash screen!

splash_pcx_load="NO"

# Set this to YES for pcx splash screen!

vesa_load="NO"

# Set this to YES to load the vesa module

bitmap_load="NO"

# Set this to YES if you want splash screen!

bitmap_name="splash.bmp"

# Set this to the name of the bmp or pcx file

bitmap_type="splash_image_data" # and place it on the module_path

Obviously, we'll have t o change t he NO in one of t hose splash lines t o a YES. Which one depends upon your pict ure form at . The t wo t ypes of im ages t hat can be loaded are bmp or pcx. Depending upon t he im age you have t o work wit h, change t he appropriat e NO t o a YES. I f t he im age also happens t o have eight or m ore bit s of color, set vesa_load t o YES. I f you have no idea what t ype or size of pict ure you're dealing wit h, use t he file com m and: # file logo.bmp logo.bmp:

PC bitmap data, Windows 3.x format, 408 x 167 x 8

This part icular logo is a bit m ap t hat is 408 167 pixels at 8 bit s of color. Don't forget t o set t he pat h of your bit m ap file, and m ake sure you rem em ber t o copy t hat bit m ap t o t he specified locat ion: bitmap_name="/boot/logo.bmp"

Leave t his line as is: bitmap_type="splash_image_data"

# and place it on the module_path

Finally, enable bit m ap loading: bitmap_load="YES"

- 131 -

When you're edit ing / boot / loader.conf, keep in m ind t hat you are asking t he loader program t o load various port ions of t he kernel. I f you have changed your kernel configurat ion file [ H a ck # 5 4 ] , double- check t hat you haven't st ripped your kernel of a funct ion you're now asking loader t o load. For exam ple, before reboot ing I should double- check t hat splash funct ionalit y is st ill in m y kernel. Here, m y new kernel configurat ion file is nam ed NEW: # grep splash /usr/src/sys/i386/conf/NEW device

splash

# Splash screen and screen saver support

splash also requires device sc, so ensure t hat is your console t ype: # grep -w sc /usr/src/sys/i386/conf/NEW device

sc

The -w flag t ells grep t o t reat sc as a word rat her t han at t em pt t o m at ch any word cont aining t he let t ers sc. Once you're happy wit h your changes, m ake sure no one is working on t he syst em and t hen reboot. Your bit m ap im age should appear right aft er you m ake your choice at t he Beast ie m enu. I t will rem ain on t he screen unt il you press a key. This behavior has t he advant age of displaying your com pany logo inst ead of t he usual st art up m essages. However, if you ever need t o see t hose m essages, sim ply press a key and your bit m ap will disappear.

3.2.3 The Terminal Screensaver As it is set up now, t he bit m ap will also act as a t erm inal screensaver t hat will kick in aft er five m inut es. To change t he screensaver's t im eout value, add t his line t o / et c/ rc.conf: blanktime="60"

The num ber you choose represent s t he num ber of seconds. I f you decide you don't like t he screensaver funct ionalit y, add t his line t o / et c/ rc.conf: saver="NO"

Those changes t o / et c/ rc.conf won't t ake effect unt il you reboot t he syst em . To enforce t hose set t ings im m ediat ely, at least unt il t he next reboot , use t he vidcontrol com m and: # vidcontrol -t 60

# vidcontrol -t off

Regardless of your t im eout set t ing, you can st ill launch t he screensaver at will—say, when you leave your t erm inal—by pressing t he Shift and Pause keys sim ult aneously. You m ay j ust want t o do t hat before you go grab your boss t o show him t hat j azzy com pany logo.

- 132 -

3.2.4 See Also • • • •

•

man loader man splash / usr/ share/ exam ples/ boot fort h/ ( boot loader exam ples for t he experienced hacker who underst ands Fort h) The Boot sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ boot .ht m l) ht t p: / / www.baldwin.cx/ splash ( splash im ages t o get you st art ed)

- 133 -

Hack 25 Protect the Boot Process

Th w a r t u na u t hor ize d ph ysica l a cce ss t o a syst e m . Creat ing a snazzy boot environm ent for users is one t hing. However, when it com es t o boot ing up servers, your m ind aut om at ically shift s gears t o securit y m ode. Your goal is t o ensure t hat only a very precious few on very rare occasions ever see t he boot process on a server. Aft er all, t he golden rule in securit y land is " physical access equals com plet e access." Here's a prim e exam ple—consider recovering from an unknown or forgot t en root password. Go int o t he server closet , reboot t hat syst em , and press a key t o int errupt t he boot process t o change t he password. A few m om ent s lat er, t he syst em cont inues t o boot as norm al. This can be a real lifesaver if an adm in leaves wit hout divulging t he root password. However, consider t he securit y im plicat ions of an unaut horized user gaining physical access t o t hat server: inst ant root access!

3.3.1 Limiting Unauthorized Reboots Let 's st art by ensuring t hat regular users can't reboot t he syst em eit her inadvert ent ly or m aliciously. By default , if a user presses Ct rl- Alt - Delet e, t he syst em will clean up and reboot . Typically t his isn't an issue for servers, as m ost adm inist rat ion is done rem ot ely and t he server is safely locked away in a server closet . However, it can wreak havoc on workst at ions, especially if t he user is used t o working in a Windows environm ent and has becom e accust om ed t o pressing Ct rl- Alt - Delet e. I t 's also wort hwhile disabling on a server, as it ensures t hat a person has t o first becom e t he superuser in order t o issue t he reboot com m and. I f you're logged int o a rem ot e m achine over SSH and t ry Ct rl- Alt Delet e, it will affect your own m achine, not t he rem ot e m achine. reboot works well over t he net work, t hough.

Disabling t his feat ure requires a kernel rebuild. ( See [ H a ck # 5 4 ] for det ailed inst ruct ions.) Add one of t hese lines t o your kernel configurat ion file, t hen rebuild and reinst all t he kernel: options SC_DISABLE_REBOOT

# if using syscons console driver

# or

options PCVT_CTRL_ALT_DEL

# if using pcvt console driver

You're probably t hinking, " I f I want ed t o reboot a syst em and didn't know t he superuser password, I 'd sim ply hit t he power but t on." Yup! That kernel opt ion cert ainly won't prevent

- 134 -

t hat , but a carefully t hought out CMOS[ 1] configurat ion will decide if and how t hat syst em will reboot . [ 1]

CMOS is bat t ery- powered m em ory t hat holds syst em set t ings such as t he t im e, dat e, and syst em configurat ion.

At a m inim um , t he CMOS configurat ion should allow only one boot device. This is t o prevent an int ruder from t rying t o boot an alt ernat e kernel from a floppy, CD- ROM drive, or ot her support ed boot device. Addit ionally, you should set a password for CMOS and record it in a safe place. This will prevent an int ruder from sim ply changing t he CMOS configurat ion. Keep in m ind t hat t his is not fail- proof; you are m erely adding layers of inconvenience. A det erm ined int ruder can sim ply pop open t he case and drain t he CMOS bat t ery, but t hat t akes t im e and addit ional effort .

3.3.2 Password Protecting Single-User Mode All t he m agic happens when you int errupt t he boot process. This is where you can change t he superuser password wit hout having t o first know t he superuser password. This is where you can unload t he current ly loaded kernel and replace it wit h anot her. This is where you can change any configurat ion file or binary wit hout worrying about securelevels or syst em flags [ H a ck # 5 6 ] . This is t he reason why you lock up your servers, m onit or access t o t he server room , and run t hem headless [ H a ck # 2 6 ] . Fort unat ely, int errupt ing t he boot process requires keyboard input , m eaning t he user needs physical access t o t he syst em . What happens when a m alicious user does bypass your physical securit y m easures, gaining physical access t o t he syst em ? All she has t o do is int errupt t hat boot process, and t he syst em is hers t o do as she wishes. On a syst em wit hout t he graphical boot m enu [ H a ck # 2 4 ] , pressing any key at t he t im er will pause t he boot process. I f t he syst em has t he graphical boot m enu, pressing 6 t o Escape to loader prompt will show t he sam e t im er. The t im er opt ion looks like t his: Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel] in 10 seconds...

I f you press any key ot her t han Ent er, you'll receive t his: Type '?' for a list of commands, 'help' for more detailed help. OK boot -s

Type boot -s t o ent er single- user m ode. The kernel will appear t o load norm ally, but , inst ead of processing t he rc script s, t his prom pt will appear: Enter full pathname of shell or RETURN for /bin/sh: #

Once you've finished m aking your desired changes, sim ply t ype exit. The syst em will cont inue t o boot int o m ult iuser m ode. Now, how do you prevent a user from doing t hat ? Password prot ect single- user m ode by edit ing / et c/ t t ys. Find t his line:

- 135 -

# If console is marked "insecure", then init will ask for the root password # when going to single-user mode. console none

unknown off secure

Follow t he com m ent s and change t he word secure t o insecure. While t hat m ay seem nonint uit ive, you're saying t he syst em is considered t o be insecure, t hus you want a password. The next t im e a user at t em pt s single- user m ode, t he kernel will load, but t he user will receive t his prom pt inst ead: Enter root password, or ^D to go multi-user Password: You m ust not forget t he root password if you password prot ect singleuser m ode!

3.3.3 Password Protecting loader Let 's ret urn t o t he t im er sect ion of t he boot process. A user can t ype m ore t han boot -s aft er int errupt ing t he boot process. I n fact , if you press ? at t hat OK prom pt , you'll see t hat you can unload t he current kernel, load anot her kernel, load and unload kernel m odules, and view and change variables. You can m uck about wit h j ust about every part of t he boot process t hat would norm ally be cont rolled by t he loader com m and. Fort unat ely, you can also require a user t o input a password before receiving t hat OK prom pt . Set t he password by adding t his line t o / boot / loader.conf: password=12345 Of course, your password should be harder t o guess t han 12345. Now t he boot process will prom pt t he user for a password. Wit hout t hat password, you cannot ent er single- user m ode or load or unload kernel m odules. You can still boot ; you j ust cannot int errupt t he boot process. Also, if your CMOS support s it , you can require a password t o boot t he m achine. However, t his is oft en considered t o be a bad t hing, especially on a co- locat ed web or m ail server. The password in / boot / loader.conf is in clear t ext . Alt hough you can't encrypt t his password, you can t ight en up it s perm issions so only t he superuser can read it : # chmod 600 /boot/loader.conf

3.3.4 See Also man boot, man loader, The Boot Process sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ boot blocks.ht m l) ,Reset t ing t he Root Password in t he FreeBSD FAQ ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ faq/ adm in.ht m l# FORGOT- ROOTPW)

- 136 -

Hack 26 Run a Headless System

For t h ose t im e s w he n you w a n t t o r u n a syst e m " h e a dle ss." Som et im es it is a sim ple m at t er of econom y. Perhaps you've m anaged t o scrounge up anot her syst em , but you don't have enough m onit ors, keyboards, or m ice t o go around. You also don't have t he budget t o purchase eit her t hose or a KVM swit ch. Som et im es it is a m at t er of securit y. Perhaps you're int roducing a PC t o a server closet and your physical securit y policy prevent s server closet devices from being at t ached t o m onit ors, keyboards, and m ice. Before you can run a syst em " headless," you need t o have an alt ernat ive for accessing t hat syst em . Once you've rem oved input and out put peripherals, your ent ry point int o t he syst em is now eit her t hrough t he net work card or a serial port . Going in t hrough t he net work card is t he easiest and is quit e secure if you're using SSH. However, you should also consider a plan B. What if for som e reason t he syst em becom es inaccessible over t he net work? How do you get int o t he syst em t hen? Do you really want t o gat her up a spare m onit or, keyboard, and m ouse and carry t hem int o t he server closet ? A m ore at t ract ive plan B m ay be t o purchase a null m odem cable as insurance. This is a crossed serial cable t hat is designed t o go from one com put er's serial port t o anot her com put er's serial port . This t ype of cable allows you t o access a syst em wit hout going t hrough t he net work, which is a real lifesaver when t he syst em isn't responding t o t he net work. You can purchase t his t ype of cable at any st ore t hat sells net working cables. Your last considerat ion is whet her t he syst em BI OS will cooperat e wit h your plan. Most newer BI OSes will. Many have a CMOS opt ion t hat can be configured t o disable " halt on errors." I t 's always a good idea t o check out your available CMOS opt ions before you st art unplugging your peripherals.

3.4.1 Preparing the System I 've j ust inst alled a new FreeBSD 5.1 syst em . Since I didn't have a null m odem cable handy, I inst alled t he old- fashioned way wit h t he m onit or and keyboard at t ached. I f you do have a null m odem cable and want t o experim ent wit h a headless inst all, follow t he direct ions in t he Handbook sect ion referenced at t he end of t his hack. Since I want t o access t he server over t he net work, I 'll double- check t hat t he NI C is properly configured and t hat sshd is running: % ifconfig ed0 ed0: flags=8843 mtu 1500 inet 192.168.2.94 netmask 0xffffff00 broadcast 192.168.2.255 ether 00:80:ad:79:4e:fd

% sockstat | grep sshd root

sshd

389

4

tcp4

*:22

- 137 -

*:*

The ifconfig com m and is used t o verify an int erface's configurat ion; in t his exam ple, t he int erface is ed0. The flags indicat e t hat t his int erface is UP and RUNNING. The int erface also has an I P address of 192.168.2.94. The sockstat com m and is sim ilar t o t he netstat com m and, but I find it provides a m ore int uit ive out put . For each open port it will display t he owner of t he service ( root) , t he nam e of t he service ( sshd) , t he PI D ( 389) , t he socket file descript or ( 4) , t he t ransport ( tcp4), t he local address ( *:22) , and t he foreign address ( *.*) . The PI D is useful if you need t o send a signal t o t he process. The local address indicat es which int erfaces on t his syst em ( in t his case, all, or *) are list ening on which port num ber ( 22) . There aren't any current sessions, as t he foreign address sect ion is *.*. I f t here were a current session, it would show t he address of t he ot her syst em followed by t he socket num ber being used for t he connect ion. I f for som e reason sshd isn't running on your syst em , add t he following line t o / et c/ rc.conf: sshd_enable="YES"

and double- check t hat it 'll be available at boot up, like so: # /etc/rc.d/sshd rcvar #sshd $sshd_enable=YES

Finally, t yping sshd as t he superuser should st art t he daem on. You can prove t his by checking t hat it 's list ening wit h sockstat | grep sshd. One last t est —I 'll m ake sure I can log int o t he syst em over t he net work: % ssh 192.168.2.94 Password: %

Now t hat I knew t he syst em was accessible over t he net work, it was t im e for t he m om ent of t rut h. Aft er halting t he syst em , I ent ered it s CMOS configurat ion. I was a lit t le bit worried because t here weren't any opt ions dealing wit h " halt errors." Undaunt ed, I left CMOS and powered off and unplugged t he m onit or, keyboard, and m ouse. I t hen opened t he case and physically rem oved t he video card. When I powered up, t he syst em responded wit h a longer t han ordinary beep. But aft er a few seconds, m y hard drive light flashed and I could hear t he operat ing syst em probing m y devices and loading t he drivers. Aft er a m om ent or so, I t ried t o ssh int o t he syst em and was greet ed wit h m y password prom pt ! Assum ing your BI OS is willing t o cooperat e, FreeBSD has no problem loading headless.

- 138 -

3.4.2 If the Headless System Becomes Inaccessible Should your syst em ever st op responding over t he net work, you'll be glad you purchased t hat null m odem serial cable. Connect one end t o t he COM port of t he headless syst em , and t he ot her end t o t he COM port of anot her syst em t hat you can access eit her direct ly or over t he net work. Program s I f t hat ot her syst em is running a Windows operat ing syst em , go t o St art Accessories Com m unicat ions HyperTerm inal ( or open hypert rm .exe) . You'll need t o creat e a new connect ion, so choose a nam e and icon for it . Under Connect using: , choose t he COM port t o which t he serial cable is at t ached. You'll also have t o configure t he port propert ies for t hat COM port . Change t he default 2400 bit s per second t o 9600. Finally, change hardware flow cont rol t o none. Press Ent er, and you should be connect ed t o t he headless syst em . I f you're not , double- check t hat you chose t he correct COM port . I f you're at t aching from a syst em running any variant of Unix, you can use eit her t he cu or tip com m ands t o connect via t he serial cable. To use cu, sim ply specify your COM port using t he line swit ch -l and a speed of 9600 baud using t he speed swit ch -s. For exam ple, t his synt ax allows you t o connect t o COM2 or cuaa1: # cu -l /dev/cuaa1 -s 9600 Connected.

You should now be able t o see what is happening on your headless syst em . One of t he advant ages of connect ing t hrough a serial cable is t hat you can wat ch t he boot process of t he syst em . You can't do t his over a net work connect ion, because init ializing t he net work occurs t oward t he end of a successful boot . Before t he net work can be init ialized, t he kernel m ust successfully load int o m em ory and t he necessary hardware m ust be probed. I f you're having problem s boot ing a syst em , it is usually due t o a m issing or corrupt kernel or a hardware problem . To disconnect from t he cu session, t ype ~., t hen press t he Ent er key. You should receive a Disconnected. m essage and receive t he prom pt of t he syst em you st art ed from . The tip ut ilit y doesn't use line or speed swit ches. I t inst ead expect s you t o use one of t he finger friendly short cut s found at t he end of t he / et c/ rem ot e file. Let 's t ake a look at t hat sect ion: # tail /etc/remote # Hardwired line cuaa0b|cua0b:dv=/dev/cuaa0:br#2400:pa=none: cuaa0c|cua0c:dv=/dev/cuaa0:br#9600:pa=none:

# Finger friendly shortcuts

- 139 -

com1:dv=/dev/cuaa0:br#9600:pa=none: com2:dv=/dev/cuaa1:br#9600:pa=none: com3:dv=/dev/cuaa2:br#9600:pa=none: com4:dv=/dev/cuaa3:br#9600:pa=none:

Not ice t hat t here is an ent ry for each COM port . This m eans t hat t o connect t o COM2, you sim ply have t o t ype: # tip com2 connected

You need a lit t le bit m ore coordinat ion t o disconnect , t hough. Hold down Shift while you press t he ~ key. Keep your finger on Shift as you press t he Ct rl key, t hen t he let t er D: # ~^D [EOT]

3.4.3 See Also • • •

man tip man cu The Advanced I nst allat ion Guide in t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ inst alladvanced.ht m l)

- 140 -

Hack 27 Log a Headless Server Remotely

M or e on he a dle ss syst e m s, bu t t h is t im e fr om t h e N e t BSD pe r spe ct ive . We've already seen in [ H a ck # 2 6 ] t hat it 's im port ant t o have an alt ernat ive m et hod for connect ing t o a headless server. I t 's also im port ant t o be able t o receive a headless syst em 's console m essages. This hack will show how t o configure bot h on a Net BSD syst em .

3.5.1 Enabling a Serial Console I f you have anot her m achine close t o your headless server, it m ay be convenient t o enable t he serial console so t hat you can connect t o it using a serial com m unicat ion program . tip, included in t he base syst em , and minicom , available t hrough t he packages collect ion, allow you t o handle t he server as if you were working on a real physical console. To enable t he serial console under Net BSD, sim ply t ell t he boot blocks t o use t he serial port as t he console; t hey will configure t he kernel on t he fly t o use it inst ead of t he physical screen. You also need kernel support for t he serial port device, which is included in t he default GENERIC kernel. However, changing t he boot blocks configurat ion is a bit t ricky because you need writ e perm issions t o t he raw root device. As we are t alking about a server, I assum e t he securelevel funct ionalit y is enabled; you m ust t em porarily disable it by adding t he options INSECURE line t o your kernel. While in t he kernel configurat ion file, double- check t hat it includes serial port support . Then, recom pile your kernel. Once you have access t o t he raw part it ion, updat e t he boot blocks using t he installboot ut ilit y. The process depends on t he Net BSD version you are using. I f you are running 2.0 or higher, use t he com m and shown next . Replace t he boot xx_ffsv1 file wit h t he one t hat m at ches your root filesyst em t ype; failure t o do so will render your syst em unboot able. # /usr/sbin/installboot -o console=com0 /dev/rwd0a /usr/mdec/bootxx_ffsv1

I f you are running 1.6, use t he following com m and inst ead: # /usr/mdec/installboot /usr/mdec/biosboot_com0.sym /dev/rwd0a

When done, rebuild your kernel wit hout t he options INSECURE line t o reenable securelevel. You can also rem ove t he console drivers wscons and pccons t o reduce t he kernel size, t hough you m ust keep t he serial port driver. As an alt ernat ive t o building an insecure kernel, you can boot from a floppy disk t o get direct access t o t he part it ion and updat e t he boot blocks as described earlier. The floppies you used t o inst all t he syst em are fine.

- 141 -

3.5.2 Setting Up the Logging Server Even if you have configured a serial console, you won't always be connect ed t o it . Therefore, it is very convenient t o redirect im port ant console m essages t o anot her m achine t hat has a physical screen connect ed t o it . syslogd let s you do t his. St art by allowing incom ing syslogd connect ions on t he m achine t hat will be receiving log m essages. ( I call m ine logger.local.) To do t his, add t he following lines t o / et c/ rc.conf: syslogd=YES syslogd_flags=

The first opt ion is not really needed, as syslogd is enabled by default . The second opt ion overrides t he secure ( s) flag t hat ot herwise would be passed t o t he daem on t hrough / et c/ default s/ rc.conf. This flag t ells syslogd not t o list en on a UDP socket , and in t his scenario we want t o receive log m essages over t he net work. Then, rest art t he daem on: # /etc/rc.d/syslogd restart

logger.local can now receive incom ing syslogd connect ions from any host . I f required, you can rest rict t his by using t he built - in firewall, ipf.

3.5.3 Setting Up the Headless System You are ready t o configure your headless server t o send m essages t o t he logger m achine. As an exam ple, we are going t o redirect all m essages t hat are act ually sent t o t he serial console t o logger.local. Open / et c/ syslog.conf in your favorit e edit or. You will not ice t hat t he first uncom m ent ed line direct s m essages t o / dev/ console. Append t he @logger.local st ring t o it , separat ed by a com m a. Aft er t he changes, you should end up wit h som et hing like: *.err;kern.*;auth.notice;authpriv.none;mail.crit

/dev/console,@logger.local

Repeat for any ot her cat egories you want t o redirect . When done, rest art syslogd as shown earlier.

- 142 -

3.5.4 Shutting Down the Server Using wsmoused The next t wo sect ions of t his hack require Net BSD 2.0 and above.

I f you are running a headless syst em at hom e, you m ay want t o shut it down at night . You could do t his by sshing int o t he server and execut ing shutdown m anually, but t his requires a second syst em . However, since you have physical access t o t he headless syst em , you can sim ply use wsmoused, which will let you execut e t wo or t hree com m ands from a m ouse—one for each m ouse but t on. wsmoused's " act ion m ode" let s you assign com m ands t o m ouse but t ons. Here's a sam ple configurat ion file t o shut down and reboot t he m achine, which you can copy t o / et c/ wsm oused.conf: device = /dev/wsmoused; modes = action;

mode action { button_0_down = "shutdown -p now"; button_2_down = "shutdown -r now"; }

Here I 've m apped t he left m ouse but t on, 0, t o t he com m and t hat will halt t he syst em and t he right m ouse but t on, 2, t o t he com m and t hat will reboot t he syst em . ( The m iddle m ouse but t on is 1.) Since I don't plan on using t his m ouse for it s usual input funct ions, such as copy and past e, t his is a really convenient way t o power off t he syst em quickly and safely. Enable t he st art up of wsmoused at boot t im e: # echo "wsmoused=YES" >> /etc/rc.conf I f you have a dial- up connect ion, you could use a sim ilar configurat ion t o connect and disconnect t he link.

3.5.5 Beep on Halt Som e headless servers don't support APM or ACP, so t he kernel can't power t hem down aut om at ically. The i386 archit ect ure has anot her opt ion: beep on halt . I t beeps t he speaker m ult iple t im es when it is safe t o power off t he m achine aft er a successful halt. To enable t his feat ure, add t he following line t o your kernel configurat ion file and rebuild it : options BEEP_ONHALT

- 143 -

I n case you do not like t he default t one, you have several ot her opt ions. Here t hey're shown wit h t heir default values: options BEEP_ONHALT_COUNT=3

# Times to beep

options BEEP_ONHALT_PITCH=1500 # Default frequency (in Hz) options BEEP_ONHALT_PERIOD=250 # Default duration (in msecs)

3.5.6 See Also • • • •

man man man man

8 installboot syslogd wsmoused shutdown

- 144 -

Hack 28 Remove the Terminal Login Banner

Give u se r s t h e infor m a t ion you w a n t t he m t o r e ce ive w h e n t h e y log in . The default login process on a FreeBSD syst em produces a fair bit of inform at ion. The t erm inal m essage before t he login prom pt clearly indicat es t hat t he m achine is a FreeBSD syst em . Aft er logging in, a user will receive a copyright m essage and a Message of t he Day ( or motd) , bot h of which cont ain m any references t o FreeBSD. This m ay or m ay not be a good t hing, depending upon t he securit y requirem ent s of your net work. Your organizat ion m ay also require you t o provide legal inform at ion regarding net work access or perhaps a banner t out ing t he benefit s of your corporat ion. Fort unat ely, a few sim ple hacks are all t hat st and bet ween t he default s and your net work's part icular requirem ent s.

3.6.1 Changing the Copyright Display Let 's st art wit h t he copyright inform at ion. That 's t his part of t he default login process: Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved.

To prevent users from seeing t his inform at ion, sim ply: # touch /etc/COPYRIGHT

3.6.2 Changing the Message of the Day Technically, you could add your own inform at ion t o / et c/ COPYRI GHT inst ead of leaving it as an em pt y file. However, it is com m on pract ice t o put your inform at ion in / et c/ m ot d inst ead. The default / et c/ m ot d cont ains very useful inform at ion t o t he new user, but it does get rat her old aft er a few hundred logins. You can edit / et c/ m ot d t o say what ever suit s your purposes—anyt hing from your favorit e sci- fi excerpt t o all t he nast y t hings t hat will happen t o som eone if t hey cont inue t o t ry t o log int o your syst em . Here's a very sim ple exam ple: # more /etc/motd ********************************************************* *****

Authorized users only!!

*****

*********************************************************

- 145 -

You'll not e t hat aft er you cust om ize your motd, users will st ill see t his t ext prepended t o it : FreeBSD 5.1-RELEASE (GENERIC) #0: Thu Jun 5 02:55:42 GMT 2003

I f you don't want t o advert ise your operat ing syst em version and kernel inform at ion, you'll need one m ore hack. Add t his line t o / et c/ rc.conf: update_motd="NO"

I f you're using FreeBSD 5.x, you no longer have t o reboot or go int o single- user m ode t o init ialize a change t o / et c/ rc.conf. I nst ead, you can use one of t he m any script s available in / et c/ rc.d. Let 's see if t here's a script t hat deals wit h motd: # ls -F /etc/rc.d | grep motd motd*

Excellent . Let 's see what synt ax t hat com m and expect s: # /etc/rc.d/motd Usage: /etc/rc.d/motd [fast|force](start|stop|restart|rcvar)

Param et ers in square bracket s are opt ional, whereas param et ers in parent heses are m andat ory. Not ice each opt ion is separat ed by t he or sym bol ( |) , m eaning you j ust pick one out of t he list . I n our case, we want t o use t he rcvar param et er. This will t ell t he motd script t o reread it s set t ing in / et c/ rc.conf: # /etc/rc.d/motd rcvar # motd $update_motd=NO OpenBSD users, read man motd and / et c/ rc ( search for motd) t o underst and how t he syst em const ruct s t he banner. Ot herwise, it 'll updat e when you least expect it !

3.6.3 Changing the Login Prompt Finally, let 's change t he t ext t hat first appears at t he login prom pt . This requires an edit t o / et c/ get t yt ab. This is a fairly im port ant file as it cont rols access t o your t erm inals, which is how users access t he syst em . Before edit ing t his file, always m ake a backup copy first : # cp /etc/gettytab /etc/gettytab.orig

Next , open up / et c/ get t yt ab in your favorit e t ext edit or and look for t his line: default:\

- 146 -

:cb:ce:ck:lc:fd#1000:im=\r\n %s/%m (%h) (%t) \r\n\r\n:sp#1200:\

See t he part in bold? That 's t he part you can replace wit h what you'd like t he world t o see when t hey receive t heir login prom pt . Right now, t hey see t his: FreeBSD/i386 (host.domain.com) (ttyv1)

That 's because t hat default st ring cont ains t he variables in Table 3- 1.

Ta ble 3 - 1 . Login pr om pt va r ia ble s Va r ia ble

M e a n in g

%s

Operat ing syst em

%m

Archit ect ure

%h

Host nam e

%t

t t y nam e

You can very carefully change t hose charact ers t o som et hing else. For exam ple, m ine looks like t his: :cb:ce:ck:lc:fd#1000:im=\r\n I'm a node in Cyberspace. Who are you? \ \r\n\r\n:sp#1200:\

Again, I 've put m y changes in bold for em phasis. Carefully double- check t hat you didn't lose any carriage ret urn ( \r) or newline ( \n) charact ers along t he way, t hen save your change.

3.6.4 Testing Your Changes I t 's im port ant t o t est your change im m ediat ely at a different t erm inal t o ensure you can st ill log int o your syst em . This way, if you did m ake a t ypo t hat prevent s logins, you can ret urn t o your previous t erm inal and fix it . I 'll press Alt - F4 t o go t o a t erm inal wit h a login prom pt . I 'll probably st ill see t he old t erm inal m essage, so I 'll log in, log out , t hen log in again: login: Password: % exit logout I'm a node in cyberspace. Who are you?

login:

- 147 -

3.6.5 See Also • • •

man motd man gettytab The / et c/ rc.d sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ configt uningrcng.ht m l)

- 148 -

Hack 29 Protecting Passwords With Blowfish Hashes

Ta k e t h e se sim ple st e ps t o t h w a r t pa ssw or d cr a ck e r s. All good adm inist rat ors know t hat passwords can be a weak link in t he securit y chain. A m alicious and det erm ined user arm ed wit h a password cracker could conceivably guess enough of your net work's passwords t o access unaut horized resources.

3.7.1 Protecting System Passwords in General Fort unat ely, you can m ake a password cracker's life very difficult in several ways. First , educat e your users t o choose com plex, hard- t o- guess passwords t hat are m eaningful enough for t hem t o rem em ber. This will t hwart dict ionary password crackers [ H a ck # 3 0 ] , which use list s of dict ionary and easy- t o- guess words. Second, be aware of who has superuser privileges and who has t he right t o backup / et c. This direct ory cont ains t he t wo password dat abases t hat are required t o run a brut e- force password cracker. As t he nam e im plies, t his t ype of cracker will event ually guess every password in your password dat abases as it syst em at ically t ries every possible keyboard com binat ion. Your best prot ect ion from t his t ype of cracker is t o prevent access t o t hose password dat abases. This includes locking up your backup t apes and m onit oring t heir access. I t is also a good idea t o increase t he am ount of t im e it would t ake a brut e- force cracker t o crack a password dat abase. FreeBSD, like m ost Unix syst em s, adds a m agic bit of random ness—known as a salt —t o t he password when it is st ored in t he password dat abase. The upshot is t hat a password cracker m ay have t o t ry up t o 4,096 different com binat ions for each and every password it t ries t o guess. Using a st rong algorit hm t o prot ect your passwords can also slow down a brut e- force cracker. FreeBSD support s a hard- t o- crack algorit hm known as Blowfish. One of t he first t hings I do aft er a FreeBSD inst all is t o configure t he password dat abase t o use Blowfish. While it is easier t o do t his before you creat e your users, it is st ill wort h your while t o im plem ent it aft er you've creat ed your user account s.

3.7.2 Protecting System Passwords with Blowfish To use Blowfish, st art by opening up / et c/ login.conf in your favorit e edit or. Look for t his line: :passwd_format=md5:\

Carefully edit it so it looks like t his: :passwd_format=blf:\

Check for t ypos before saving your change.

- 149 -

You m ay have not iced t his com m ent when you m odified / et c/ login.conf: # Remember to rebuild the database after each change to this file: # #

cap_mkdb /etc/login.conf

#

Let 's t ake a closer look at what we're being asked t o do. According t o t hat com m ent , login.conf is m ore t han a configurat ion file, it is a dat abase. Not only t hat , it is a capabilit y dat abase, a dat abase t hat support s different capabilit ies. That is t he reason behind t he weird synt ax wit hin login.conf. Whenever you edit a capabilit y dat abase, you have t o use t he cap_mkdb com m and t o int egrat e your changes wit hin t he dat abase. So, follow t he direct ions: # cap_mkdb /etc/login.conf

3 .7 .2 .1 Con ve r t in g e x ist ing pa ssw or ds I f you have any exist ing users, you need t o convert t heir passwords from MD5 t o Blowfish. This is why it 's a good idea t o m ake t he change before you creat e your users. I f you've already creat ed users, it 's back t o t he password dat abase t o find all of t he act ive account s. I nact ive account s—account s t hat don't allow logins—have t he * charact er inst ead of an encrypt ed password. Since we want t o find all of t he lines in t he password dat abase t hat do not cont ain an ast erisk, we need an invert ed grep: # grep -v '*' /etc/master.passwd root:$1$ywXbyPT/$GC8tXN91c.lsKRpLZori61:0:0::0:0:Charlie &:/root:/bin/csh dru:$1$GFm1nh6I$jh3v4I.QNf450ARgltZU5.:1008:0::0:0:User &:/home/dru:/bin/csh

Well, t hat worked, but we could m ake t he out put look m uch pret t ier: # grep -v '*' /etc/master.passwd | cut -d ':' -f 1 root dru

Let 's pick apart t hat com m and synt ax. grep -v creat es a reverse filt er. I n effect , it says, " Show m e t he lines in / et c/ m ast er.passwd t hat do not cont ain an *." Since t hose lines are long and cont ain m uch m ore t han j ust t he usernam e, I piped t he out put t o t he cut ut ilit y t o lit erally cut out t he port ions I don't need t o see. Not ice t hat t he usernam es are t he very first t hing in each line, and t hey are always followed by t he : field separat or. -d t ells cut t o consider t he colon charact er, not t he t ab charact er, as t he separat or. -f 1 t ells cut t hat I 'm int erest ed in t he very first field of t hat line.

- 150 -

I t looks like m y part icular syst em has t wo act ive account s: root and dru. Not ice in t he original out put t he long sequence of charact ers t hat st art s wit h $1 and ends wit h : . No, m y users' passwords aren't quit e t hat com plex. Rat her, you're seeing t he password aft er it 's been encrypt ed by t he MD5 algorit hm . That $1 m eans MD5. I t 'll be $2 aft er we swit ch t o Blowfish encrypt ion. ( Be aware t hat you can't edit t he file direct ly; t he ent ire password m ust be changed.) I 'll now change t hose t wo passwords: # passwd dru Changing local password for dru New Password: Retype New Password:

# passwd Changing local password for root New Password: Retype New Password:

Not e t hat t he superuser can change any user's password by specifying t he appropriat e usernam e. I f you don't specify a nam e, you will inst ead change t he root password. When you're finished, repeat t he original grep -v com m and and double- check t hat all of t he encrypt ed passwords now st art wit h $2. Don't forget t o t ell your users t hat you have changed t heir passwords! Also caut ion t hem t o use passwd t o reset t heir password t o a value known only t o t hem selves.

3 .7 .2 .2 For cin g n e w pa ssw or ds t o u se Blow fish Finally, configure t he adduser ut ilit y t o use Blowfish whenever you creat e a new user by edit ing / et c/ aut h.conf. Look for t his line: # crypt_default = md5 des

and carefully change it t o: crypt_default = blf Once you've saved your change, t est it by creat ing a new user. The easiest way t o do t his is t o t ype adduser and follow t he prom pt s.

3.7.3 See Also man passwd, man adduser, Blowfish inform at ion by Bruce Schneier, t he creat or of t he algorit hm , at ht t p: / / www.schneier.com / blowfish.ht m l

- 151 -

Hack 30 Monitor Password Policy Compliance

W h e n t o use a pa ssw or d cr a ck e r u t ilit y. Now t hat you've t ight ened up your password policy t o t hwart password crackers, it 's t im e t o learn how t o use a password cracker t o m onit or t he effect iveness of t hat password policy. You're probably t hinking, " Hey, wait a m inut e! I sn't t hat som e sort of oxym oron? An adm inist rat or cracking passwords?" Well, it depends upon t he t ype of password cracker you plan on using. A brut e- force password cracker such as John the ripper or slurpie will syst em at ically t ry every possible keyboard com binat ion unt il it has cracked every password in t he password dat abase. Does an adm inist rat or need t o know every password in his net work? Definit ely not . However, an adm inist rat or does need t o know if her users are choosing easy- t o- guess passwords, especially if she's responsible for enforcing com pliance t o t he net work's password policy. A properly t weaked dict ionary password cracker such as crack is an effect ive way t o m onit or t hat com pliance. I t is im port ant t hat a net work's securit y policy indicat es in writ ing who runs t he dict ionary cracker, when it is run, and how t he result s are handled. For exam ple, if t he password policy forces users t o change t heir passwords every 30 days, t he following day is an excellent t im e for t he delegat ed adm inist rat or t o run t he cracker. I deally, t he cracker will ret urn no result s. This m eans all users chose a st rong password. Should t he cracker find som e weak passwords, t he securit y policy should clearly out line t he procedure used t o ensure t hat noncom pliant users change t heir passwords t o ones t hat are harder t o guess.

3.8.1 Installing and Using crack Let 's t ake a look at t he m ost com m only used dict ionary password cracker used on Unix syst em s, crack. You'll have t o be t he superuser for t his ent ire hack because, fort unat ely, only t he superuser has perm ission t o crack t he passwd dat abase. crack should build on any Unix syst em ; I 'll dem onst rat e on FreeBSD: # cd /usr/ports/security/crack # make install clean

On m y syst em , t his creat es t he / usr/ local/ crack direct ory which only t he superuser can access. I need t o cd int o t hat direct ory in order t o crack passwords. I 'll st art wit h a sim ple crack, t hen show you how t o t weak t his ut ilit y t o serve your part icular net work. # cd /usr/local/crack # ./Crack -fmt bsd /etc/master.passwd

- 152 -

Crack is a Bourne shell script cont ained wit hin t his direct ory, so you'll have t o run it wit h t he com m and ./Crack. Use t he -fmt swit ch t o indicat e t he t ype of syst em ; in m y case, it is bsd. Finally, pass t he pat h of t he dat abase cont aining t he act ual password hashes. On m y syst em , t his is t he BSD shadow password dat abase at / et c/ m ast er.passwd. The com m and and out put on m y t est syst em is: # ./Crack -fmt bsd /etc/master.passwd Crack 5.0a: The Password Cracker. (c) Alec Muffett, 1991, 1992, 1993, 1994, 1995, 1996 System: FreeBSD genisis 5.1-RELEASE FreeBSD 5.1-RELEASE #7: \ Tue Jul 29 09:54:11 EDT 2003 dru@genisis:/usr/obj/usr/src/sys/NEW i386 Home: /usr/local/crack Invoked: ./Crack -fmt bsd /etc/master.passwd Stamp: freebsd-5-i386_

Crack: making utilities in run/bin/freebsd-5-i386_ find . -name "*~" -print | xargs -n50 rm -f ( cd src; for dir in * ; do ( cd $dir ; make clean ) ; done ) rm -f dawglib.o debug.o rules.o stringlib.o *~ /bin/rm -f *.o tags core rpw destest des speed libdes.a .nfs* *.old \ *.bak destest rpw des speed rm -f *.o *~ `../../run/bin/freebsd-5-i386_/libc5.a' is up to date. all made in util Crack: The dictionaries seem up to date... Crack: Sorting out and merging feedback, please be patient... Crack: Merging password files... Crack: Creating gecos-derived dictionaries mkgecosd: making non-permuted words dictionary mkgecosd: making permuted words dictionary Crack: launching: cracker -kill run/Kgenisis.27478 Done

Not e t hat t he word Done is a bit of a m isnom er. The gecos t est is finished, but t he act ual dict ionary at t ack has j ust begun and is quiet ly perking along in t he background:

- 153 -

# ps -acux | grep cracker root

14013 97.0

2.8

9448 8916

v5

R

10:32AM

4:17.68 cracker

3 .8 .1 .1 M on it or ing t he r e su lt s Let 's t ake a look at m y current result s, t hen analyze what is happening here: # ./Reporter -quiet ---- passwords cracked as of Mon Nov 17 10:33:18 EST 2003 ----

1069099872:Guessed test [test]

User & [/etc/master.passwd /bin/csh]

---- done ----

The Reporter script , which is also found in t he / usr/ local/ crack/ direct ory, sends t he current result s of t he dict ionary crack t o st andard out put . I ran Reporter short ly aft er Crack had ret urned m y prom pt . Not ice t hat it found t hat t he password for t he test account was test. The reason why it found t his password so quickly is because of t he gecos field in / et c/ m ast er.passwd. I f you're fam iliar wit h man master.passwd, you know t hat t he gecos field cont ains t he user's full nam e, possibly followed by her ext ension, office phone num ber, and hom e phone num ber. This m eans t hat if a user uses any of t hose values for a password, her password can be cracked wit hin a second or t wo. The act ual dict ionary at t ack will t ake a while t o run. How long will depend upon t he speed of your CPU. However, you should expect crack t o run for a good port ion of a business day. Why so long? I f you've ever had t he opport unit y t o run a dict ionary cracker on a non- Unix syst em , you m ay have had your result s back in well under an hour. The answer is t hat BSD password hashes are prot ect ed by a salt . I n sim ple t erm s, t he salt adds random charact ers t o a user's password before t he encrypt ion algorit hm creat es t he hash. Those are encrypt ed hashes, not t he act ual passwords, st ored in / et c/ m ast er.passwd. I n order for t he password cracker t o bypass t he salt , it has t o t ry m any variat ions of t he sam e word before it can det erm ine if t hat word is indeed t he user's password. You m ay want t o writ e a script t hat will t ell you when Crack is finished. Here is a sim ple exam ple: #!/bin/sh #script to see if Crack is still running #and to display current report

while ps -acux | grep -l "cracker" > /dev/null do sleep 600 echo "Still running. Here's the latest report:"

- 154 -

cd /usr/local/crack && ./Reporter -quiet done

echo "Execution is complete."

This script uses a sim ple while loop t hat runs every t en m inut es ( 600 seconds) . I f cracker st ill shows up as a running process in t he ps out put , t he ./Reporter -quiet script will run. Ot herwise, t he script ends, print ing Execution is complete. I f you'd like t o receive a pop- up m essage showing t he result s of t he script , see [ H a ck # 1 0 0 ] .

3 .8 .1 .2 Cle a n up Your securit y policy should also provide guidelines on how t o clean up aft er crack finishes. The program st ores several working files in t he run subdirect ory. They will all have a num eric ext ension: # ls run D.boot.69783

Egenisis.69783

bin/

Dgenisis.69783

Kgenisis.69783

dict/

When you rem ove t hose files, ensure you leave t he subdirect ories int act : # cd run # rm *.69783

# ls bin/

dict/

3.8.2 Customizing Password Dictionaries Once you im plem ent regular dict ionary cracks, you'll find t hat aft er a few m ont hs, your users will st art t o consist ent ly choose st rong passwords. However, bear in m ind t hat a dict ionary cracker is only as good as it s dict ionaries. The dict ionaries t hat com e wit h crack are a good st art if your users speak English. Let 's st art by seeing what dict ionaries crack included: # ls dict/1/ abbr.dwg

list.dwg

- 155 -

assurnames.dwg

male-names.dwg

asteroids.dwg

movies.dwg

bad_pws.dat.dwg

myths-legends.dwg

biology.dwg

names.french.dwg

cartoon.dwg

numbers.dwg

chars.dwg

other-names.dwg

common-passwords.txt.dwg

paradise.lost.dwg

crl.words.dwg

phrases.dwg

dosref.dwg

places.dwg

family-names.dwg

python.dwg

famous.dwg

roget.words.dwg

fast-names.dwg

sf.dwg

female-names.dwg

sports.dwg

given-names.dwg

trek.dwg

jargon.dwg

unix.dict.dwg

junk.dwg

yiddish.dwg

lcarrol.dwg

Not ice t hat each built - in dict ionary ends wit h a dwg ext ension. However, crack underst ands any dict ionary or word list , even if it is com pressed ( i.e., it s filenam e ends in eit her .Z or .gz) . I f you use t he file com m and on t he dwg files, you'll find t hat each file is ASCI I t ext . Mind you, t he cont ent s don't look like t he average dict ionary file: # head abbr.dwg #!xdawg 02bon2b 04sa7ya 0bbroyg 6bvgw 0egbdf 0fsasya 0gok 0oottfogvh

- 156 -

0roygbiv

Don't worry, t hose aren't t he act ual words. I nst ead, t he num bers sort t he words by likelihood. That is, t he words don't appear in alphabet ical order, but rat her in t he order t hey're likely t o appear as a password. For exam ple, t he word password is m uch m ore likely t o be used as a password t han pasul. I f your users speak ot her languages, consider downloading addit ional dict ionaries. St art at t he Cerias sit e m ent ioned at t he end of t his hack. I t 's well wort h your while t o browse t hrough t he sit e's dict ionaries, local, and wordlist s subdirect ories looking for dict ionaries t hat suit your part icular needs. Let 's go t here now and check out t he possible word list s: # ftp ftp.cerias.purdue.edu Connected to ftp.cerias.purdue.edu.

Name (ftp.cerias.purdue.edu:dru): anonymous 331 Guest login ok, send your complete e-mail address as password. 230 Logged in anonymously. Remote system type is UNIX. Using binary mode to transfer files. ftp> cd pub/dict/wordlists 250 "/pub/dict/wordlists" is new cwd. ftp> ls 227 Entering Passive Mode (128,10,252,10,169,45) 150 Data connection accepted from 1.2.3.4:49460; transfer starting.

-rw-rw-r--

1 ftpuser

ftpusers

1971 Jun 14

2000 README.gz

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 aussie

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 chinese

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 computer

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 danish

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 dictionaries

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 dutch

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 french

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 german

- 157 -

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 italian

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 japanese

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 literature

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 movieTV

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 names

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 norwegian

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 places

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 random

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 religion

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 science

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 spanish

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 swedish

drwxrwxr-x

2 ftpuser

ftpusers

4096 Jun 14

2000 yiddish

226 Listing completed.

My net work includes several French- speaking users, so I 'll t ake a look at t he French word list : ftp> cd french 250 "/pub/dict/wordlists/french" is new cwd. ftp> ls 227 Entering Passive Mode (128,10,252,10,175,158) 150 Data connection accepted from 1.2.3.4:49530; transfer starting. -rw-rw-r--

1 ftpuser

ftpusers

332537 Jun 14

2000 dico.gz

226 Listing completed.

Before downloading t he word list , I 'll use t he local change direct ory com m and t o ensure I 'm downloading t he file t o t he correct direct ory on m y syst em : ftp> lcd /usr/local/crack/dict/1 Local directory now /usr/local/crack/dict/1 ftp> get dico.gz local: dico.gz remote: dico.gz 227 Entering Passive Mode (128,10,252,10,175,160) 150 Data connection accepted from 1.2.3.4:49531;

- 158 -

transfer starting for dico.gz (332537 bytes). 226 Transfer completed. 332537 bytes received in 00:02 (142.24 KB/s) ftp> bye 221 Goodbye.

Now t hat I have a new word list in / usr/ local/ crack/ dict / 1/ , I 'll run t he following com m and: # cd /usr/local/crack # make rmdict # rm -rf run/dict

That 's it . The next t im e I run ./Crack, I 'll see t he following m essage appended t o t he usual Crack m essage: Crack: making dictionary groups, please be patient... doing group 1... doing group 2... doing group 3... mkdictgrps: uniq'ing dictionary groups... group 1 and 2... group 1 and 3... group 2 and 3... mkdictgrps: compressing dictionary groups... Crack: Created new dictionaries... Crack: Sorting out and merging feedback, please be patient... Crack: Merging password files... Crack: Creating gecos-derived dictionaries mkgecosd: making non-permuted words dictionary mkgecosd: making permuted words dictionary Crack: launching: cracker -kill run/Kgenisis.55941 Done

This indicat es t hat crack has found t he new dict ionary and is m erging it int o it s logic.

- 159 -

3.8.3 See Also • •

The crack web sit e ( ht t p: / / www.crypt icide.org/ users/ alecm ) The Cerias FTP sit e cont aining cracker dict ionaries ( ft p: / / ft p.cerias.purdue.edu/ pub/ dict / )

- 160 -

Hack 31 Create an Effective, Reusable Password Policy

Tr a dit ion a lly, it h a s be e n difficult for a Unix a dm in ist r a t or t o cr e a t e a nd e n for ce a r e u sa ble pa ssw or d policy. For t u n a t e ly, PAM a ddr e sse s t h is. I f you're using FreeBSD 5.0 or higher, your syst em has a PAM ( Pluggable Aut hent icat ion Modules) m odule specifically designed t o assist in t he creat ion and enforcem ent of a reusable password policy. I f you're running a different version of BSD, see t he end of t his hack for ot her sources for t his m odule.

3.9.1 Introducing pam_passwdqc Before using t his m odule, spend som e t im e reading man pam_passwdqc, as it t horoughly covers each opt ion and it s possible values. Any values cont ained wit hin parent heses are default s. As you read t hrough t his m anpage, com pare t hose default s wit h your own net work's securit y policy and m ake not e of any values t hat will require a change. This PAM m odule is fairly com prehensive, allowing you t o enable m any of t he feat ures expect ed in a password policy. Here's an overview of t he configurable feat ures: • • • • •

• • • •

Minim um and m axim um password lengt hs Force a m ix of digit s, lowercase, uppercase, sym bols, and non- ASCI I charact ers Minim um num ber of words in a passphrase Minim um num ber of charact ers t o consider as a st ring ( dict ionary word) Abilit y t o search for st rings t hat are words writ t en backwards, or are words writ t en in a m ix of upper- and lowercase Check new password for sim ilar st ring cont ained wit hin old password Suggest a random ly generat ed password Set t ing t o eit her warn about weak passwords or enforce st rong passwords How m any t im es a user is allowed t o ret ry set t ing a password if he fails t o choose a st rong password

3.9.2 Enabling pam_passwdqc Once you've finished perusing t he m anpage, you should have a list of values t hat you'll want t o m odify t o reflect your net work's securit y policy. Enabling pam_passwdqc is sim ply a m at t er of adding or edit ing a line so t hat it cont ains your cust om ized opt ions. On FreeBSD 4.x, add t hat line t o t he password sect ion of / et c/ pam .conf. On 5.x, edit inst ead t he password sect ion of / et c/ pam .d/ passwd. Let 's look at t hat file on a FreeBSD 5.1 syst em : # more /etc/pam.d/passwd # $FreeBSD: src/etc/pam.d/passwd,v 1.1 2002/04/15 03:01:31 des Exp $ # PAM configuration for the "passwd" service # passwd(1) does not use the auth, account or session services.

- 161 -

# password #password

requisite

password

required

pam_passwdqc.so pam_unix.so

enforce=users no_warn try_first_pass

Obviously, you'll need t o uncom m ent t he pam_passwdqc.so line t o enable t he m odule. Not e t he one included opt ion, enforce=users, overrides t he default set t ing of enforce=everyone. Let 's see what happens when I rem ove t hat rem ark and t hen t ry t o use passwd as a regular user nam ed test. Even t hough passwords aren't echoed t o t he t erm inal, I 've shown in t his out put t he passwords t hat I t yped in: % passwd Changing local password for test Old Password: test You can now choose the new password or passphrase. A valid password should be a mix of upper and lower case letters, digits and other characters.

You can use an 8 character long

password with characters from at least 3 of these 4 classes, or a 7 character long password containing characters from all the classes.

Characters that form a common pattern are discarded by

the check. A passphrase should be of at least 3 words, 12 to 40 characters long and contain enough different characters. Alternatively, if noone else can see your terminal now, you can pick this as your password: "inward!smell:Milan".

As you can see, t he password policy is provided, along wit h an exam ple of a st rong password t hat m eet s t he policy requirem ent s. Except for t hat one opt ion, t his part icular policy includes t he default set t ings m ent ioned in man pam_passwdqc. Enter new password: test Weak password: is the same as the old one. Try again.

Here I t ried t o use t he sam e password. Even worse, it doesn't m eet any of t he password policy's requirem ent s. However, pam_passwdqc rej ect ed t he password, gave m e anot her t ry, and pat ient ly repeat ed t he password policy along wit h anot her password suggest ion:

- 162 -

You can now choose the new password or passphrase. A valid password should be a mix of upper and lower case letters, digits and other characters.

You can use an 8 character long

password with characters from at least 3 of these 4 classes, or a 7 character long password containing characters from all the classes.

Characters that form a common pattern are discarded by

the check. A passphrase should be of at least 3 words, 12 to 40 characters long and contain enough different characters. Alternatively, if noone else can see your terminal now, you can pick this as your password: "Sony,seed,cereal". Enter new password: test1 Weak password: too short. Try again.

Well, I t ried anot her variat ion of m y old password, but it is st ill t oo short . Here we go again: You can now choose the new password or passphrase. A valid password should be a mix of upper and lower case letters, digits and other characters.

You can use an 8 character long

password with characters from at least 3 of these 4 classes, or a 7 character long password containing characters from all the classes.

Characters that form a common pattern are discarded by

the check. A passphrase should be of at least 3 words, 12 to 40 characters long and contain enough different characters. Alternatively, if noone else can see your terminal now, you can pick this as your password: "torso&lotus_burly". Enter new password: test1234 Weak password: not enough different characters or classes for this length. passwd: pam_chauthtok( ): authentication token failure %

- 163 -

Looks like t he default ret ry count is t hree, as I was boot ed out aft er t hree t ries. This t im e t he password was long enough at eight charact ers, but only cont ained num bers and lowercase charact ers. The inst ruct ions clearly st at e t hat an eight - charact er password needs a m ix of t hree different t ypes of charact ers. I t 's im port ant t o not e t hat if t he superuser changes a user's password, she will receive t he sam e error m essages if t he password does not com ply wit h t he policy. However, aft er t he error m essage, t he superuser will be asked t o ret ype t hat poor password and it will be accept ed. Why? Because of t hat enforce=users opt ion. I f you rem ove t hat opt ion, it will default back t o enforce=everyone, which requires even t he superuser t o choose good passwords. The m et hod you choose will depend upon t he securit y requirem ent s of your password policy.

3.9.3 Adding Your Own Options I t 's easy t o change t he default set t ings. Sim ply add your opt ion t o t he end of t he pam _passwdqc.so line. Then, t est your change as a regular user t o see what effect it has. You m ay want t o creat e a t est account for j ust t his purpose. For exam ple, t o force users t o choose a password t hat is 10 charact ers long and a m ix of uppercase let t ers, lowercase let t ers, num bers, and sym bols, set N4 t o 10 and disable t he ot her opt ions. Don't know what N4 is? Bet t er reread t hat sect ion of t he m anpage before changing t his param et er. password

requisite

pam_passwdqc.so \

min=disabled,disabled,disabled,disabled,10

Or, t o force users t o use t he random ly picked password: password

requisite

pam_passwdqc.so

random=42,only

Here I 've used t he default random value of 42. You can experim ent by increasing t hat num ber unt il t he random ly generat ed passwords m eet your st rengt h requirem ent s. Set t ings m uch higher t han 70 m ay produce error m essages; t his is what t he end user will see: System configuration error. Please contact your administrator. passwd: pam_chauthtok(1): authentication token failure

The superuser will see: This system is configured to use randomly generated passwords only, but the attempt to generate a password has failed. This could happen for a number of reasons: you could have requested an impossible password length, or the access to kernel random number pool could have failed. passwd: pam_chauthtok(1): authentication token failure

- 164 -

That 's your hint t o choose a lower random num ber. Once you've set t led on a reasonable num ber, t his is what users will see when t hey change t heir passwords: % passwd Changing local password for test Old Password:

You can now choose the new password. This system is configured to permit randomly generated passwords only.

If noone else can see your terminal now, you can pick this

as your password: "lounge-mummy:cellar-dozen".

Otherwise, come back later.

Enter new password:

A user who hat es t hat password can ret ry a few t im es t o see ot her possibilit ies. Pressing Ent er will generat e anot her random password. Typing in anyt hing ot her t han t he random ly generat ed password will cause t he password change t o fail.

3.9.4 Additional Configuration You m ay have not iced t hat pam_passwdqc does not cont rol how oft en a user is forced t o change his password. Set t his inst ead in / et c/ login.conf. Besides t he act ual expiry period, you can also change t he am ount of advance warning users will receive about an im pending password change. I f you m ake any changes t o / et c/ login.conf, t est your changes by im m ediat ely logging in at anot her t erm inal. A t ypo in t his file can prevent logins t o a syst em ! For exam ple, adding t hese lines t o t he default:\ sect ion will set a password expiry of 30 days, giving 5 days warning: :warnpassword=5d:\ :passwordtime=30d:\ I f one of t hose ent ries happens t o be t he final ent ry in t he default:\ sect ion, don't include t he t railing \ in t hat last ent ry.

Don't forget t o rebuild t he dat abase once you've saved your changes: # cap_mkdb /etc/login.conf

- 165 -

3.9.5 See Also • • • •

man pam_passwdqc man login.conf The Pluggable Password Checking web sit e ( ht t p: / / www.openwall.com / passwdqc/ README.sht m l) The PAM Essent ials sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ art icles/ pam / index.ht m l)

- 166 -

Hack 32 Automate Memorable Password Generation

M a k e it e a sie r for you r u se r s t o ch oose good pa ssw or ds. I t doesn't m at t er whet her you're an adm inist rat or responsible for enforcing a password policy or an end user t rying t o com ply wit h said policy. You're st ruggling against hum an nat ure when you ask users t o choose—and rem em ber—hard- t o- guess passwords. Passwords t hat aren't random are easy t o guess, and passwords t hat are t oo random t end t o m anifest t hem selves on st icky not es under users' keyboards or in t heir t op drawers. Wouldn't it be great if you could som ehow offer users random but m em orable password choices? There's a st andard designed for j ust t his purpose: APG, t he Aut om at ed Password Generat or.

3.10.1 Installing and Using apg I f you're running FreeBSD, you can inst all apg from t he port s collect ion: # cd /usr/ports/security/apg # make install clean

Once t he port is inst alled, any user can run apg t o generat e a list of random , but pronounceable and m em orable, passwords: % apg -q -m 10 -x 10 -M NC -n 10 plerOcGot5 (pler-Oc-Got-FIVE) fobEbpigh6 (fob-Eb-pigh-SIX) Ekjigyerj7 (Ek-jig-yerj-SEVEN) CaujIvOwk8 (Cauj-Iv-Owk-EIGHT) yenViapag0 (yen-Viap-ag-ZERO) Fiwioshev3 (Fi-wi-osh-ev-THREE) Twomitvac4 (Twom-it-vac-FOUR) varbidCyd2 (varb-id-Cyd-TWO) KlepezHap0 (Klep-ez-Hap-ZERO) Naccudhav8 (Nac-cud-hav-EIGHT)

Not ice t hat each password com es wit h a pronunciat ion guide, since it 's easier t o rem em ber som et hing you can pronounce.

- 167 -

Also, not e t hat synt ax. We're definit ely going t o have t o do som et hing about all of t hose swit ches! But first , let 's t ake a look at Sect ion 3.2 and m ake sure we underst and t hem .

Ta ble 3 - 2 . a pg sw it ch e s Opt ion

Ex pla na t ion

-q

Suppresses warnings ( t hink quiet ) , which will be useful when we writ e a script

-m 10

Set s t he m inim um password lengt h t o 10 charact ers

-x 10

Set s t he m axim um password lengt h t o 10 charact ers

-M NC

Requires num erals and capit als

-n 10

Generat es 10 password choices

While t his ut ilit y is very handy, we can definit ely hack in our own im provem ent s. For st art ers, users aren't going t o use a ut ilit y t hat requires a line's wort h of swit ches. Second, we don't want t o inst all t his ut ilit y on every syst em in our net work. I nst ead, let 's work out a CGI script . That way users can access t he script from t heir web browsers.

3.10.2 Improving apg First , let 's sort out all of t he swit ches we'll use in t he script . We need som et hing t o add a punct uat ion charact er in t he m iddle, or we won't m eet Air Force password regulat ions. The sim plest fix is t o run apg t wice wit h sm aller password requirem ent s, concat enat ing t he result s. The first run, wit hout punct uat ion charact ers, looks like t his: % apg -q -m 4 -x 4 -M NC -E Ol -n 10 Dij6 (Dij-SIX) Voj6 (Voj-SIX) Pam0 (Pam-ZERO) Dev9 (Dev-NINE) Non6 (Non-SIX) Eyd7 (Eyd-SEVEN) Vig9 (Vig-NINE) Not8 (Not-EIGHT) Nog2 (Nog-TWO) Von9 (Von-NINE)

Here I 've reduced t he m inim um and m axim um password lengt h t o four charact ers. I 've also added t he opt ion -E Ol t o exclude capit al " oh" and sm all " ell" from passwords, because t hey're easily confused wit h t he digit s zero and one. The second run includes t he -S opt ion, which m akes t he password generat or use special charact ers:

- 168 -

% apg -q -m 4 -x 4 -M S -E Ol -n 10 orc) (orc-RIGHT_PARENTHESIS) tof| (tof-VERTICAL_BAR) fed^ (fed-CIRCUMFLEX) gos@ (gos-AT_SIGN) sig& (sig-AMPERSAND) eif) (eif-RIGHT_PARENTHESIS) eds{ (eds-LEFT_BRACE) lek> (lek-GREATER_THAN) tij: (tij-COLON) rot] (rot-RIGHT_BRACKET)

Now for a CGI script t o past e t he result s t oget her. I 've num bered each line of t he script for explanat ion purposes. Don't include line num bers when you creat e your own script . This script is writ t en in t he Korn shell, but can be m odified for any shell. To run as is, inst all t he Korn shell from / usr/ port s/ shells/ ksh93. 1

#!/bin/ksh

2

# run apg twice, concatenate results.

3

# exclude most special characters requiring shift key,

4

# capital "oh" (looks like zero),

5

# lowercase "ell" (looks like digit "one")

6

PATH=/bin:/usr/bin:/usr/local/bin; export PATH

7

umask 077

8

a=/tmp/apg.$RANDOM

9

b=/tmp/apg.$RANDOM

10

cat $a

26

apg -q -m 4 -x 4 -M S

27

# tr command is for bug workaround; apg is not supposed to

28

# include characters specified after -E option.

29

paste $a $b |

30

tr 'l' 'L' |

31

awk '

32

BEGIN {

33

printf "Password\tRough guess at pronunciation\n"

34

}

35

{

36 37

38 39

-E '!@#$%^&*( )\\' -n 10 > $b

printf "%s%s\t%s %s\n", $1, $3, $2, $4 }'

cat > /etc/ssh/sshd_config

Alt ernat ively, I could have j ust added t hose t hree users direct ly: # echo 'AllowUsers genisis biko dru' >> /etc/ssh/sshd_config

Any user who does not m at ch eit her AllowGroups or AllowUsers will st ill receive a password prom pt when at t em pt ing t o connect t o t he SSH daem on. However, t he connect ion at t em pt will fail wit h a permission denied m essage, even if t he user provides a correct usernam e and password. The SSH daem on will print a m essage regarding t he failed at t em pt t o it s console, sending a copy t o / var/ log/ m essages and em ailing t o root as part of t he daily securit y run out put . To be even pickier, if your users always log in from t he sam e syst em , you can do t his: AllowUsers [email protected] [email protected] [email protected]

However, don't be t hat picky if your users don't have st at ic I Ps! Rem em ber, if you m ake any changes t o t he SSH daem on's configurat ion file, you'll need t o send a " signal one" t o sshd t o not ify it of t he changes: # killall -1 sshd

Aft er inform ing sshd of t he changes, im m ediat ely use a ssh client t o t est your changes. For exam ple, if I inst ead add t he line Allowusers genisis biko dru, I 'll find t hat user nastygirl is st ill able t o connect . Why? The param et ers in / et c/ ssh/ sshd_config are casesensit ive. You don't want t o find out six m ont hs lat er t hat anyone was allowed t o connect when you t hought you had rest rict ed connect ions t o cert ain users.

3.12.4 /etc/login.conf We've rest rict ed who can log in and from where for bot h local and rem ot e ssh logins, but we st ill haven't rest rict ed when t hose users can log in. To do t hat , let 's look at som e ot her opt ions t hat are available in our old friend / et c/ login.conf [ H a ck # 3 0 ] .

- 178 -

This file support s t he opt ions times.allow and times.deny. For exam ple, t o allow all users t o log in bet ween 9: 00 AM and 5: 00 PM every Monday t hrough Friday, add t his line t o t he default:\ sect ion: :times.allow=Mo-Fr0900-1700:\

Once you int roduce t he times.allow opt ion, access will aut om at ically be denied for t he t im e period not list ed. The converse also works. That is, you can specify t he denied t im es in times.deny, and all ot her t im es will be allowed. Rem em ber, whenever you m ake a change t o / et c/ login.conf, rebuild t he dat abase wit h cap_mkdb /etc/login.conf and t est your changes.

3.12.5 See Also • • • •

man man man man

ttys login.access sshd_config login.conf

- 179 -

Chapter 4. Backing Up I nt roduct ion Sect ion 35. Back Up FreeBSD wit h SMBFS Sect ion 36. Creat e Port able POSI X Archives Sect ion 37. I nt eract ive Copy Sect ion 38. Secure Backups Over a Net work Sect ion 39. Aut om at e Rem ot e Backups Sect ion 40. Aut om at e Dat a Dum ps for Post greSQL Dat abases Sect ion 41. Perform Client - Server Cross- Plat form Backups wit h Bacula

- 180 -

Introduction I began gat hering cont ribut ions for t his book, it soon becom e obvious t hat t here would be an ent ire chapt er on backups. Not only do BSD users follow t he m ant ra " backup, backup, backup," but every adm in seem s t o have hacked his own solut ion t o t ake advant age of t he t ools at hand and t he environm ent t hat needs t o be backed up. I f you're looking for t ut orials on how t o use dump and tar, you won't find t hem here. However, you will find nonobvious uses for t heir less well- known count erpart s pax and cpio. I 've also included a hack on backing up over ssh, t o int roduce t he novice user t o t he art of com bining t ools over a secure net work connect ion. You'll also find script s t hat fellow users have creat ed t o get t he m ost out of t heir favorit e backup ut ilit y. Finally, t here are hacks t hat int roduce som e very useful open source t hirdpart y ut ilit ies.

- 181 -

Hack 35 Back Up FreeBSD with SMBFS

A good ba ck up ca n sa ve t he da y w he n t hin gs go w r on g. A ba d—or m issin g—ba ck up ca n r u in t he w hole w e e k . Regular backups are vit al t o good adm inist rat ion. You can perform backups wit h hardware as basic as a SCSI t ape drive using 8m m t ape cart ridges or as advanced as an AI T t ape library syst em using cart ridges t hat can st ore up t o 50 GB of com pressed dat a. But what if you don't have t he luxury of dedicat ed hardware for each server? Since m ost net works are com prised of m ult iple syst em s, you can archive dat a from one server across t he net work t o anot her. We'll back up a FreeBSD syst em using t he tar and gzip archiving ut ilit ies and t he smbutil and mount_smbfs com m ands t o t ransport t hat dat a t o net work shares. These procedures were t est ed on FreeBSD 4.6- STABLE and 5.1RELEASE.

4.2.1 Adding NETSMB Kernel Support Since SMB is a net work- aware filesyst em , we need t o build SMB support int o t he kernel. This m eans adding t he proper options lines t o t he cust om kernel configurat ion file. For inform at ion on building a cust om kernel, see [ H a ck # 5 4 ] , t he Building and I nst alling a Cust om Kernel sect ion ( 9.3) of t he FreeBSD Handbook, and relevant inform at ion cont ained in / usr/ src/ sys/ i386/ conf. Add t he following opt ions under t he makeoptions sect ion: options

NETSMB

# SMB/CIFS requester

options

NETSMBCRYPTO

# encrypted password support for SMB

options

LIBMCHAIN

# mbuf management library

options

LIBICONV

options

SMBFS

Once you've saved your changes, use t he make buildkernel and make installkernel com m ands t o build and inst all t he new kernel.

4.2.2 Establishing an SMB Connection with a Host System The next st ep is t o decide which syst em on t he net work t o connect t o. Obviously, t he dest inat ion server needs t o have an act ive share on t he net work, as well as enough disk space available t o hold your archives. I t will also need a valid user account wit h which you can log in. You'll probably also want t o choose a syst em t hat 's backed up regularly t o rem ovable m edia. I 'll use a m achine nam ed smbserver1.

- 182 -

The smbutil and mount_smbfs com m ands bot h com e st andard wit h t he base inst all of FreeBSD. Their only requirem ent s are t he five kernel opt ions list ed in t he preceding sect ion.

Once you have chosen t he proper host , m ake an SMB connect ion m anually wit h t he smbutil login com m and. This connect ion will rem ain act ive, allowing you t o int eract wit h t he SMB server, unt il you issue t he smbutil logout com m and. So, t o log in: # smbutil login //jwarner@smbserver1 Password: Connected to smbserver1

And t o log out : # smbutil logout //jwarner@smbserver1 Password: Connection unmarked as permanent and will be closed when possible

4.2.3 Mounting a Share Once you're sure you can m anually init iat e a connect ion wit h t he host syst em , creat e a m ount point where you can m ount t he rem ot e share. I 'll creat e a m ount point direct ory called / backup: # mkdir /backup

Next , reest ablish a connect ion wit h t he host syst em and m ount it s share: # smbutil login //jwarner@smbserver1 Password: Connected to smbserver1

# mount_smbfs -N //jwarner@smbserver1/sharename /backup

Not e t hat I used t he -N swit ch t o mount_smbfs t o avoid having t o supply a password a second t im e. I f you prefer t o be prom pt ed for a password when m ount ing t he share, sim ply om it t he -N swit ch.

- 183 -

4.2.4 Archiving and Compressing Data with tar and gzip Aft er connect ing t o t he host server and m ount ing it s net work share, t he next st ep is t o back up and copy t he necessary files. You can get as com plicat ed as you like, but I 'll creat e a sim ple shell script , bkup, inside t he m ount ed share t hat com presses im port ant files and direct ories. This script will m ake com pressed archives of t he / boot , / et c, / hom e, and / usr/ local/ et c direct ories. Add t o or edit t his list as you see fit . At a m inim um , I recom m end including t he / et c and / usr/ local/ et c direct ories, as t hey cont ain im port ant configurat ion files. See man hier for a com plet e descript ion of t he FreeBSD direct ory st ruct ure. #!/bin/sh # script that backs up the following four directories: tar cvvpzf boot.tar.gz /boot tar cvvpzf etc.tar.gz

/etc

tar cvvpzf home.tar.gz /home tar cvvpzf usr_local_etc.tar.gz /usr/local/etc This script is an exam ple t o get you st art ed. There are m any ways t o use tar. Read man 1 tar carefully, and t ailor t he script t o suit your needs.

Be sure t o m ake t his file execut able: # chmod 755 bkup

Run t he script t o creat e t he archives: # ./bkup tar: Removing leading / from absolute path names in the archive. drwxr-xr-x root/wheel

0 Jun 23 18:19 2002 boot/

drwxr-xr-x root/wheel

0 May 11 19:46 2002 boot/defaults/

-r--r--r-- root/wheel -r--r--r-- root/wheel

10957 May 11 19:46 2002 boot/defaults/loader.conf 512 Jun 23 18:19 2002 boot/mbr

(snip)

Aft er t he script finishes running, you'll have * .t ar.gz files of t he direct ories you chose t o archive: # ls | more bkup

- 184 -

boot.tar.gz etc.tar.gz home.tar.gz usr_local_etc.tar.gz

Once you've t est ed your shell script m anually and are happy wit h your result s, add it t o t he cron scheduler t o run on scheduled days and t im es. Rem em ber, how you choose t o im plem ent your backups isn't im port ant —backing up regularly is. Facing t he problem of delet ed or corrupt ed dat a isn't a m at t er of " if" but rat her a m at t er of " when." This is why good backups are essent ial.

4.2.5 Hacking the Hack Things t o consider when m odifying t he script t o suit your own purposes: •

•

Add ent ries t o aut om at ically m ount and unm ount t he share ( see [ H a ck # 6 8 ] for an exam ple) . Use your backup ut ilit y of choice. You're not lim it ed t o j ust tar!

4.2.6 See Also • • • • • •

man 1 smbutil man 8 mount_smbfs man 7 hier man 1 tar man 1 gzip The Building and I nst alling a Cust om Kernel sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ kernelconfigbuilding.ht m l)

- 185 -

Hack 36 Create Portable POSIX Archives

Cr e a t e por t a ble t a r a r ch ive s w it h pa x . Som e POSI X operat ing syst em s ship wit h GNU tar as t he default tar ut ilit y ( Net BSD and QNX6, for exam ple) . This is problem at ic because t he GNU tar form at is not com pat ible wit h ot her vendors' tar im plem ent at ions. GNU is an acronym for " GNU's not UNI X" —in t his case, GNU's not POSI X eit her.

4.3.1 GNU Versus POSIX tar For filenam es or pat hs longer t han 100 charact ers, GNU uses it s own @LongName tar form at ext ension. Som e vendors' tar ut ilit ies will choke on t he GNU ext ensions. Here is what Solaris's archivers say about such an archive: % pax -r < gnu-archive.tar pax: ././@LongLink : Unknown filetype % tar xf gnu-archive.tar tar: directory checksum error

There definit ely appears t o be a disadvant age wit h t he dist ribut ion of non- POSI X archives. A solut ion is t o use pax t o creat e your tar archives in t he POSI X form at . I 'll also provide som e t ips about using pax's feat ures t o com pensat e for t he loss of som e part s of GNU tar's ext ended feat ure set .

4.3.2 Replacing tar with pax The Net BSD and QNX6 pax ut ilit y support s a tar int erface and can also read t he @LongName GNU tar form at ext ension. You can use pax as your tar replacem ent , since it can read your exist ing GNU- form at archives and can creat e POSI X archives for fut ure backups. Here's how t o m ake t he quick conversion. First , replace / usr/ bin/ t ar. That is, renam e GNU tar and save it in anot her direct ory, in case you ever need t o rest ore GNU tar t o it s previous locat ion: # mv /usr/bin/tar /usr/local/bin/gtar

Next , creat e a sym link from pax t o tar. This will allow t he pax ut ilit y t o em ulat e t he tar int erface if invoked wit h t he tar nam e: # ln -s /bin/pax /usr/bin/tar

Now when you use t he tar ut ilit y, your archives will really be creat ed by pax.

- 186 -

4.3.3 Compress Archives Without Using Intermediate Files Let 's say you're on a syst em t hat doesn't have issues wit h tar. Why else would you consider using pax as your backup solut ion? For one, you can use pax and pipelines t o creat e com pressed archives, wit hout using int erm ediat e files. Here's an exam ple pipeline: % find /home/kirk -name '*.[ch]' | pax -w | pgp -c

The pipeline's first st age uses find t o generat e t he exact list of files t o archive. When using tar, you will oft en creat e t he file list using a subshell. Unfort unat ely, t he subshell approach can be unreliable. For exam ple, t his user has so m uch source code t hat t he com plet e file list does not fit on t he com m and line: % tar cf kirksrc.tar $(find /home/kirk -name '*.[ch]') /bin/ksh: tar: Argument list too long

However, in m ore cases, t he pipeline approach will work as expect ed. During t he second st age, pax reads t he list of files from st din and writ es t he archive t o st dout . The pax found on all of t he BSDs has built - in gzip support , so you can also com press t he archive during t his st age by adding t he -z argum ent . When creat ing archives, invoke pax wit hout t he -v ( verbose) argum ent . This way, if t here are any pax error m essages, t hey won't get lost in t he ext ra out put . The t hird st age com presses and/ or encrypt s t he archive. An int erm ediat e tar archive isn't required as t he ut ilit y reads it s dat a from t he pipeline. This exam ple uses pgp, t he Pret t y Good Privacy encrypt ion syst em , which can be found in t he port s collect ion.

4.3.4 Attribute-Preserving Copies POSI X provides t wo ut ilit ies for copying file hierarchies: cp -R and pax -rw. For regular users, cp -R is t he com m on m et hod. But for adm inist rat ive use, pax -rw preserves m ore of t he original file at t ribut es, including hard- link count s and file access t im es. pax -rw also gives you a bet t er copy of t he original file hierarchy. For an exam ple, let 's back up t hree execut ables. Not e t hat egrep, fgrep, and grep are all hard links t o t he sam e execut able.The link count is t hree, and all have t he sam e inode num ber. ls -li displays t he inode num ber in colum n 1 and t he link count in colum n 3: # ls -il /usr/bin/egrep /usr/bin/fgrep /usr/bin/grep 31888 -r-xr-xr-x

3 root

wheel

73784 Sep

8

2002 /usr/bin/egrep

31888 -r-xr-xr-x

3 root

wheel

73784 Sep

8

2002 /usr/bin/fgrep

31888 -r-xr-xr-x

3 root

wheel

73784 Sep

8

2002 /usr/bin/grep

Wit h pax -rw, we will creat e one execut able wit h t he sam e dat e as t he original:

- 187 -

# pax -rw /usr/bin/egrep /usr/bin/fgrep /usr/bin/grep /tmp/ # ls -il /tmp/usr/bin/ 47 -r-xr-xr-x

3 root

wheel

73784 Sep

8

2002 egrep

47 -r-xr-xr-x

3 root

wheel

73784 Sep

8

2002 fgrep

47 -r-xr-xr-x

3 root

wheel

73784 Sep

8

2002 grep

Can we do t he sam e t hing using cp -R? Nope. I nst ead, we creat e t hree new files, each wit h a unique inode num ber, a link count of one, and a new dat e: # rm /tmp/usr/bin/* # cp -R /usr/bin/egrep /usr/bin/fgrep /usr/bin/grep /tmp/usr/bin/ # ls -il /tmp/usr/bin/ 49 -r-xr-xr-x

1 root

wheel

73784 Dec 19 11:26 egrep

48 -r-xr-xr-x

1 root

wheel

73784 Dec 19 11:26 fgrep

47 -r-xr-xr-x

1 root

wheel

73784 Dec 19 11:26 grep

4.3.5 Rooted Archives and the Substitution Argument I f you have ever used GNU tar and received t his m essage: tar: Removing leading `/' from absolute path names in the archive

t hen you were using a tar archive t hat was root ed, where t he files all had absolut e pat hs st art ing wit h t he forward slash ( /) . I t is not a good idea t o clobber exist ing files unint ent ionally wit h foreign binaries, which is why t he GNU tar ut ilit y aut om at ically st rips t he leading / for you. To be safe, you want your unarchiver t o creat e files relat ive t o your current working direct ory. Root ed archives t ry t o violat e t his rule by creat ing files relat ive t o t he root of t he filesyst em , ignoring t he current working direct ory. I f t hat archive cont ained / et c/ passwd, unarchiving it could replace your current password file wit h a foreign copy. You m ay be surprised when you cannot log int o your syst em anym ore! You can use t he pax subst it ut ion argum ent t o rem ove t he leading /. This will ensure t hat t he unarchived files will be creat ed relat ive t o your current working direct ory, inst ead of at t he root of your filesyst em : # pax -A -r -s '-^/--' < rootedarchive.tar

Here, t he -A argum ent request s t hat pax not st rip t he leading / aut om at ically, as we want t o do t his ourselves. This argum ent is required only t o avoid a bug in t he Net BSD pax im plem ent at ion t hat int erferes wit h t he -s argum ent . We also want pax t o unarchive t he file, so we pass t he -r argum ent .

- 188 -

The -s argum ent specifies an ed- st yle subst it ut ion expression t o be perform ed on t he dest inat ion pat hnam e. I n t his exam ple, t he leading / will be st ripped from t he dest inat ion pat hs. See man ed for m ore inform at ion. I f we used t he t radit ional / delim it er, t he subst it ut ion expression would be /^\///. ( The second / isn't a delim it er, so it has t o be escaped wit h a \.) You will find t hat / is t he worst delim it er, because you have t o escape all t he slashes found in t he pat hs. Fort unat ely, you can choose anot her delim it er. Pick one t hat isn't present in t he pat hs, t o m inim ize t he num ber of escape charact ers you have t o add. I n t he exam ple, we used t he - charact er as t he delim it er, and t herefore no escapes were required. The subst it ut ion argum ent can be used t o renam e files for a bet a soft ware release, for exam ple. Say you develop X11R6 soft ware and have m ult iple developm ent versions on your box: /usr/X11R6.saturday /usr/X11R6.working /usr/X11R6.notworking /usr/X11R6.released

and you want t o inst all t he / usr/ X11R6.working direct ory as usr/ X11R6 on t he bet a syst em : # pax -A -w -s '-^/usr/X11R6.working-usr/X11R6-' /usr/X11R6.working \ > /tmp/beta.tar

This t im e, t he -s argum ent specifies a subst it ut ion expression t hat will replace t he beginning of t he pat h / usr/ X11R6.working wit h usr/ X11R6 in t he archive.

4.3.6 Useful Resources for Multiple Volume Archives POSI X does not specify t he form at of m ult ivolum e archive headers, m eaning t hat every archiver m ay use a different int ervolum e header form at . I f you have a lot of m ult ivolum e tar archives and plan t o swit ch t o a different tar im plem ent at ion, you should t est whet her you can st ill recover your old m ult ivolum e archives. This pract ice m ay have been m ore com m on when Minix/ QNX4 users archived t heir 20 MB hard disks t o a st ack of floppy disks. Minix/ QNX4 users had t he vol ut ilit y t o handle m ult iple volum es; inst ead of adding t he m ult ivolum e funct ionalit y t o t he archiver it self, it was handled by a separat e ut ilit y. You should be able t o swit ch archiver im plem ent at ions t ransparent ly because vol did t he split t ing, not t he archiver. The vol ut ilit y perform s t he following operat ions: • • •

At t he end- of- m edia, prom pt s for t he next volum e Verifies t he ordering of t he volum es Concat enat es t he m ult iple volum es

Unfort unat ely, t he vol ut ilit y isn't part of t he Net BSD package collect ion. I f you creat e a lot of m ult ivolum e archives, you m ay want t o look int o port ing one of t he following ut ilit ies:

- 189 -

vol Creat es volum e headers for tar; developed by Brian Yost and available at ht t p: / / groups.google.com / groups?selm = 80% 40m irror.UUCP&out put = gplain

multivol Provides m ult iple volum e support ; creat ed by Marc Schaefer and available at ht t p: / / www.ibiblio.org/ pub/ Linux/ syst em / backup/ m ult ivol- 2.1.t ar.bz2

4.3.7 See Also • • •

•

man pax Net BSD's PGP package ( ft p: / / ft p.Net BSD.org/ pub/ Net BSD/ packages/ pkgsrc/ securit y/ pgp2/ README.ht m l) The GNU tar m anpage on GNU tar and POSI X tar ( ht t p: / / www.gnu.org/ soft ware/ t ar/ m anual/ ht m l_node/ t ar_117.ht m l) The pax -A bug report and fix ( ht t p: / / www.Net BSD.org/ cgi- bin/ query- prsingle.pl?num ber= 23776)

- 190 -

Hack 37 Interactive Copy

W h e n cp a lon e doe sn't qu it e m e e t you r copy ne e ds. The cp com m and is easy t o use, but it does have it s lim it at ions. For exam ple, have you ever needed t o copy a bat ch of files wit h t he sam e nam e? I f you're not careful, t hey'll happily overwrit e each ot her.

4.4.1 Finding Your Source Files I recent ly had t he urge t o find all of t he script s on m y syst em t hat creat ed a m enu. I knew t hat several port s used script s nam ed configure and t hat som e of t hose script s used dialog t o provide a m enu select ion. I t was easy enough t o find t hose script s using find: % find /usr/ports -name configure -exec grep -l "dialog" /dev/null {

} \;

/usr/ports/audio/mbrolavox/scripts/configure /usr/ports/devel/kdesdk3/work/kdesdk-3.2.0/configure /usr/ports/emulators/vmware2/scripts/configure (snip)

This com m and asks find t o st art in / usr/ port s, looking for files -named configure. For each found file, it should search for t he word dialog using -exec grep. The -l flag t ells grep t o list only t he nam es of t he m at ching files, wit hout including t he lines t hat m at ch t he expression. You m ay recognize t he /dev/null { } \; from [ H a ck # 1 3 ] . Norm ally, I could t ell cp t o use t hose found files as t he source and t o copy t hem t o t he specified dest inat ion. This is done by enclosing t he find com m and wit hin a set of backt icks ( `) , locat ed at t he far t op left of your keyboard. Not e what happens, t hough: % mkdir ~/scripts % cd ~/scripts % cp `find /usr/ports -name configure -exec grep -l "dialog" \ /dev/null {

} \;` .

% ls ~/scripts configure

Alt hough each file t hat I copied had a different pat hnam e, t he filenam e it self was configure. Since each copied file overwrot e t he previous one, I ended up wit h one rem aining file.

- 191 -

4.4.2 Renaming a Batch of Source Files What 's needed is t o renam e t hose source files as t hey are copied t o t he dest inat ion. One approach is t o replace t he slash ( /) in t he original file's pat hnam e wit h a different charact er, result ing in a unique filenam e t hat st ill reflect s t he source of t hat file. As we saw in [ H a ck # 1 5 ] , sed is designed t o do such replacem ent s. Here's an approach: % pwd /usr/home/dru/scripts % find /usr/ports -name configure -exec grep -l "dialog" /dev/null { -exec sh -c 'cp {

} `echo {

} \; \

} | sed s:/:=:g`' \;

% ls =usr=ports=audio=mbrolavox=scripts=configure =usr=ports=devel=kdesdk3=work=kdesdk-3.2.0=configure =usr=ports=emulators=vmware2=scripts=configure (snip)

This invocat ion of find st art s off t he sam e as m y original search. I t t hen adds a second exec, which passes an argum ent -c as input t o t he sh shell. The shell will cp t he source files ( specified by { }) , but only aft er sed has replaced each slash in t he pat hnam e wit h an equals sign ( =) . Not e t hat I changed t he sed delim it er from t he default slash t o t he colon ( : ) so I didn't have t o escape m y / st ring. You don't have t o use = as t he new charact er; choose what ever suit s your purposes. awk can also perform t his renam ing feat . The following com m and is m ore or less equivalent t o t he previous com m and: % find /usr/ports -name configure -exec grep -l "dialog" /dev/null {

} \; \

| awk '{dst=$0;gsub("/","=",dst); print "cp",$0,dst}' | sh

4.4.3 Renaming Files Interactively Depending upon how m any files you plan on copying over and how picky you are about t heir dest inat ion nam es, you m ay prefer t o do an int eract ive copy. Despit e it s nam e, cp's int eract ive swit ch ( -i) will fail m iserably in m y scenario: % cp -i `find /usr/ports -name configure -exec grep -l "dialog" \ /dev/null {

} \;` .

overwrite ./configure? (y/n [n]) n not overwritten

- 192 -

overwrite ./configure? (y/n [n]) (snip)

Since each file is st ill nam ed configure, m y only choices are eit her t o overwrit e t he previous file or t o not copy over t he new file. However, bot h cpio and pax are capable of int eract ive copies. Let 's st art wit h cpio: % find /usr/ports -name configure -exec grep -l "dialog" /dev/null {

} \; \

| cpio -o > ~/scripts/test.cpio && cpio -ir < ~/scripts/test.cpio

Here I 've piped m y find com m and t o cpio. Norm ally, I would invoke cpio once in copypass m ode. Unfort unat ely, t hat m ode doesn't support -r, t he int eract ive renam e swit ch. So, I direct ed cpio t o send it s out put ( -o >) t o an archive nam ed ~ / script s/ t est .cpio. I nst ead of piping t hat archive, I used && t o delay t he next cpio operat ion unt il t he previous one finishes. I t hen used -ir t o perform an int eract ive copy in t hat archive so I could t ype in t he nam e of each dest inat ion file. Here are t he result s: cpio: /usr/ports/audio/mbrolavox/scripts/configure: truncating inode number cpio: /usr/ports/devel/kdesdk3/work/kdesdk-3.2.0/configure: truncating inode number cpio: /usr/ports/emulators/vmware2/scripts/configure: truncating inode number (snip other archive messages) 5136 blocks rename /usr/ports/audio/mbrolavox/scripts/configure -> mbrolavox.configure rename /usr/ports/devel/kdesdk3/work/kdesdk-3.2.0/configure -> kdesdk3.configure rename /usr/ports/emulators/vmware2/scripts/configure -> vmware2.configure (snip remaining rename operations) 5136 blocks

Aft er creat ing t he archive, cpio showed m e t he source nam e so I could renam e t he dest inat ion file. While requiring int eract ion on m y part , it does let m e fine- t une exact ly what I 'd like t o call each script . I m ust adm it t hat m y nam es are m uch nicer t han t hose cont aining all of t he equals signs. pax is even m ore efficient . I n t he preceding com m and, t he first cpio has t o wait unt il find com plet es, and t he second cpio has t o wait unt il t he first cpio finishes. Com pare t hat t o t his com m and: % find /usr/ports -name configure -exec grep -l "dialog" /dev/null {

- 193 -

} \; \

| pax -rwi .

Here, I can pipe t he result s of find direct ly t o pax, and pax has very user- friendly swit ches. I n t his com m and, I asked t o read and writ e int eract ively t o t he current direct ory. There's no t em porary archive required, and everyt hing happens at once. Even bet t er, pax st art s working on t he int eract ion before find finishes. Here's what it looks like: ATTENTION: pax interactive file rename operation. -rwxr-xr-x Nov 11 07:53 /usr/ports/audio/mbrolavox/scripts/configure Input new name, or a "." to keep the old name, or a "return" to skip this file. Input > mbrovalox.configure Processing continues, name changed to: mbrovalox.configure

This repeat s for each and every file t hat m at ched t he find result s.

4.4.4 See Also • • •

man cp man cpio man pax

- 194 -

Hack 38 Secure Backups Over a Network

When it com es t o backups, Unix syst em s are ext rem ely flexible. For st art ers, t hey com e wit h built - in ut ilit ies t hat are j ust wait ing for an adm inist rat or's im aginat ion t o com bine t heir t alent s int o a cust om ized backup solut ion. Add t hat t o one of Unix's great est st rengt hs: it s abilit y t o see everyt hing as a file. This m eans you don't even need backup hardware. You have t he abilit y t o send your backup t o a file, t o a m edia, t o anot her server, or t o what ever is available. As wit h any cust om ized solut ion, your success depends upon a lit t le foret hought . I n t his scenario, I don't have any backup hardware, but I do have a net work wit h a 100 Mbps swit ch and a syst em wit h a large hard drive capable of holding backups.

4.5.1 Initial Preparation On t he syst em wit h t hat large hard drive, I have sshd running. ( An alt ernat ive t o consider is t he scponly shell; see [ H a ck # 6 3 ] ) . I 've also creat ed a user and a group called rembackup: # pw groupadd rembackup # pw useradd rembackup -g rembackup -m -s /bin/csh # passwd rembackup Changing local password for rembackup New Password: Retype New Password: #

I f you're new t o t he pw com m and, t he -g swit ch put s t he user in t he specified group ( which m ust already exist ) , t he -m swit ch creat es t he user's hom e direct ory, and t he -s swit ch set s t he default shell. ( There's really no good m nem onic; perhaps no one rem em bers what , if anyt hing, pw st ands for.) Next , from t he syst em I plan on backing up, I 'll ensure t hat I can ssh in as t he user rembackup. I n t his scenario, t he syst em wit h t he large hard drive has an I P address of 10.0.0.1: % sshd -l rembackup 10.0.0.1 The authenticity of host '10.0.0.1 (10.0.0.1)' can't be established. DSA key fingerprint is e2:75:a7:85:46:04:71:51:db:a8:9e:83:b1:5c:7a:2c. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.2.93' (DSA) to the list of known hosts. Password:

- 195 -

% % exit logout Connection to 10.0.0.1 closed.

Excellent . Since I can log in as rembackup, it looks like bot h syst em s are ready for a t est backup.

4.5.2 The Backup I 'll st art by t est ing m y com m and at a com m and line. Once I 'm happy wit h t he result s, I 'll creat e a backup script t o aut om at e t he process. # tar czvf - /usr/home | ssh [email protected] "cat > genisis_usr_home.tgz" usr/home/ usr/home/dru/ usr/home/dru/.cshrc usr/home/dru/mail/ usr/home/mail/sent-mail Password:

This tar com m and creat es ( c) a com pressed ( z) backup t o a file ( f) while showing t he result s verbosely ( v) . The m inus charact er ( -) represent s t he specified file, which in t his case is st dout . This allows m e t o pipe st dout t o t he ssh com m and. I 've provided / usr/ hom e, which cont ains all of m y users' hom e direct ories, as t he hierarchy t o back up. The result s of t hat backup are t hen piped ( |) t o ssh, which will send t hat out put ( via cat) t o a com pressed file called genisis_usr_hom e.t gz in t he rembackup user's hom e direct ory. Since t hat direct ory holds t he backups for m y net work, I chose a filenam e t hat indicat es t he nam e of t he host , genisis, and t he cont ent s of t he backup it self. 4 .5 .2 .1 Aut om a t ing t h e ba ck up Now t hat I can securely back up m y users' hom e direct ories, I can creat e a script . I t can st art out as sim ple as t his: # more /root/bin/backup #!/bin/sh # script to backup /usr/home to backup server tar czvf - /usr/home | ssh [email protected] "cat > genisis_usr_home.tgz"

However, whenever I run t hat script , I 'll overwrit e t he previous backup. I f t hat 's not m y int ent ion, I can include t he dat e as part of t he backup nam e:

- 196 -

tar czvf - /usr/home | ssh [email protected] "cat > \ genisis_usr_home.`date +%d.%m.%y`.tgz"

Not ice I insert ed t he date com m and int o t he filenam e using backt icks. Now t he backup file will include t he day, m ont h, and year separat ed by dot s, result ing in a filenam e like genisis_usr_hom e.21.12.03.t gz. Once you're happy wit h your result s, your script is an excellent candidat e for a cron j ob.

4.5.3 See Also • • •

man tar man ssh man pw

- 197 -

Hack 39 Automate Remote Backups

M a k e r e m ot e ba ck ups a u t om a t ic a nd e ffor t le ss. One day, t he I DE cont roller on m y web server died, leaving t he files on m y hard disk hopelessly corrupt ed. I faced what I had known in t he back of m y m ind all along: I had not been m aking regular rem ot e backups of m y server, and t he local backups were of no use t o m e now t hat t he drive was corrupt ed. The reason for t his, of course, is t hat doing rem ot e backups wasn't aut om at ic and effort less. Adm it t edly, t his was no one's fault but m y own, but m y frust rat ion was sufficient enough t hat I decided t o writ e a t ool t hat would m ake aut om at ed rem ot e snapshot s so easy t hat I wouldn't ever have t o worry about it again. Ent er rsnapshot.

4.6.1 Installing and Configuring rsnapshot I nst allat ion on FreeBSD is a sim ple m at t er of: # cd /usr/ports/sysutils/rsnapshot # make install

I didn't include t he clean t arget here, as I 'd like t o keep t he work subdirect ory, which includes som e useful script s. I f you're not using FreeBSD, see t he original HOWTO at t he proj ect web sit e for det ailed inst ruct ions on inst alling from source.

The inst all process neit her creat es nor inst alls t he config file. This m eans t hat t here is absolut ely no possibilit y of accident ally overwrit ing a previously exist ing config file during an upgrade. I nst ead, copy t he exam ple configurat ion file and m ake changes t o t he copy: # cp /usr/local/etc/rsnapshot.conf.default /usr/local/etc/rsnapshot.conf

The rsnapshot .conf config file is well com m ent ed, and m uch of it should be fairly selfexplanat ory. For a full reference of all t he various opt ions, please consult man rsnapshot. rsnapshot uses t he / .snapshot s/ direct ory t o hold t he filesyst em snapshot s. This is referred t o as t he snapshot root . This m ust point t o a filesyst em where you have lot s of free disk space. Not e t hat fields are separat ed by t abs, not spaces. This m akes it easier t o specify file pat hs wit h spaces in t hem .

- 198 -

4 .6 .1 .1 Spe cifyin g ba ck u p in t e r va ls rsnapshot has no idea how oft en you want t o t ake snapshot s. I n order t o specify how m uch dat a t o save, you need t o t ell rsnapshot which int ervals t o keep, and how m any of each. By default , a snapshot will occur every four hours, or six t im es a day ( t hese are t he hourly int ervals) . I t will also keep a second set of snapshot s, t aken once a day and st ored for a week ( or seven days) : interval

hourly

6

interval

daily

7

Not e t hat t he hourly int erval is specified first . This is very im port ant , as t he first int erval line is assum ed t o be t he sm allest unit of t im e, wit h each addit ional line get t ing successively bigger. Thus, if you add a yearly int erval, it should go at t he bot t om , and if you add a m inut es int erval, it should go before t he hourly int erval. I t 's also wort h not ing t hat t he snapshot s are pulled up from t he sm allest int erval t o t he largest . I n t his exam ple, t he daily snapshot s are pulled from t he oldest hourly snapshot , not direct ly from t he m ain filesyst em . The backup sect ion t ells rsnapshot which files you act ually want t o back up: backup

/etc/

localhost/etc/

I n t his exam ple, backup is t he backup point , / et c/ is t he full pat h t o t he direct ory we want t o t ake snapshot s of, and localhost / et c/ is a subdirect ory inside t he snapshot root where t he snapshot s are st ored. I f you are t aking snapshot s of several m achines on one dedicat ed backup server, it 's a good idea t o use host nam es as direct ories t o keep t rack of which files cam e from which server. I n addit ion t o full pat hs on t he local filesyst em , you can also back up rem ot e syst em s using rsync over ssh. I f you have ssh enabled ( via t he cmd_ssh param et er) , specify a pat h sim ilar t o t his: backup

[email protected]:/etc/

example.com/etc/

This behaves fundam ent ally t he sam e way as specifying local pat hnam es, but you m ust t ake a few ext ra t hings int o account : • • •

•

The ssh daem on m ust be running on exam ple.com . You m ust have access t o t he specified account on t he rem ot e m achine ( in t his case, t he backup user on exam ple.com ) . See [ H a ck # 3 8 ] for inst ruct ions on set t ing t his up. You m ust have key- based logins enabled for t he specified user at exam ple.com , wit hout passphrases. This backup occurs over t he net work, so it m ay be slower. Since t his uses rsync, t his is m ost not iceable during t he first backup. Depending on how m uch your dat a changes, subsequent backups should go m uch fast er.

- 199 -

One t hing you can do t o m it igat e t he pot ent ial dam age from a backup server breach is t o creat e alt ernate users on t he client m achines wit h t heir UI Ds and GI Ds set t o 0, but wit h a m ore rest rict ive shell, such as scponly [ H a ck # 6 3 ] .

4 .6 .1 .2 Pr e pa r in g for scr ipt a ut om a t ion Wit h t he backup_script param et er, t he second colum n is t he full pat h t o an execut able backup script , and t he t hird colum n is t he local pat h in which you want t o st ore it . For exam ple: backup_script

/usr/local/bin/backup_pgsql.sh

localhost/postgres/

You can find t he backup_pgsql.sh exam ple script in t he ut ils/ direct ory of t he source dist ribut ion. Alt ernat ively, if you didn't include t he clean t arget when you inst alled t he FreeBSD port , t he file will be locat ed in / usr/ port s/ sysut ils/ rsnapshot / work/ rsnapshot - 1.0.9/ ut ils.

Your backup script only needs t o dum p it s out put int o it s current working direct ory. I t can creat e as m any files and direct ories as necessary, but it should not put it s files in any predet erm ined pat h. This is because rsnapshot creat es a t em p direct ory, changes t o t hat direct ory, runs t he backup script , and t hen syncs t he cont ent s of t he t em p direct ory t o t he local pat h you specified in t he t hird colum n. A t ypical backup script m ight look like t his: #!/bin/sh

/usr/bin/mysqldump -uroot mydatabase > mydatabase.sql /bin/chown 644 mydatabase.sql

There are a couple of exam ple script s in t he ut ils/ direct ory of t he rsnapshot source dist ribut ion t o give you m ore ideas. Rem em ber t hat backup script s will be invoked as t he user running rsnapshot. Make sure your backup script s are not writ able by anyone else.

4 .6 .1 .3 Te st in g your con fig file Aft er m aking your changes, verify t hat t he config file is synt act ically valid and t hat all t he support ing program s are where you t hink t hey are: # rsnapshot configtest

I f all is well, t he out put should say Syntax OK. I f t here's a problem , it should t ell you exact ly what it is.

- 200 -

The final st ep t o t est your configurat ion is t o run rsnapshot wit h t he -t flag, for t est m ode. This will print out a verbose list of t he t hings it will do, wit hout act ually doing t hem . For exam ple, t o sim ulat e an hourly backup: # rsnapshot -t hourly

4 .6 .1 .4 Sch e du lin g r sn a pshot Now t hat you have your config file set up, it 's t im e t o schedule rsnapshot t o run from cron. Add t he following lines t o root's cront ab: 0 */4 * * *

/usr/local/bin/rsnapshot hourly

30 23 * * *

/usr/local/bin/rsnapshot daily

4.6.2 The Snapshot Storage Scheme All backups are st ored wit hin a configurable snapshot root direct ory. I n t he beginning it will be em pt y. rsnapshot creat es subdirect ories for t he various defined int ervals. Aft er a week, t he direct ory should look som et hing like t his: # ls -l /.snapshots/ drwxr-xr-x

7 root

root

4096 Dec 28 00:00 daily.0

drwxr-xr-x

7 root

root

4096 Dec 27 00:00 daily.1

drwxr-xr-x

7 root

root

4096 Dec 26 00:00 daily.2

drwxr-xr-x

7 root

root

4096 Dec 25 00:00 daily.3

drwxr-xr-x

7 root

root

4096 Dec 24 00:00 daily.4

drwxr-xr-x

7 root

root

4096 Dec 23 00:00 daily.5

drwxr-xr-x

7 root

root

4096 Dec 22 00:00 daily.6

drwxr-xr-x

7 root

root

4096 Dec 29 00:00 hourly.0

drwxr-xr-x

7 root

root

4096 Dec 28 20:00 hourly.1

drwxr-xr-x

7 root

root

4096 Dec 28 16:00 hourly.2

drwxr-xr-x

7 root

root

4096 Dec 28 12:00 hourly.3

drwxr-xr-x

7 root

root

4096 Dec 28 08:00 hourly.4

drwxr-xr-x

7 root

root

4096 Dec 28 04:00 hourly.5

Each of t hese direct ories cont ains a full backup of t hat point in t im e. The dest inat ion direct ory pat hs you specified as t he backup and backup_script param et ers are placed direct ly under t hese direct ories. I n t he exam ple: backup

/etc/

localhost/etc/

- 201 -

t he / et c/ direct ory will init ially back up int o / .snapshot s/ hourly.0/ localhost / et c/ . Each subsequent t im e rsnapshot is run wit h t he hourly com m and, it will rot at e t he hourly.X direct ories, " copying" t he cont ent s of t he hourly.0 direct ory ( using hard links) int o hourly.1. When rsnapshot daily runs, it will rot at e all t he daily.X direct ories, t hen copy t he cont ent s of hourly.5 int o daily.0. hourly.0 will always cont ain t he m ost recent snapshot , and daily.6 will always cont ain a snapshot from a week ago. Unless t he files change bet ween snapshot s, t he full backups are really j ust m ult iple hard links t o t he sam e files. This is how rsnapshot uses space so efficient ly. I f t he file changes at any point , t he next backup will unlink t he hard link in hourly.0, replacing it wit h a brand new file. This will now use t wice t he disk space it did before, but it is st ill considerably less space t han 13 full, unique copies would occupy. Rem em ber, if you are using different int ervals t han t he ones in t his exam ple, t he first int erval list ed is t he one t hat get s updat es direct ly from t he m ain filesyst em . All subsequent ly list ed int ervals pull from t he previous snapshot s.

4.6.3 Accessing Snapshots When rsnapshot first runs, it will creat e t he configured snapshot _root direct ory. I t assigns t his direct ory t he perm issions 0700 since t he snapshot s will probably cont ain files owned by all sort s of users on your syst em . The sim plest but least flexible solut ion is t o disallow access t o t he snapshot root alt oget her. The root user will st ill have access, of course, and will be t he only one who can pull backups. This m ay or m ay not be desirable, depending on your sit uat ion. For a sm all set up, t his m ay be sufficient . I f users need t o be able t o pull t heir own backups, you will need t o do a lit t le ext ra work up front . The best opt ion seem s t o be creat ing a cont ainer direct ory for t he snapshot root wit h 0700 perm issions, giving t he snapshot root direct ory 0755 perm issions, and m ount ing t he snapshot root for t he users as read- only using NFS or Sam ba. Let 's explore how t o do t his using NFS on a single m achine. First , set t he snapshot_root variable in rsnapshot .conf: snapshot_root

/usr/.private/.snapshots/

Then, creat e t he cont ainer direct ory, t he real snapshot root , and a read- only m ount point : # mkdir /usr/.private/ # mkdir /usr/.private/.snapshots/ # mkdir /.snapshots/

Set t he proper perm issions on t hese new direct ories: # chmod 0700 /usr/.private/ # chmod 0755 /usr/.private/.snapshots/

- 202 -

# chmod 0755 /.snapshots/

I n / et c/ export s, add / usr/ .privat e/ .snapshot s/ as a read- only NFS export : /usr/.private/.snapshots/

127.0.0.1(ro)

I f your version of NFS support s it , include t he no_root_squash opt ion. ( Place it wit hin t he bracket s aft er ro wit h a com m a—not a space—as t he separat or.) This opt ion allows t he root user t o see all t he files wit hin t he read- only export .

I n / et c/ fst ab, m ount / usr/ .privat e/ .snapshot s/ read- only under / .snapshot s/ : localhost:/usr/.private/.snapshots/

/.snapshots/

nfs

ro

0 0

Rest art your NFS daem on and m ount t he read- only snapshot root : # /etc/rc.d/nfsd restart # mount /.snapshots/

To t est t his, t ry adding a file as t he superuser: # touch /.snapshots/testfile

This should fail wit h insufficient perm issions. This is what you want . I t m eans t hat your users won't be able t o m ess wit h t he snapshot s eit her. Users who wish t o recover old files can go int o t he / .snapshot s direct ory, select t he int erval t hey want , and browse t hrough t he filesyst em unt il t hey find t he files t hey are looking for. NFS will prevent t hem from m aking m odificat ions, but t hey can copy anyt hing t hat t hey had perm ission t o read in t he first place.

4.6.4 See Also • •

man rsnapshot The original rsnapshot HOWTO ( ht t p: / / www.rsnapshot .org/ rsnapshot - HOWTO.ht m l)

- 203 -

Hack 40 Automate Data Dumps for PostgreSQL Databases

Bu ilding you r ow n ba ck up u t ilit y doe sn 't h a ve t o be sca r y. Post greSQL is a robust , open source dat abase server. Like m ost dat abase servers, it provides ut ilit ies for creat ing backups. Post greSQL's prim ary t ools for creat ing backup files are pg_dump and pg_dumpall. However, if you want t o aut om at e your dat abase backup processes, t hese t ools have a few lim it at ions: • • •

pg_dump dum ps only one dat abase at a t im e. pg_dumpall dum ps all of t he dat abases int o a single file. pg_dump and pg_dumpall know not hing about m ult iple backups.

These aren't crit icism s of t he backup t ools—j ust an observat ion t hat cust om izat ion will require a lit t le script ing. Our result ing script will backup m ult iple syst em s, each t o t heir own backup file.

4.7.1 Creating the Script This script uses Pyt hon and it s abilit y t o execut e ot her program s t o im plem ent t he following backup algorit hm : 1. Change t he working direct ory t o a specified dat abase backup direct ory. 2. Renam e all backup files ending in .gz so t hat t hey end in .gz.old. Exist ing files ending in .gz.old will be overwrit t en. 3. Clean up and analyze all Post greSQL dat abases using it s vacuumdb com m and. 4. Get a current list of dat abases from t he Post greSQL server. 5. Dum p each dat abase, piping t he result s t hrough gzip, int o it s own com pressed file. Why Pyt hon? My choice is one of personal preference; t his t ask is achievable in j ust about any script ing language. However, Pyt hon is cross- plat form and easy t o learn, and it s script s are easy t o read.

4.7.2 The Code #!/usr/local/bin/python

# /usr/local/bin/pg2gz.py

# This script lists all PostgreSQL # databases and pipes them separately # through gzip into .gz files.

# INSTRUCTIONS

- 204 -

# 1.

Review and edit line 1 to reflect the location

#

of your python command file.

# 2.

Redefine the save_dir variable (on line 22) to

#

your backup directory.

# 3.

To automate the backup process fully, consider

#

scheduling the regular execution of this script

#

using cron.

import os, string

# Redefine this variable to your backup directory. # Be sure to include the slash at the end. save_dir = '/mnt/backup/databases/'

# Rename all *.gz backup files to *.gz.old. curr_files = os.listdir(save_dir) for n in curr_files: if n[len(n)-2:] =

= 'gz':

os.popen('mv ' + save_dir + n + " " + save_dir + n + '.old') else: pass

# Vacuum all databases os.popen('vacuumdb -a -f -z')

# 'psql -l' produces a list of PostgreSQL databases. get_list = os.popen('psql -l').readlines( )

# Exclude header and footer lines. db_list = get_list[3:-2]

# Extract database names from first element of each row.

- 205 -

for n in db_list: n_row = string.split(n) n_db = n_row[0]

# Pipe database dump through gzip # into .gz files for all databases # except template*. if n_db =

= 'template0':

pass elif n_db =

= 'template1':

pass else: os.popen('pg_dump ' + n_db + ' | gzip -c > ' + save_dir + n_db + '.gz')

4.7.3 Running the Hack The script assum es t hat you have a working inst allat ion of Post greSQL. You'll also need t o inst all Pyt hon, which is available t hrough t he port s collect ion or as a binary package. The Pyt hon m odules used are inst alled by default . Double- check t he locat ion of your Pyt hon execut able using: % which python /usr/local/bin/python

and ensure t he first line of t he script reflect s your locat ion. Don't forget t o m ake t he script execut able using chmod +x. On line 22 of t he script , redefine t he sav_dir variable t o reflect t he locat ion of your backup direct ory. As is, t he script assum es a backup direct ory of / m nt / backup/ dat abases/ . You'll probably want t o add t he script t o t he pgsql user's cront ab for periodic execut ion. To schedule t he script for execut ion, log in as pgsql or, as t he superuser, su t o pgsql. Once you're act ing as pgsql, execut e: % crontab -e

t o open t he cront ab file in t he default edit or. Given t he following cront ab file, / usr/ local/ bin/ pg2gz.py will execut e at 4 AM every Sunday.

- 206 -

# more /var/cron/tabs/pgsql SHELL=/bin/sh PATH=/var/cron/tabs:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin

#minute

hour

mday

month

wday

command

0

4

*

*

0

/usr/local/bin/pg2gz.py

4.7.4 See Also • •

The Post greSQL web sit e ( ht t p: / / www.post gresql.org/ ) The Pyt hon web sit e ( ht t p: / / www.pyt hon.org/ )

- 207 -

Hack 41 Perform Client-Server Cross-Platform Backups with Bacula

D on 't le t t h e ca m py n a m e fool you . Ba cu la is a pow e r fu l, fle x ible , ope n sou r ce ba ck u p pr ogr a m . . Having problem s finding a backup solut ion t hat fit s all your needs? One t hat can back up bot h Unix and Windows syst em s? That is flexible enough t o back up syst em s wit h irregular backup needs, such as lapt ops? That allows you t o run script s before or aft er t he backup j ob? That provides browsing capabilit ies so you can decide upon a rest ore point ? Bacula m ay be what you're looking for.

4.8.1 Introducing Bacula Bacula is a client - server solut ion com posed of several dist inct part s:

Direct or The Direct or is t he m ost com plex part of t he syst em . I t keeps t rack of all client s and files t o be backed up. This daem on t alks t o t he client s and t o t he st orage devices.

Client / File Daem on The Client ( or File) Daem on runs on each com put er which will be backed up by t he Direct or. Som e ot her backup solut ions refer t o t his as t he Agent .

St orage Daem on The St orage Daem on com m unicat es wit h t he backup device, which m ay be t ape or disk.

Console The Console is t he prim ary int erface bet ween you and t he Direct or. I use t he com m and- line Console, but t here is also a GNOME GUI Console. Each File Daem on will have an ent ry in t he Direct or configurat ion file. Ot her im port ant ent ries include FileSet s and Jobs. A FileSet ident ifies a set of files t o back up. A Job specifies a single FileSet , t he t ype of backup ( increm ent al, full, et c.) , when t o do t he backup, and what St orage Device t o use. Backup and rest ore j obs can be run aut om at ically or m anually.

- 208 -

4.8.2 Installation Bacula st ores det ails of each backup in a dat abase. You can use eit her SQLit e or MySQL, and st art ing wit h Bacula Version 1.33, Post greSQL. Before you inst all Bacula, decide which dat abase you want t o use. FreeBSD 4.x ( prior t o 4.10- RELEASE) and FreeBSD 5.x ( Version 5.2.1 and earlier) have a pt hreads bug t hat could cause you t o lose dat a. Refer t o plat form / freebsd/ pt hreads- fix.t xt in your Bacula source direct ory for full det ails.

The exist ing Bacula docum ent at ion provides det ailed inst allat ion inst ruct ions if you're inst alling from source. To inst all inst ead t he SQLit e version of t he FreeBSD port : # cd /usr/ports/sysutils/bacula # make install

Or, if you prefer t o inst all t he MySQL version: # cd /usr/ports/sysutils/bacula # make -DWITH_MYSQL install Don't use t he clean t arget wit h your make com m and, because t here are som e script s in t he work direct ory you'll need t o use.

4.8.3 Configuration Files Bacula inst alls several configurat ion files t hat should work for your environm ent wit h few m odificat ions. 4 .8 .3 .1 File D a e m on on t he ba ck u p clie n t The first configurat ion file, / usr/ local/ et c/ bacula- fd.conf, is for t he File Daem on. This file needs t o reside on each m achine you want t o back up. For securit y reasons, only t he Direct ors specified in t his file will be able t o com m unicat e wit h t his File Daem on. The nam e and password specified in t he Direct or resource m ust be supplied by any connect ing Direct or. You can specify m ore t han one Director { } resource. Make sure t he password m at ches t he one in t he Client resource in t he Direct or's configurat ion file. The FileDaemon { } resource ident ifies t his syst em and specifies t he port on which it will list en for Direct ors. You m ay have t o creat e a direct ory m anually t o m at ch t he one specified by t he Working Directory.

4 .8 .3 .2 St or a ge D a e m on on t h e ba ck u p se r ve r

- 209 -

The next configurat ion file, / usr/ local/ et c/ bacula- sd.conf, is for t he St orage Daem on. The default values should work unless you need t o specify addit ional st orage devices. As wit h t he File Daem on, t he Director { } resource specifies t he Direct or( s) t hat m ay cont act t his St orage Daem on. The password m ust m at ch t hat found in t he St orage resource in t he Direct or's configurat ion file. 4 .8 .3 .3 D ir e ct or on t h e ba ck up se r ve r The Direct or's configurat ion is by necessit y t he largest of t he daem ons. Each Client , Job, FileSet , and St orage Device is defined in t his file. I n t he following exam ple configurat ion, I 've defined t he Job Client1 t o back up t he files defined by t he FileSet Full Set on a lapt op. The backup will be perform ed t o t he File st orage device, which is really a disk locat ed at lapt op.exam ple.org. This isn't an opt im al solut ion for a real backup, as I 'm j ust backing up files from t he lapt op t o som ewhere else on t he lapt op. I t is sufficient for dem onst rat ion and t est ing, t hough. # more /usr/local/etc/bacula-dir.conf

Director { Name

= laptop-dir

DIRport

= 9101

QueryFile

= "/usr/local/etc/query.sql"

WorkingDirectory

= "/var/db/bacula"

PidDirectory

= "/var/run"

Maximum Concurrent Jobs = 1 Password

= "lLftflC4QtgZnWEB6vAGcOuSL3T6n+P7jeH+HtQOCWwV"

Messages

= Standard

} Job { Name

= "Client1"

Type

= Backup

Client

= laptop-fd

FileSet

= "Full Set"

Schedule

= "WeeklyCycle"

Storage

= File

Messages

= Standard

- 210 -

Pool

= Default

Write Bootstrap = "/var/db/bacula/Client1.bsr" Priority

= 10

} FileSet { Name = "Full Set" Include = signature=MD5 { /usr/ports/sysutils/bacula/work/bacula-1.32c }

# If you backup the root directory, the following two excluded #

files can be useful

# Exclude = { /proc /tmp /.journal /.fsck } } Client { Name

= laptop-fd

Address

= laptop.example.org

FDPort

= 9102

Catalog

= MyCatalog

Password

= "laptop-client-password"

File Retention = 30 days Job Retention

= 6 months

AutoPrune

= yes

} # Definition of file storage device Storage { Name

= File

Address

= laptop.example.org

SDPort

= 9103

Password

= "TlDGBjTWkjTS/0HNMPF8ROacI3KlgIUZllY6NS7+gyUp"

Device

= FileStorage

- 211 -

Media Type = File }

Not e t hat t he password given by any connect ing Console m ust m at ch t he one here.

4.8.4 Database Setup Now t hat you've m odified t he configurat ion files t o suit your needs, use Bacula's script s t o creat e and define t he dat abase t ables t hat it will use. To set up for MySQL: # cd /usr/ports/sysutils/bacula/work/bacula-1.32c/src/cats # ./grant_mysql_privileges # ./create_mysql_database # ./make_mysql_tables

I f you have a password set for t he MySQL root account , add -p t o t hese com m ands and you will be prom pt ed for t he password. You now have a working dat abase suit able for use by Bacula.

4.8.5 Testing Your Tape Drive Som e t ape drives are not st andard. They require t heir own propriet ary soft ware and can be t em peram ent al when used wit h ot her soft ware. Regardless of what soft ware it uses, each drive m odel can have it s own lit t le quirks t hat need t o be cat ered t o. Fort unat ely, Bacula com es wit h btape, a handy lit t le ut ilit y for t est ing your drive. My t ape drive is at / dev/ sa1. Bacula prefers t o use t he non- rewind variant of t he device, but it can handle t he raw variant as well. I f you use t he rewinding device, t hen only one backup j ob per t ape is possible. This com m and will t est t he non- rewind device / dev/ nrsa1: # /usr/local/sbin/btape -c /usr/local/etc/bacula-sd.conf /dev/nrsa1

4.8.6 Running Without Root I t is a good idea t o run daem ons wit h t he lowest possible privileges. The St orage Daem on and t he Direct or Daem on do not need root perm issions. However, t he File Daem on does, because it needs t o access all files on your syst em . I n order t o run daem ons wit h nonroot account s, you need t o creat e a user and a group. Here, I used vipw t o creat e t he user. I select ed a user I D and group I D of 1002, as t hey were unused on m y syst em . bacula:*:1002:1002::0:0:Bacula Daemon:/var/db/bacula:/sbin/nologin

I also added t his line t o / et c/ group:

- 212 -

bacula:*:1002:

The bacula user ( as opposed t o t he Bacula daem on) will have a hom e direct ory of / var/ db/ bacula, which is t he default locat ion for t he Bacula dat abase. Now t hat you have bot h a bacula user and a bacula group, you can secure t he bacula hom e direct ory by issuing t his com m and: # chown -R bacula:bacula /var/db/bacula/

4.8.7 Starting the Bacula Daemons To st art t he Bacula daem ons on a FreeBSD syst em , issue t he following com m and: # /usr/local/etc/rc.d/bacula.sh start

To confirm t hey are all running: # ps auwx | grep bacula

root 63416 0.0 0.3 2040 1172 ?? Ss 4:09PM 0:00.01 /usr/local/sbin/bacula-sd -v -c /usr/local/etc/bacula-sd.conf root 63418 0.0 0.3 1856 1036 ?? Ss 4:09PM 0:00.00 /usr/local/sbin/bacula-fd -v -c /usr/local/etc/bacula-fd.conf root 63422 0.0 0.4 2360 1440 ?? Ss 4:09PM 0:00.00 /usr/local/sbin/bacula-dir -v -c /usr/local/etc/bacula-dir.conf

4.8.8 Using the Bacula Console The console is t he m ain int erface t hrough which you run j obs, query syst em st at us, and exam ine t he Cat alog cont ent s, as well as label, m ount , and unm ount t apes. There are t wo consoles available: one runs from t he com m and line, and t he ot her is a GNOME GUI . I will concent rat e on t he com m and- line console. To st art t he console, I use t his com m and: #

/usr/local/sbin/console -c /usr/local/etc/console.conf

Connecting to Director laptop:9101 1000 OK: laptop-dir Version: 1.32c (30 Oct 2003) *

- 213 -

You can obt ain a list of t he available com m ands wit h t he help com m and. The status all com m and is a quick and easy way t o verify t hat all com ponent s are up and running. To label a Volum e, use t he label com m and. Bacula com es wit h a preset backup j ob t o get you st art ed. I t will back up t he direct ory from which Bacula was inst alled. Once you get going and have creat ed your own j obs, you can safely rem ove t his j ob from t he Direct or configurat ion file. Not surprisingly, you use t he run com m and t o run a j ob. Once t he j ob runs, t he result s will be sent t o you via em ail, according t o t he Messages resource set t ings wit hin your Direct or configurat ion file. To rest ore a j ob, use t he restore com m and. You should choose t he rest ore locat ion carefully and ensure t here is sufficient disk space available. I t is easy t o verify t hat t he rest ored files m at ch t he original: # diff -ruN \ /tmp/bacula-restores/usr/ports/sysutils/bacula/work/bacula-1.32c \ /usr/ports/sysutils/bacula/work/bacula-1.32c #

4.8.9 Creating Backup Schedules For m y t est ing, I want ed t o back up files on m y Windows XP m achine every hour. I creat ed t his schedule: Schedule { Name = "HourlyCycle" Run

= Full 1st sun at 1:05

Run

= Differential 2nd-5th sun at 1:05

Run

= Incremental Hourly

}

Any Job t hat uses t his schedule will be run at t he following t im es: • •

•

A full backup will be done on t he first Sunday of every m ont h at 1: 05 AM. A different ial backup will be run on t he 2nd, 3rd, 4t h, and 5t h Sundays of every m ont h at 1: 05 AM. Every hour, on t he hour, an increm ent al backup will be done.

4.8.10 Creating a Client-only Install So far we have been t est ing Bacula on t he server. Wit h t he FreeBSD port , inst alling a client only version of Bacula is easy: # cd /usr/ports/sysutils/bacula

- 214 -

# make -DWITH_CLIENT_ONLY install

You will also need t o t ell t he Direct or about t his client by adding a new Client resource t o t he Direct or configurat ion file. You will also want t o creat e a Job and FileSet resource. When you change t he Bacula configurat ion files, rem em ber t o rest art t he daem ons: # /usr/local/etc/rc.d/bacula.sh restart Stopping the Storage daemon Stopping the File daemon Stopping the Director daemon Starting the Storage daemon Starting the File daemon Starting the Director daemon #

4.8.11 See Also • •

The Bacula web sit e ( ht t p: / / www.bacula.org/ ) ht t p: / / www.onlam p.com / pub/ a/ onlam p/ 2004/ 01/ 09/ bacula.ht m l ( t he original Bacula art icle from ONLam p)

- 215 -

Chapter 5. Networking Hacks I nt roduct ion Sect ion 42. See Console Messages Over a Rem ot e Login Sect ion 43. Spoof a MAC Address Sect ion 44. Use Mult iple Wireless NI C Configurat ions Sect ion 45. Survive Cat ast rophic I nt ernet Loss Sect ion 46. Hum anize t cpdum p Out put Sect ion 47. Underst and DNS Records and Tools Sect ion 48. Send and Receive Em ail Wit hout a Mail Client Sect ion 49. Why Do I Need sendm ail? Sect ion 50. Hold Em ail for Lat er Delivery Sect ion 51. Get t he Most Out of FTP Sect ion 52. Dist ribut ed Com m and Execut ion Sect ion 53. I nt eract ive Rem ot e Adm inist rat ion

- 216 -

Introduction You probably spend m ost of your t im e accessing servers on t he I nt ernet or on your own net work. I n fact , net working has becom e so prevalent , it 's becom ing increasingly difficult t o t olerat e even short periods of net work out ages. This chapt er cont ains m any ideas for accessing net working services when t he convent ional avenues seem t o be unavailable. Have you ever want ed t o t rain your syst em t o not ify you of it s new net work configurat ion when it s prim ary link becom es unavailable? Would you like t o check your em ail from a syst em t hat doesn't cont ain a preconfigured em ail client ? How can you m aint ain net work connect ivit y when your I SP's DHCP server no longer recognizes your DHCP client ? You'll also gain insight int o how som e of t he net working services and t ools we oft en t ake for grant ed work. Becom e a tcpdump guru—or at least lose t he int im idat ion fact or. Underst and your DNS m essages and how t o t roubleshoot your DNS servers. Tam e your sendmail daem on. Finally, m eet t wo excellent open source ut ilit ies t hat allow you t o perform rout ine t asks sim ult aneously on all of your servers.

- 217 -

Hack 42 See Console Messages Over a Remote Login

Vie w a se r ve r 's con sole m e ssa ge s r e m ot e ly As a Unix syst em adm inist rat or, you can do 99% of your work rem ot ely. I n fact , it is very rare indeed t hat you'll need t o sit down in front of a server ( assum ing t he server even has an at t ached keyboard! [ H a ck # 2 6 ] ) . However, one of t he key funct ionalit ies you lose in rem ot e adm inist rat ion is t he abilit y t o see t he rem ot e server's console. All is not lost, t hough. First , let 's answer t hese quest ions: " What do you m ean by t he console, and why would you want t o see it ?"

5.2.1 The Console I f you're physically sit t ing at a syst em , t he console is t he virt ual t erm inal you see when you press Alt - F1. I f you've ever logged int o t his part icular virt ual t erm inal, you've probably not iced t hat error m essages appear here. These m essages can be rat her disconcert ing when you're working at t he console, especially if you're fight ing your way t hrough vi and bright whit e error m essages occasionally overwrit e your t ext . I f you ever find yourself in t hat sit uat ion, Esc- Ct rl- r will refresh your screen. Bet t er yet , don't log int o Alt - F1 when you're physically sit t ing at a syst em . I nst ead, log int o a different t erm inal, say, t he one at Alt - F2. However, when you access a rem ot e syst em , you can't log int o a virt ual t erm inal, and t he console is considered t o be a virt ual t erm inal. ( You access it by pressing Alt - F1 at t he local keyboard, aft er all) . I nst ead, you log int o a pseudot erm inal ( also known as a net work t erm inal) . Here's an exam ple. I 'm sit t ing at a syst em and have logged int o t he virt ual t erm inals at Alt F2 and Alt - F3. From Alt - F3, I 've used ssh t o log int o t he localhost . I f I run t he w com m and, I 'll see t his: % w 12:25

up 22 mins, 3 users, load averages: 0:00, 0:00, 0:00

USER

TTY

FROM

LOGIN@

IDLE WHAT

genisis

v1

-

12:25PM

- -csh (csh)

genisis

v2

-

12:25PM

- ssh localhost

genisis

p0

localhost

12:25PM

- w

Not ice t hat t he virt ual ( or physical access t o keyboard) t erm inals begin wit h a v in t he TTY sect ion. Since t erm inals st art num bering at 0, I 'm logged int o t he second ( v1) and t hird ( v2) virt ual t erm inals. I 'm also connect ed t o t he first pseudot erm inal, p0, so I 'm current ly t he only user logged in over t he net work. I n m y ssh session, if I press Alt - F1, I 'll access t he console on m y local syst em ( where I am sit t ing) , not t he console on t he rem ot e syst em .

- 218 -

5.2.2 Seeing Remote Console Messages I f Alt - F1 won't do it , how can you see rem ot e console m essages? A quick hack for your current session is t o run t his com m and: % tail -f /var/log/messages &

tail shows t he end of a file, m uch like head shows t he st art . I n t his case, t he file is / var/ log/ m essages. This part icular log cont ains a copy of t he m essages t hat appear on t he syst em console. When run wit h t he -f swit ch, tail will rem ain open, allowing you t o see when new ent ries are added t o t hat logfile. The t railing am persand ( &) runs t he com m and in t he background, so you'll get your prom pt back if you press Ent er or t ype in anot her com m and. As t he syst em writ es console ent ries t o t his file, tail will also display t o your current pseudot erm inal. I f you're in t he m iddle of t yping som et hing when a log m essage is displayed, Ct rl- r will refresh your com m and prom pt line so you can see where you left off t yping.

5.2.3 An Alternate Method There's always m ore t han one way t o skin a cat . Since syslog is responsible for logfiles, you can also change it s configurat ion file. Let 's st art by seeing why m essages are sent t o t he console: % grep console /etc/syslog.conf *.err;kern.debug;auth.notice;mail.crit

/dev/console

# uncomment this to log all writes to /dev/console to /var/log/console.log #console.info

/var/log/console.log

See how m essages are sent t o / dev/ console by default ? This file also gives a hint on how t o send t hose m essages elsewhere—t o a file called console.log. By uncom m ent ing t hat console.info line, you can send t hose m essages t o / var/ log/ console.log. I f you decide t o rem ove t hat #, don't forget t o creat e an em pt y logfile wit h t he specified nam e and t o inform syslogd of your changes by sending it a signal one: # touch /var/log/console.log # killall -1 syslogd

Now you're probably t hinking, big deal. So I 've sent console m essages t o a different filenam e. I st ill have t o run t hat tail -f com m and t o see t hem . Well, how about changing t hat console.info line t o t his inst ead: console.info

root,genisis

Don't forget t o killall -1 syslogd once you save your changes.

- 219 -

Now when I ssh int o t hat syst em as t he user genisis, I don't have t o rem em ber t o run t he tail com m and. As long as I 'm t he user genisis, even if I becom e t he superuser, all console m essages will be sent t o m y t erm inal.

5.2.4 Hacking the Hack You m ay have not iced t hat uncom m ent ing t he console.info line result s in m essages being sent t wice: once t o / var/ log/ console.log and once t o eit her t he original console or t he specified users. I f you prefer t o only have m essages sent t o eit her t he log or t he console or user, recom m ent t he console.info line and indicat e in t he line t hat originally specified / dev/ console where you want t he inform at ion t o go. For exam ple, t o log only t o a file: *.err;kern.debug;auth.notice;mail.crit

/var/log/console

Or t o log only t o t he specified users: *.err;kern.debug;auth.notice;mail.crit

root,genisis

Again, don't forget t o inform syslogd of any changes you m ake t o / et c/ syslog.conf.

5.2.5 See Also • •

man w man syslog.conf

- 220 -

Hack 43 Spoof a MAC Address

Eve n good gu ys ca n u se se cr e t ide n t it ie s. Okay, I know what you're t hinking. There's never a legit im at e reason t o spoof any t ype of address, right ? Even if t here were, why would you bot her t o spoof a MAC address, ot her t han t o prove t hat it can be done? Consider t he following scenario. I was adm inist rat ing a sm all net work where t he I SP rest rict ed t he num ber of I P addresses a DHCP client was allowed t o receive. Their DHCP server kept t rack of t he leased addresses by using a com binat ion of t he client 's MAC address and an OS ident ifier. One day I needed t o replace t hat net work's ext ernal NI C. I t t ook m e a while t o figure out why t he new NI C refused t o pick up a DHCP address from t he I SP. Once t he rest rict ion was explained t o m e, I cont em plat ed m y available courses of act ion. One was t o spend t he aft ernoon list ening t o Musak in t he hopes t hat I 'd event ually get t o speak t o one of t he I SP's cust om er service represent at ives. I decided m y t im e would be bet t er spent if I inst ead t ook 30 seconds and spoofed t he old MAC address. This provided a quick solut ion t hat allowed t he owner t o get back online unt il he could m ake arrangem ent s wit h t he I SP regarding t he new MAC address.

5.3.1 Spoofing on FreeBSD Before I could accom plish t he spoof, I needed t wo pieces of inform at ion. The first was t he MAC address for t he old NI C. Fort unat ely, I record such t hings in a binder. However, I init ially found out t hat inform at ion using ifconfig. I n t his scenario, t he int erface in quest ion was called rl0: % ifconfig rl0 rl0: flags=8843 mtu 1500 inet 192.168.2.12 netmask 0xffffff00 broadcast 192.168.2.255 ether 00:05:5d:d2:19:b7 media: Ethernet autoselect (10baseT/UTP)

The MAC address is t he hex num ber im m ediat ely following ether. Second, I needed t o know t he ident ifier used by t he I SP's DHCP server. This was found in t he DHCP lease: % more /var/db/dhclient.leases | grep host option host-name "00-05-5d-d2-19-b7-36-33"

Som e I SPs use option host-name, while ot hers use option dhcp-client-identifier. Choose t he opt ion in t he lease t hat is associat ed wit h t he MAC address. I n t his exam ple, m y ident ifier was t he MAC address, followed by -36-33.

- 221 -

Arm ed wit h t he inform at ion I needed, I could spoof t he old MAC address ont o t he new NI C card. I n m y case, t he new card was an ed0: # ifconfig ed0 ether 00:05:5d:d2:19:b7

# # ifconfig ed0 | grep ether ether 00:05:5d:d2:19:b7

Not e t hat you have t o be t he superuser t o change t hese set t ings. This part icular change won't survive a reboot , as t he NI C will give t he kernel it s burnt - in MAC address during t he hardware probe t hat occurs during boot up. I f you int end t o reboot before sort ing out t he sit uat ion wit h t he I SP, carefully add t his line t o / et c/ rc.conf: ifconfig_ed0_alias0="ether 00:05:5d:d2:19:b7"

This will creat e an alias for ed0 t hat uses t he desired MAC address, rat her t han t he MAC address burnt int o t he physical card. Think of an alias as an alt ernat e set of inst ruct ions an int erface can give t o t he kernel—a kind of net working nicknam e. Next , I 'll edit / et c/ dhclient .conf: # vi /etc/dhclient.conf # $FreeBSD: src/etc/dhclient.conf,v 1.3 2001/10/27 03:14:37 rwatson Exp $ # #

This file is required by the ISC DHCP client.

#

See ``man 5 dhclient.conf'' for details.

# #

In most cases an empty file is sufficient for most people as the

#

defaults are usually fine.

# interface "ed0" { send host-name "00-05-5d-d2-19-b7-36-33"; send dhcp-client-identifier "00-05-5d-d2-19-b7-36-33"; }

By default , t his file cont ains only com m ent s; I added a sect ion for int erface ed0. When edit ing your own file, rem em ber t o include t he opening and closing curly braces ( {}) . Each

- 222 -

st at em ent m ust also end in a sem icolon ( ;) . Here, I 've set bot h t he host-name and t he dhcp-client-identifier opt ions t o t he values expect ed by t he I SP. Now it 's t im e t o t est t hat t hese changes did indeed work. You don't need t o reboot in order t o t est t hat alias in / et c/ rc.conf. This com m and will do t he t rick: # /etc/netstart Doing stage one network startup: Doing initial network setup:. ed0: flags=8843 mtu 1500 inet 192.168.2.95 netmask 0xffffff00 broadcast 192.168.2.255 ether 00:05:5d:d2:19:b7 lo0: flags=8049 mtu 16384 inet 127.0.0.1 netmask 0xff000000 Additional routing options: ignore ICMP redirect=YES log ICMP redirect=YES drop SYN+FIN packets=YESsysctl: unknown oid 'net.inet.tcp.drop_synfin' . Routing daemons:.

Excellent . The new NI C kept t he spoofed MAC address. Now let 's see how t he DHCP server responds when I release and t ry t o renew an address: # dhclient -r ed0 #

Using -r wit h dhclient forces t he DHCP client t o give up it s old address and request a new lease from t he DHCP server. I f t his succeeds, t he prom pt will ret urn wit hout any error m essages. Running ifconfig ed0 will show t hat t he I SP's DHCP server did indeed give t his int erface a public I P address.

5.3.2 Spoofing on NetBSD The current version of ifconfig t hat ships wit h Net BSD does not support t his funct ionalit y. To allow MAC address changes, t ry Dheeraj Reddy's ifconfig pat ch, available from ht t p: / / news.gw.com / net bsd.t ech.net / % 3C20030808072355.GA616% 40bharat i.sudheeraj .n et % 3E. You will need t o apply t his pat ch t o Net BSD sources and build a new version of ifconfig. To begin, download t he syst em sources, unpack t hem , and change t he working direct ory t o src/ sbin/ ifconfig. Download t he pat ch and apply it wit h: # patch > ifconfig.patch

- 223 -

Build a new binary wit h: # make

Rem em ber t hat t his code is experim ent al and m ay not always work as advert ised, so it is crucial t hat you back up t he original ifconfig binary in som e safe place. When you have t he new binary, run it wit h: # ifconfig interface-name lladdr MAC-addr

5.3.3 Spoofing with OpenBSD The st andard ifconfig t hat ships wit h OpenBSD does not cont ain an opt ion t o change t he MAC addresses of int erface cards. I f you need it , you will have t o build your own t ool for t hat purpose wit h sea.c. Download it from ht t p: / / www.devguide.net / books/ openbsdfw- 02ed/ Build sea as follows: # gcc -Wall -o sea sea.c -lkvm

Next , boot OpenBSD int o single- user m ode: # reboot boot> boot -s

Then, once in single- user m ode, use sea t o spoof t he desired address on t he specified NI C: # sea

interface-name

MAC-addr

5.3.4 See Also • •

man ifconfig man dhclient.conf

- 224 -

Hack 44 Use Multiple Wireless NIC Configurations

Ta k e t h e pa in ou t of con figu r ing you r la pt op's w ir e le ss in t e r fa ce . I f you use a lapt op and have rem ot e sit es t hat you visit regularly, configuring your wireless int erface can be int erest ing. For exam ple, every wireless net work has a unique service set ident ifier ( SSI D) . Each sit e t hat uses WEP will also require a unique encrypt ion key. Som e net works m ay use st at ic I P addresses, while ot hers m ay use a DHCP server. You could keep a copy of each net work's configurat ion in your wallet and reconfigure your NI C m anually at each sit e, but wouldn't you rat her aut om at e t he various net work configurat ions and choose t he desired configurat ion aft er boot up? For t he purpose of t his exercise, we will assum e t hat t he wireless access point s have been properly configured and act ivat ed.

5.4.1 Initial Preparation Before you can script t he net work configurat ions, you'll need t o collect t he inform at ion list ed next . I 've associat ed t he necessary inform at ion wit h ifconfig's keywords where possible. You will see t hese keywords in t he configurat ion script . • • • • • • • • •

ssid, t he nam e of t he wireless net work authmode, t he net work's aut horizat ion m ode ( none, open, or shared) nwkey, t he encrypt ion key, in hexadecim al Whet her t o use a st at ic I P address or dhclient t o obt ain dynam ic I P address inform at ion inet, t he st at ic I P address, if necessary netmask, t he net m ask, for st at ic net work configurat ion The default gat eway, for st at ic I P configurat ion Nam eservers, for st at ic I P configurat ion The net work device ( wi0, an0, et c.)

You can obt ain all but t he final it em from whoever set up t he wireless access point s for each sit e. I f you don't know t he nam e of your net work device, review t he out put of dmesg for net working prot ocol nam es ( Et hernet , 802.11) and MAC addresses. Here's t he com m and I use and t he relevant lines from m y lapt op: # dmesg | grep address rl0: Ethernet address: 00:08:02:9e:df:b8 wi0: 802.11 address: 00:06:25:17:74:be

rl0 is t he device nam e for t he cabled Et hernet port , and wi0 is t he device nam e for t he wireless PCMCI A card.

- 225 -

5.4.2 Preparing the Script Here are a few not es regarding t he net work device configurat ion script : • •

•

• • •

•

• •

The script is nam ed for t he net work device it cont rols. The script will live in / usr/ local/ et c/ rc.d. Since we do not want t he script act ivat ed at boot up, t he script nam e m ust not end in .sh. Each net work device should have it s own script so t hat t he connect ion can be easily dropped using t he argum ent stop. Each configurat ion will have it s own sect ion in a case const ruct . Each sect ion's nam e will consist of a d ( t o use DHCP) or an s ( t o use a st at ic I P address) followed by a locat ion nam e. The script will accept a sect ion nam e as a com m and line argum ent for configurat ion select ion. I n order t o use WEP wit h DHCP, t he device m ust be configured wit h t he encrypt ed code prior t o calling dhclient. A status sect ion will give us current net work inform at ion for t he device. A wildcard sect ion will print a list of t he sect ion nam es when given an invalid argum ent .

Since m y net work device is wi0, I 'll save t he script as / usr/ local/ et c/ rc.d/ wi0. I t end t o use m y lapt op in t hree locat ions: at hom e wit h DHCP and WEP, at hom e wit h a st at ic I P address and WEP, and at m y sist er's hom e wit h DHCP and WEP. Tables Table 5- 1 t hrough Table 5- 3 list t he appropriat e configurat ions.

Ta ble 5 - 1 . Usin g D H CP a n d W EP in m y h om e n e t w or k Opt ion na m e

Va lu e

section name

dhome

ssid

myhome

authmode

shared

nwkey

0x123456789a

ip address

Use dhclient t o obt ain t he I P address, net m ask, gat eway, and nam eservers

Ta ble 5 - 2 . Usin g a st a t ic I P a ddr e ss a n d W EP in m y h om e n e t w or k Opt ion na m e

Va lu e

section name

shome

ssid

myhome

authmode

shared

nwkey

0x123456789a

ip address

192.168.1.21

netmask

255.255.255.0

gateway

192.168.1.1

name servers

24.204.0.4, 24.204.0.5

- 226 -

Ta ble 5 - 3 . Usin g D H CP a n d W EP a t m y sist e r 's h om e Opt ion na m e

Va lu e

section name

dsister

ssid

sisterhome

authmode

shared

nwkey

0x987654321a

ip address

Use dhclient t o obt ain t he I P address, net m ask, gat eway, and nam eservers

5.4.3 The Code Here is t he result ing script : #!/bin/sh # /usr/local/etc/rc.d/wi0 # Configure wireless interface

# See the ifconfig(8), dhclient(8) and route(8) man pages for further # assistance.

NIC=wi0

case $1 in dhome) ifconfig ${NIC} ssid "myhome" authmode "shared" nwkey 0x123456789a dhclient ${NIC} echo ${NIC} ;; shome) ifconfig ${NIC} inet 192.168.1.21 ssid "myhome" authmode "shared" nwkey 0x123456789a netmask 255.255.255.0 route add default 192.168.1.1 echo nameserver 24.204.0.4 > /etc/resolv.conf echo nameserver 24.204.0.5 >> /etc/resolv.conf

- 227 -

echo ${NIC} ;; dsister) ifconfig ${NIC} ssid "sisterhome" authmode "shared" nwkey \ 0x987654321a dhclient ${NIC} echo ${NIC} ;; stop) [ -s /var/run/dhclient.pid ] && kill `cat /var/run/dhclient.pid` \ && rm /var/run/dhclient.pid ifconfig ${NIC} remove echo " ${NIC} removed" ;; status) ifconfig ${NIC} ;; *) echo "usage: /usr/local/etc/${NIC} [dhome|shome|dsister|stop|status]" ;; esac

Not e t hat t he stop opt ion kills dhclient. I f you will be using m ult iple net work int erfaces, you m ay wish t o delet e t he line t hat reads: [ -s /var/run/dhclient.pid ] && kill `cat /var/run/dhclient.pid` && rm \ /var/run/dhclient.pid

The script should be owned by root and be readable by root only. I f you creat e your script as a norm al user, you need t o change it s owner. Becom e t he superuser, and: # chown root:wheel /usr/local/etc/rc.d/wi0 # chmod 700 /usr/local/etc/wi0

- 228 -

5.4.4 Running the Hack Using t he script is fairly st raight forward. To act ivat e t he dhome configurat ion ( DHCP at hom e) : # /usr/local/etc/rc.d/wi0 dhome wi0

To rem ove t he wi0 int erface and kill t he connect ion: # /usr/local/etc/rc.d/wi0 stop wi0 removed

I f I ent er an erroneous argum ent , I 'll get a list of valid argum ent s: # /usr/local/etc/rc.d/wi0 badargument usage: /usr/local/etc/wi0 [dhome|shome|dsister|stop|status]

Now you can choose an exist ing net work configurat ion wit hout having t o rem em ber any net work det ails. A sim ilar script will work for cabled net work devices. Sim ply change t he device nam e and rem ove t he wireless keywords ( ssid, authmode, and nwkey) and values.

5.4.5 Hacking the Hack For all t he geek point s, you could put your wireless card in prom iscuous m ode ( if it support s it ) , sniff for t he available ESSI Ds and t heir signal st rengt hs, and choose t he appropriat e configurat ion based on t hat inform at ion. I f you go t his rout e, inst all t he net/bsd-airtools port and rem em ber t o ask for perm ission before using som eone else's resources.

5.4.6 See Also • • •

man dhclient man ifconfig man route

- 229 -

Hack 45 Survive Catastrophic Internet Loss

Se t up your n e t w or k t o r e cove r fr om a fu ll I n t e r n e t loss. Som eday t his all t oo com m on event m ay happen: while you're away from your net work, your connect ion dies. Whet her t he I SP drops it , t he cable get s unplugged or t he server behind your NAT box dies, it is gone. You are now lost at sea, not knowing what is act ually going on back at hom e. You ping, telnet, and pray t o t he net work gods, but not hing seem s t o work. Wouldn't it be bet t er if your net work could recognize t hat it has lost t hat connect ion and find a way for you t o get back in t ouch? The syst em t hat I set up did j ust t hat . All it t ook was a well- configured OpenBSD firewall wit h NAT and a short Ruby program t hat uses t he Jabber prot ocol t o get m y at t ent ion.

5.5.1 Hardware Configuration I use OpenBSD on a 486 t o m ake m y net work resist ant t o t ot al connect ivit y failure. The com put er has t wo net work cards, one for t he DSL bridge and t he ot her for t he rest of t he net work. I n addit ion, I m anaged t o find a 56k I SA m odem . Since t his com put er provides lit t le m ore t han firewall and NAT services, it 's m ore t han capable of serving a sm all hom e or business net work. The DSL bridge provides t he prim ary I nt ernet connect ion wit h a st at ic I P. The service t hrough m y provider is usually quit e good, but t here have been t roubled t im es. The house has only one phone line, which is plugged int o t he 56k m odem in t he sam e com put er as t he DSL line. You could easily m ake t he m odem com put er a different m achine ent irely, but I found t hat t his 486 is quit e com pact and sufficient for m y purposes.

5.5.2 Connectivity Software The current OpenBSD operat ing syst em ( Version 3.4 as of t his writ ing) com es wit h a wonderful firewall and NAT package, nam ed Packet Filt er ( PF) . PF works well on a day- t oday basis m oving m y packet s from t he net work t o t he I nt ernet . Unfort unat ely, it does not handle t he loss of t he connect ion t o t he I SP. A full discussion for configuring PF is beyond t he scope of t his hack, but you can find what you need from t he OpenBSD PF FAQ at ht t p: / / www.openbsd.org/ faq/ pf/ index.ht m l. When t he unt hinkable happens and your net work falls off t he I nt ernet , you m ay fall back t o your t rust y 56k m odem . The idea is t hat t he m odem will dial out aut om at ically once your m ain connect ion goes away. First , t hough, you need som e way t o det ect t hat your connect ion is lost . I use a slow ping t o t he rout er on t he ot her end of m y DSL connect ion. I run t his heart beat from cron inst ead of using a daem on process. I t sends t hree pings at t wo- second int ervals every 10 m inut es—a very conservat ive t est , especially if you are only sending t o your local gat eway. Here is t he cron ent ry: */10 * * * * /usr/local/testconnect/testconnect.sh

The t est connect .sh script resem bles t his:

- 230 -

#!/bin/sh

# First gather data about your connection PINGS=`ping -c 1 -i 2 [your gateway] | wc -l`

# Apply test and execute on result if [ -f /tmp/lostconnection.lock ] then echo "Lockfile in place" else echo "No lockfile" if [ $PINGS -lt 8 ] then echo "Connection lost, commencing dialup" touch /tmp/lostconnection.lock pfctl -d ppp -nat internet ruby /usr/local/testconnect/send_new_ip.rb else echo "All is well" fi fi

I f t he gat eway is unavailable, t hen t he pings will t im e out and generat e a short ping result . By count ing t he num ber of let t ers ( wit h wc -l) and applying a lengt h t est ( if [ $PINGS lt 8]) , t he script can t ell if t he pings failed. I n t he case of failure, t he script goes t hrough t he st eps t o give you connect ivit y t hrough alt ernat ive m eans and t o st op it from doing it every 10 m inut es if t hings go really wrong. First , it creat es a lockfile t o ensure fut ure runs of t his script do not dial out over and over again. Second, it shut s down t he current NAT int erface t o m ake way for t he next st ep. Third, it fires up t he m odem and connect s t o m y em ergency I SP using a preconfigured ppp.conf profile called internet. Here, I enabled NAT ( -nat) over PPP so t hat com put ers at m y house will only not ice t hat t he service is slow. The I nt ernet connect ion will st ill funct ion in t he sam e way. Finally, I run a script t o alert m e t o t he failure. You m ay have not iced one flaw in t his set up. Most cheap I SP services usually do not give you t he sam e I P address when you dial int o t hem . How do you know how t o cont act your reconnect ed gat eway from t he out side? Easy: have t he com put er t ell you.

- 231 -

5.5.3 Jabber and Ruby to the Rescue! There are m any ways a com put er can cont act you wit h it s current st at us. I decided t o use Jabber because I spend a fair am ount of t im e wit h a Jabber session running. This script will not ify m e quickly if som et hing unt oward happens t o m y connect ion at hom e, such as an incident involving t he vacuum cleaner. I figured t hat a m essage from m y com put er wit h t he current net work configurat ion would provide enough inform at ion t o allow m e t o log in rem ot ely. The m ost im port ant inform at ion is t he current I P address of t he backup PPP connect ion. I decided t o creat e a Ruby script using t he Jabber4r m odule t o accom plish t his: require 'jabber4r/jabber4r'

now

= `date`.chomp!

ipdata = `/sbin/ifconfig tun0`

session = Jabber::Session.bind_digest("user@jabberserver/modem", "secret") session.new_chat_message("user@jabberserver"). set_body("I had to dial up for internet access at #{now}.\n#{ipdata}\n") .send

sleep 5 session.close

The Ruby script grabs t he current t im e and st at e of t he tun0 int erface, which cont ains t he current I P address assigned by t he dial- up I SP. Arm ed wit h t hat I P address, you can t hen ssh int o your com put er and begin t o diagnose t he sit uat ion. The Jabber4r m odule lives at ht t p: / / j abber4r.rubyforge.org/ . You will also need t he REXML m odule from ht t p: / / www.germ ane- soft ware.com / soft ware/ rexm l/ . Bot h of t hese inst alled wit hout issue on t op of t he Ruby package t hat shipped wit h OpenBSD 3.4.

5.5.4 The Last Piece Aft er your connect ion has been rest ored, you need t o clean up. You will need t o st op ppp, st art PF again—hopefully wit h pfctl—and rem ove t he lockfile t hat prevent s t he / t m p/ t est connect .sh script from dialing out over and over. Aft er t hat , you should be back t o norm al, at least unt il t he next m ishap.

5.5.5 See Also • •

The Jabber web sit e ( ht t p: / / www.j abber.org/ ) The Ruby web sit e ( ht t p: / / www.ruby- lang.org/ en/ )

- 232 -

Hack 46 Humanize tcpdump Output

M a k e fr ie nds w it h tcpdump. One of t he m ost useful ut ilit ies in a net work adm inist rat or's t ool belt is tcpdump. While you probably agree, I bet t he very t hought of wading t hrough a tcpdump sniff m akes you groan. Take heart : I 'll walk you t hrough som e concret e exam ples t hat show how t o zero in on t he inform at ion you need t o solve t he part icular net work problem t hat prom pt ed you t o consider doing a packet sniff in t he first place. You m ight be t hinking, " Why bot her? There are m uch nicer ut ilit ies out t here." That 's t rue. My personal favorit e happens t o be ethereal. However, you don't always have t he luxury of working on a syst em t hat allows you t o inst all t hird- part y ut ilit ies or, for t hat m at t er, even has X inst alled. tcpdump is guarant eed t o be on your BSD syst em . I t 's t here, it 's quick, it 's dirt y, and it 's darn effect ive if you know how t o harness it s power.

5.6.1 The Basics Let 's st art wit h t he basics: st art ing a capture. Before you can capt ure any packet s, you need t o be t he superuser. You also need t o have t he bpf device in your kernel. I f you're using t he GENERI C kernel, you're set . I f you've creat ed your own cust om kernel [ H a ck # 5 4 ] , double- check you st ill have t hat device. I n t his exam ple, m y kernel configurat ion file is called CUSTOM: # grep bpf /usr/src/sys/i386/conf/CUSTOM # The 'bpf' device enables the Berkeley Packet Filter. device

bpf

#Berkeley packet filter

You also need t o know t he nam es of your int erfaces and which int erface is cabled t o t he net work you wish t o sniff. You can find t his wit h ifconfig: # ifconfig rl0: flags=8802 mtu 1500 inet 192.168.3.20 netmask 0xffffff00 broadcast 192.168.3.255 ether 00:05:5d:d2:19:b7 media: Ethernet autoselect (10baseT/UTP) rl1: flags=8802 mtu 1500 inet 192.168.12.43 netmask 0xffffff00 broadcast 192.168.12.255 ether 00:05:5d:d1:ff:9d media: Ethernet autoselect (10baseT/UTP)

- 233 -

ed0: flags=8843 mtu 1500 inet 192.168.2.95 netmask 0xffffff00 broadcast 192.168.2.255 ether 00:50:ba:de:36:33 lp0: flags=8810 mtu 1500 lo0: flags=8049 mtu 16384 inet 127.0.0.1 netmask 0xff000000

This part icular syst em has t hree Et hernet ( ether) cards at t ached t o t hree different net works. Since I 'm int erest ed in t he t raffic on t he 192.168.2.0 net work, I 'll use t he ed0 int erface. To st art a capt ure, sim ply specify t he int erface you're int erest ed in, wit h t he int erface ( -i) swit ch: # tcpdump -i ed0 tcpdump: listening on ed0 Ctrl t tcpdump: 24 packets received by filter, 0 packets dropped by kernel Ctrl c 33 packets received by filter 0 packets dropped by kernel

You will lose your prom pt for t he durat ion of t he dum p, and capt ured packet s will be displayed t o your t erm inal ( t hese weren't shown in t his exam ple's out put ) . I f you press Ct rlt , you can see how m any packet s have been capt ured so far and how m any have been dropped, if any. I f you're dropping packet s, t hat m eans packet s are arriving fast er t han tcpdump can process t hem . To end your sniff, press Ct rl- c and you'll ret urn t o your prom pt . Unless you're a speed reader or have a very boring net work, you'll probably prefer t o send t he capt ured packet s t o a file. Use t he -w ( writ e) swit ch t o specify t he nam e of t he file you'd like t o creat e: # tcpdump -i ed0 -w dumpfile tcpdump: listening on ed0 Ctrl t load: 0:00

cmd: tcpdump 1458 [bpf] 0.01u 0.00s 0% 1576k

Ctrl c 56 packets received by filter 0 packets dropped by kernel

- 234 -

Not e t hat you won't be able t o read t hat file wit h a pager or edit or, as it is writ t en in a form at t hat only tcpdump or anot her packet - sniffer ut ilit y can underst and. I nst ead, use t he -r ( read) swit ch and specify t he nam e of t he file: # tcpdump -r dumpfile | more

5.6.2 Display Filters I f you t ry t he previous exam ples on a m oderat ely busy net work, you'll probably rem ind yourself why you don't like using tcpdump. I n a m inut e you can capt ure hundreds of seem ingly unint elligible lines of num bers. You're wast ing t im e and brain cells if you're wading t hrough hundreds of lines and you're int erest ed in only t wo or t hree of t hem . You can save on bot h of t hose precious resources if you spend a few m inut es creat ing a display filt er. There's always a reason behind a packet sniff. tcpdump is a very int elligent ut ilit y, but it 's not a m ind reader. However, if you can convert your reason int o synt ax t hat tcpdump underst ands, you can creat e a filt er t hat will display only int erest ing packet s. Let 's say t hat you suspect broadcast packet s are slowing down a net work segm ent . This incant at ion will capt ure only broadcast packet s: # tcpdump -i ed0 broadcast

When you end your capt ure, you'll find t hat t he num ber of packet s received by t he filt er will be great er t han t he num ber of packet s displayed t o your screen. This m eans t hat tcpdump will st ill capt ure all packet s, but will display only t he packet s m at ching your filt er. This can give you a good idea of rat io. For exam ple, if you capt ured 100 packet s in a m inut e and only 4 of t hose packet s were broadcast s, t hen broadcast s probably aren't an issue on t hat net work. Next exam ple: a part icular workst at ion is having problem s connect ing t o a server. Creat e a filt er t hat zeros in on t he packet s bet ween t hose t w o syst em s, in t his case, genisis and server1: # tcpdump -i ed0 host genisis and server1

I n t his exam ple, I only have t o use t he host keyword once, as it is assum ed unt il I specify a different keyword. I f I really like t o t ype ( which I don't ) , it would have been j ust as correct t o t ype host genisis and host server1. You can also fine- t une t hat synt ax t o unidirect ional t raffic like so: # tcpdump -i ed0 src host genisis and dst host server1

That will show only t he t raffic t hat was creat ed at genesis and is dest ined for server1. This t im e I had t o repeat t he word host, as one incant at ion was src host while t he ot her was dst host. Suppose you're int erest ed in only I CMP t raffic:

- 235 -

# tcpdump -i ed0 icmp

or perhaps only ARP t raffic: # tcpdump -i ed0 arp

Perhaps you're having a problem wit h I KE, which uses UDP port 500: # tcpdump -i ed0 udp port 500

As you can see, tcpdump com es wit h m any keywords t hat assist you in creat ing a display filt er suit ed t o your needs. These keywords are building blocks for m ore com plex expressions. When you do your own com binat ions, you m ight find it easier t o use t he words and, or, and not. For exam ple, t his will capt ure all t raffic on net work 192.168.2.0 t hat is not ARP- based: # tcpdump -i ed0 net 192.168.2 and not arp

Of course, you can find all of t he keywords, along wit h exam ples, in man tcpdump. I 've highlight ed only t he m ost com m only used keywords.

5.6.3 More Complicated Filters tcpdump is capable of zeroing in on any part icular field in a packet . I n order t o harness t his power, it 's useful t o have a pict ure of t he various t ypes of headers in front of you. Once you have a pict ure of t he fields cont ained wit hin t he part icular header you're int erest ed in, t he exam ples in man tcpdump will m ake a lot m ore sense. You'll know you're creat ing a very specific filt er if your tcpdump expression cont ains t he nam e of a prot ocol followed by square bracket s ( [ ]) . Let 's t ake a look at t his exam ple from t he m anpage, which is designed t o capture only SYN- 1s, t he first packet in t he TCP t hree- way handshake. Rem em ber t hat square bracket s m ay have special m eaning t o t he shell, so quot e com plex expressions t o prevent weird synt ax errors: # tcpdump -i ed0 'tcp[13] =

= 2'

I f you're fam iliar wit h t he t hree- way handshake, you know t hat it involves t he flags field of a TCP header. Let 's find t hat part icular field wit hin t he TCP header. Figure 5- 1 shows t he header fields of a TCP packet .

- 236 -

Figu r e 5 - 1 . TCP pa ck e t h e a de r s

The num ber enclosed wit hin t he [ ] represent s how m any oct et s int o t he header a part icular field occurs. Each line, or word, of a header is 4 oct et s long. The Flags field is aft er t he first t hree words ( i.e., 12 oct et s) and occurs one m ore oct et in, j ust aft er t he Dat a Offset and Reserved fields. So, t his part icular TCP field occurs in oct et 13 and is represent ed by tcp[13]. St ill wit h m e? Okay, where'd t he = = 2 com e from ? For t hat one, you need t o know t he nam es of t he flags as well as t he decim al equivalent s for each binary bit t hat represent s a flag. These are list ed in Table 5- 4.

Ta ble 5 - 4 . TCP fla gs a n d t h e ir de cim a l e qu iva le n t s Fla g na m e

D e cim a l e qu iva le n t

URG

32

ACK

16

PSH

8

RST

4

SYN

2

FI N

1

Finally, you need t o know t hat t he first packet in t he t hree- way handshake is dist inguished by j ust t he SYN flag being t urned on. Since all of t he ot her flags will be t urned off and will t herefore cont ain a value of 0, a value of 2 in t his field indicat es t hat only t he SYN bit is enabled. I f m at h isn't your st rong point , t here is an alt ernat e way t o writ e t his part icular expression: # tcpdump -i ed0 'tcp[tcpflags] =

=tcp-syn'

I f t he part icular field you're int erest ed in happens t o be t he TCP flags field, t he I CMP t ype field, or t he I CMP code field, you're in luck. Those t hree fields are predefined, so you don't have t o count how m any oct et s in t hat field occurs in t he header. So: • •

tcp[13] is t he sam e expression as tcp[tcpflags]. icmp[1] is t he sam e expression as icmp[icmpcode].

- 237 -

•

icmp[2] is t he sam e expression as icmp[icmptype].

Again, t he m anpage list s which I CMP t ypes have predefined keywords. To specify t he ot her t ypes or t he codes, look up t he desired num ber from t he official list at ht t p: / / www.iana.org/ assignm ent s/ icm p- param et ers.

5.6.4 Deciphering tcpdump Output Okay, you've m anaged t o capt ure j ust t he packet s you're int erest ed in. Now, can you underst and your result s? Let 's look at som e sam ple lines from a dum pfile. This part icular dum p is t he first few packet s from a POP3 session: # tcpdump -r dumpfile 17:22:36.611386 arp who-has 192.168.2.100 tell genisis. 17:22:36.611642 arp reply 192.168.2.100 is-at 0:48:54:1e:2c:76

ARP packet s are fairly com prehensible. I n t his exam ple, m y ARP t able didn't cont ain an ent ry for m y default gat eway, 192.168.2.100. My syst em , genisis, sent out a request looking for t hat gat eway. The gat eway responded wit h it s MAC address, 0: 48: 54: 1e: 2c: 76. 17:22:36.620320 genisis..49570 > nscott11.bellnexxia.net.domain:

40816+

\A? pop1.sympatico.ca. (35) 17:22:36.628557 nscott11.bellnexxia.net.domain > genisis..49570:

40816

\1/4/4 A 209.226.175.83 (203) (DF)

Once ARP had sort ed out t he MAC address, a DNS lookup had t o occur. The word domain in t hese lines indicat e a DNS lookup request followed by a DNS reply. Let 's see if we can decipher bot h t he request and t he reply. Each st art s wit h a t im est am p, which is com posed of t he t im e and a random num ber, separat ed by a dot . Since m any packet s can be sent wit hin t he sam e second, t he random num ber is used t o different iat e bet ween packet s. The t wo host s are separat ed by a great er- t han sign. I f you can visualize it as an arrow, like -->, you can see t hat genisis sent t hat first packet t o nscott11.bellnexxia.net.domain. Each host nam e has an ext ra dot , followed by eit her a port num ber or a resolved port nam e. I n t his case, genisis used port 49570, and nscott11.bellnexxia.net used t he domain port . I f you com e across a port nam e you're not fam iliar wit h, look it up in / et c/ services: % grep -w domain /etc/services domain

53/tcp

#Domain Name Server

domain

53/udp

#Domain Name Server

- 238 -

The next num ber, 40816, is an I D num ber t hat is shared by bot h t he DNS client ( genisis) and t he DNS server. The client t hen asked a quest ion ( ?) regarding t he A record for pop1.sympatico.ca. The ent ire packet it self was 35 byt es long. The second packet , from t he DNS server, shared t he sam e I D num ber. I t was also a longer packet , 203 byt es, as it cont ained t he answer. See t he 1/4/4? This m eans t hat t here is one ent ry in t he answer sect ion, four ent ries in t he aut horit y sect ion, and four ent ries in t he addit ional sect ion. ( See [ H a ck # 4 7 ] for an explanat ion of t hese sect ions.) The DNS server also sent t he request ed A record, which cont ains t he request ed I P address, 209.226.175.83. Now t hat nam e resolut ion has succeeded, a packet can be sent t o t he POP3 server: 17:22:36.629268 genisis..49499 > 209.226.175.83.pop3: S \2697729992:2697729992(0) win 65535 (DF) 17:22:36.642617 209.226.175.83.pop3 > genisis..49499: S \2225396806:2225396806(0) ack 2697729993 win 25920 (DF)

This out put is m uch easier t o read if you have a pict ure of a TCP header handy, as t he out put det ails t he inform at ion found in t hat header. Each line st art s out as before: t he t im est am p, source port , >, and dest inat ion port . We t hen see an S, which refers t o t hat SYN flag. This is followed by t he sequence num ber and, alm ost always, by t he ack num ber. The only packet t hat doesn't have an ack num ber is t he SYN- 1, t he first packet in t his exam ple. This is because a SYN- 1 is t he first TCP packet , so t here is not hing t o acknowledge yet . All ot her TCP packet s aft er t he SYN- 1 will have an ack. Next com es t he window size. I f t he packet has any opt ions, t hey will be enclosed wit hin angle bracket s. Finally, t he I P header had t he " don't fragm ent " flag, DF, set . This is im port ant enough t o be print ed at t he end of any line represent ing a TCP or UDP header.

5.6.5 See Also • • • • •

man tcpdump ht t p: / / www.t cpdum p.org/ ht t p: / / www.et hereal.com / " TCP Prot ocol Layers Explained," a FreeBSD Basics colum n ( ht t p: / / www.onlam p.com / pub/ a/ bsd/ 2001/ 03/ 14/ FreeBSD_Basics.ht m l) " Exam ining I CMP Packet s," a FreeBSD Basics colum n ( ht t p: / / www.onlam p.com / pub/ a/ bsd/ 2001/ 04/ 04/ FreeBSD_Basics.ht m l)

- 239 -

Hack 47 Understand DNS Records and Tools

D e m yst ify D N S r e cor ds. DNS is one of t hose net work services t hat has t o be configured carefully and t est ed regularly. A m isconfigured DNS server can prevent t he world from finding your web and m ail servers. Worse, a m isconfigured DNS server can allow t he world t o find m ore t han j ust your web and m ail servers. Even if you're not a DNS adm inist rat or, you should st ill know som e handy DNS com m ands. The sim ple t rut h is, if DNS isn't working, you're not going anywhere. That m eans no surfing, no downloading, and no em ail for you.

5.7.1 Exploring Your ISP's DNS On your hom e syst em , you m ost likely receive your DNS inform at ion from your I SP's DHCP server. Do you know where t o find your prim ary and secondary DNS server addresses? I f not , t ry t his: % more /etc/resolv.conf search domain.org nameserver 204.101.251.1 nameserver 204.101.251.2

Anot her m et hod is t o use t he dig ( dom ain inform at ion groper) ut ilit y. Here, I 'll ask for t he nam eservers ( ns) for t he sym pat ico.ca net work: % dig ns sympatico.ca

; DiG 8.3 ns sympatico.ca ;; res options: init recurs defnam dnsrch ;; got answer: ;; ->>HEADER> important_file important_file: Operation not permitted.

Finally, I 'll t ry m oving, delet ing, and copying t hat file: % mv important_file test mv: rename important_file to test: Operation not permitted

% rm important_file override rw-r--r--

dru/wheel uchg for important_file? y

rm: important_file: Operation not permitted

% cp important_file test %

Not ice an im port ant difference bet ween t he mv and rm com m ands and t he cp com m and. Since mv and rm require a change t o t he original file it self, t hey are prevent ed by t hat unchangeable flag. However, t he cp com m and doesn't t ry t o change t he original file; it sim ply creat es a new file wit h t he sam e cont ent s. However, if you t ry ls -lo on t hat new file, t he uchg flag will not be set . This is because new files inherit perm issions and flags from t he parent direct ory. ( Okay, t hat 's not t he whole st ory. See man umask for m ore gory det ails.)

6.4.2 Watch Your Directories What do you t hink will happen if you place all of your im port ant files in a direct ory and recursively set uchg on t hat direct ory? % mkdir important_stuff

- 290 -

% cp resume important_stuff/ % chflags -R important_stuff/ % ls -lo important_stuff/ drwxr-xr-x

2 dru

wheel

uchg

drwxr-xr-x

34 dru

wheel

-

-rw-r--r--

1 dru

wheel

uchg

512 Dec

1 11:23 ./

3072 Dec

1 11:36 ../

14 Dec

1 11:13 resume

So far so good. That file inherit ed t he uchg flag from t he direct ory, so it is now prot ect ed from changes. What if I t ry t o add a new file t o t hat direct ory? % cp coverletter important_stuff cp: important_stuff/coverletter: Operation not permitted

Because t he direct ory it self is not allowed t o change, I can't add or rem ove any files from t he direct ory. I f t hat 's what you want , great . I f not , keep t hat in m ind when playing wit h direct ory flags. What if you change your m ind and really do want t o change a file? I f you own t he file, you can unset t he flag by repeat ing t he chflags com m and wit h t he no word. For exam ple: % chflags nouchg resume

will allow m e t o m ake edit s t o m y résum é. However, I won't be able t o delet e it from t hat prot ect ed direct ory unless I also use t he nouchg flag on t he im port ant _st uff direct ory.

6.4.3 Preventing Some Changes and Allowing Others Som et im es, t he uchg flag is a bit t oo drast ic. For exam ple, if you want t o be able t o edit a file but not inadvert ent ly delet e t hat file, use t his flag inst ead: % chflags uunlnk thesis %

I can now edit t hat file t o m y heart 's cont ent . However, if I t ry t o m ove or delet e t hat file, I 'll receive t hose Operation not permitted error m essages again. The uappnd flag is m ore int erest ing. I t allows you t o append changes t o a file but prevent s you from m odifying t he exist ing cont ent s. This m ight be useful for a blog: % chflags uappnd myblog %

- 291 -

Then again, it m ight not . echoing com m ent s t o t he end of t he file works nicely. However, opening it in an edit or does not . Not e t hat t his flag also prevent s you from m oving or delet ing t he file.

6.4.4 Log Protection Let 's m ove on t o t he rest of t he flags, which can be m anaged only by t he superuser. sappnd, schg, and sunlnk work exact ly t he sam e as t heir u equivalent s. So, t hink s for superuser and u for user. The append flag was a bit weird for a regular user, but it is ideal for prot ect ing t he syst em logs. One of t he first t hings an int ruder will do aft er breaking int o a syst em is t o cover up his t racks by changing or delet ing logs. This com m and will t hwart t hose at t em pt s: # chflags -R sappnd /var/log

Now is a good t im e t o m ent ion a securit y t rut h: securit y is a m yt h. I n realit y, securit y is a process of m aking t hings m ore inconvenient in t he hopes t hat a m iscreant will go elsewhere. Rem em ber, t hough, t hat inconvenience doesn't j ust affect t he bad guys; it also affect s you. That com m and seem s ideal because it allows logs t o be appended t o but not m odified or delet ed. That 's great if you live in t he world of unlim it ed disk space. Of course, it also j ust broke newsyslog, and you've j ust delegat ed yourself t he j oys of m anual log rot at ion. There's one ot her t hing you need t o consider when you st art playing wit h t he superuser flags. I f your securelevel is set t o 0 or - 1, t he superuser can unset any flag by adding no t o it . I f your at t acker has heard of flags before and has m anaged t o gain access t o t he superuser account , all of your flag set t ing was for naught . Having said t hat , suppose you're hardening a server and want t o prot ect t he logs. Your securelevel is set at 1 or higher, and you plan on using t hat previous chflags com m and. Since you're now responsible for log rot at ion, you m ight as well st art by t aking st ock of t he cont ent s of / var/ log before t urning on t hat sappnd flag. Rem ove any unnecessary logs now, before set t ing t he flag. Next , edit / et c/ cront ab and com m ent t he newsyslog line so it looks like t his: # Rotate log files every hour, if necessary. #0

*

*

*

*

root

newsyslog

Com m ent out any lines in / et c/ syslog.conf t hat refer t o logs you rem oved. You should also consider using som et hing like t he following script t o warn you if a part it ion is filling up: #!usr/local/bin/bash # checkfreespace.sh # check that a device has sufficient free space # thanks to David Lents and Arnold Robbins for awk/gawk/nawk suggestions

- 292 -

# set the following variables as necessary PARTITION="/var/log" THRESHOLD="80"

USED=$( eval "df | awk -- '\$6 =

= ENVIRON[\"PARTITION\"]

{ printf( \"%0.d\", \$5 ) }'" );

if [ "$USED" -ge $THRESHOLD ] then echo "Used space of $USED above $THRESHOLD on $PARTITION" else # disable this if running through cron echo "Enough free space" fi

I f you schedule t his program t hrough cron, it will m ail any out put t o t he user owning t he cron j ob. Edit t he t wo variables at t he t op of t he script t o change t he part it ion t o scan and t he t hreshold above which t he script will warn. Wit h t he variables set as shown, t he script will warn if / var/ log is m ore t han 80% full. Rem em ber, once you disable newsyslog, it becom es your responsibilit y t o m onit or disk space in / var/ log. You won't be able t o com press or delet e log files unless t he superuser t em porarily unset s t he sappnd flag. This can be a real pain if your securelevel is 1 or higher, as t he syst em first has t o be dropped down t o single- user m ode. This usually isn't an opt ion on busy syst em s as it will disconnect all current connect ions. Carefully consider t he size of / var/ log and how oft en t he syst em realist ically can be put int o single- user m ode before set t ing t his flag.

6.4.5 Protecting Binaries When a syst em is com prom ised, t he at t acker m ay inst all a root kit t hat will t ry t o change your syst em 's binaries. For exam ple, it m ight replace ps wit h a version t hat doesn't display t he root kit 's processes. Or, it m ight replace a com m only used ut ilit y wit h anot her program t hat execut es som et hing nast ier t han expect ed. [ H a ck # 5 8 ] shows how t o creat e your own file int egrit y checking program t hat will alert you if any of your binaries or ot her im port ant files are changed. An addit ional layer of prot ect ion is t o use chflags t o prevent t hose files from being changed in t he first place. Usually, t he schg flag is used t o prevent any m odificat ions. Useful candidat es for t his flag are:

- 293 -

• • •

/ usr/ bin, which cont ains user program s / usr/ sbin, which cont ains syst em program s / et c, which cont ains syst em configurat ions

Again, evaluat e your part icular scenario before im plem ent ing t his flag. The prot ect ion provided by t his flag usually far out weighs t he inconvenience. The only t im e t he cont ent s of / usr/ bin or / usr/ sbin should change is when you upgrade t he operat ing syst em or rebuild your world. Doing t hat requires a reboot anyway, so dropping t o single- user m ode t o unset schg shouldn't be a problem . How oft en do you change your configurat ion files in / et c? I f you t ypically configure a syst em only when it is inst alled and rarely m ake changes aft erward, prot ect your configurat ions wit h schg. However, keep in m ind t hat a rare configurat ion change m ay require you t o drop all connect ions in order t o im plem ent it . Also, if you need t o add m ore users t o your syst em , rem em ber t o rem ove t hat flag from / et c/ passwd, / et c/ m ast er.passwd, and / et c/ group first . Things are a bit m ore problem at ic for a syst em running inst alled applicat ions. Most port s inst all t heir binaries int o / usr/ local/ bin or / usr/ X11R6/ bin. I f you set t he schg flag on t hose direct ories, you won't be able t o pat ch or upgrade t hose binaries unless you t em porarily unset t he flag. You'll have t o balance your need t o keep your server up and running wit h t he prot ect ion you gain from t he schg flag and how oft en you have t o pat ch a part icular binary.

6.4.6 Controlling Backups The last t wo flags, arch and nodump, affect backups. The superuser can ensure a part icular file or direct ory will always be backed up, regardless of whet her t he cont ent s have been alt ered, by set t ing t he arch flag. Sim ilarly, when using dump t o back up an ent ire filesyst em , t he superuser can specify which port ions of t hat filesyst em will not be included by set t ing t he nodump flag.

6.4.7 See Also • • • •

man securelevel man -a chflags ( t o view all m anpages t hat m at ch chflags, not j ust t he first one) man newsyslog [ H a ck # 5 8 ]

- 294 -

Hack 57 Tighten Security with Mandatory Access Control

I n cr e a se t h e se cu r it y of you r syst e m s w it h M AC pa r a noia . Ever feel like your Unix syst em s are leaking out ext ra unsolicit ed inform at ion? For exam ple, even a regular user can find out who is logged int o a syst em and what t hey're current ly doing. I t 's also an easy m at t er t o find out what processes are running on a syst em . For t he securit y- m inded, t his m ay be t oo m uch inform at ion in t he hands of an at t acker. Fort unat ely, t hanks t o t he Trust edBSD proj ect , t here are m ore t ools available in t he adm in's arsenal. One of t hem is t he Mandatory Access Cont rol ( MAC) fram ework. As of t his writ ing, FreeBSD's MAC is st ill considered experim ent al for product ion syst em s. Thoroughly t est your changes before im plem ent ing t hem on product ion servers.

6.5.1 Preparing the System Before you can im plem ent Mandat ory Access Cont rol, your kernel m ust support it . Add t he following line t o your kernel configurat ion file: options MAC

You can find full inst ruct ions for com piling a kernel in [ H a ck # 5 4 ] . While your kernel is recom piling, t ake t he t im e t o read man 4 mac, which list s t he available MAC m odules. Som e of t he current m odules support sim ple policies t hat can cont rol an aspect of a syst em 's behavior, whereas ot hers provide m ore com plex policies t hat can affect every aspect of syst em operat ion. This hack dem onst rat es sim ple policies designed t o address a single problem .

6.5.2 Seeing Other Users One problem wit h open source Unix syst em s is t hat t here are very few secret s. For exam ple, any user can run ps -aux t o see every running process or run sockstat -4 or netstat -an t o view all connect ions or open socket s on a syst em . The MAC_SEEOTHERUIDS m odule addresses t his. You can load t his kernel m odule m anually t o experim ent wit h it s feat ures: # kldload mac_seeotheruids Security policy loaded: TrustedBSD MAC/seeotheruids (mac_seeotheruids)

- 295 -

I f you'd like t his m odule t o load at boot t im e, add t his t o / boot / loader.conf: mac_seeotheruids_load="YES"

I f you need t o unload t he m odule, sim ply t ype: # kldunload mac_seeotheruids Security policy unload: TrustedBSD MAC/seeotheruids (mac_seeotheruids)

When t est ing t his m odule on your syst em s, com pare t he before and aft er result s of t hese com m ands, run as bot h a regular user and t he superuser: • • • •

ps -aux netstat -an sockstat -4 w

Your before result s should show processes and socket s owned by ot her users, whereas t he aft er result s should show only t hose owned by t he user. While t he out put from w will st ill show which users are on which t erm inals, it will not display what ot her users are current ly doing. By default , t his m odule affect s even t he superuser. I n order t o change t hat , it 's useful t o know which sysctl MI Bs cont rol t his m odule's behavior: # sysctl -a | grep seeotheruids security.mac.seeotheruids.enabled: 1 security.mac.seeotheruids.primarygroup_enabled: 0 security.mac.seeotheruids.specificgid_enabled: 0 security.mac.seeotheruids.specificgid: 0 sysctl is used t o m odify kernel behavior wit hout having t o recom pile t he kernel or reboot t he syst em . The behaviors t hat can be m odified are known as MI Bs.

See how t here are t wo MI Bs dealing wit h specificgid? The enabled one is off, and t he ot her one specifies t he num eric group I D t hat would be exem pt if it were on. So, if you do t his: # sysctl -w security.mac.seeotheruids.specificgid_enabled=1 security.mac.seeotheruids.specificgid_enabled: 0 -> 1

you will exem pt group 0 from t his policy. I n FreeBSD, t he wheel group has a GI D of 0, so users in t he wheel group will see all processes and socket s.

- 296 -

You can also set t hat primarygroup_enabled MI B t o 1 t o allow users who share t he sam e group I D t o see each ot her's processes and socket s. Not e t hat while you can change t hese MI Bs from t he com m and line, you will be able t o see t hem only wit h t he appropriat e kernel m odule loaded.

6.5.3 Quickly Disable All Interfaces ifconfig allows you t o enable and disable individual int erfaces as required. For exam ple, t o st op t raffic on ed0: # ifconfig ed0 down

To bring t he int erface back up, sim ply repeat t hat com m and, replacing t he word down wit h up. However, ifconfig does not provide a convenient m et hod for st opping or rest art ing t raffic flow on all of a syst em 's int erfaces. That abilit y can be quit e convenient for t est ing purposes or t o quickly rem ove a syst em from a net work t hat is under at t ack. The MAC_IFOFF m odule is a bet t er t ool for t his purpose. Let 's load t his m odule and see how it affect s t he syst em : # kldload mac_ifoff Security policy loaded: TrustedBSD MAC/ifoff (mac_ifoff) # sysctl -a | grep ifoff security.mac.ifoff.enabled: 1 security.mac.ifoff.lo_enabled: 1 security.mac.ifoff.other_enabled: 0 security.mac.ifoff.bpfrecv_enabled: 0

By default , t his m odule disables all int erfaces, except t he loopback lo device. When it 's safe t o reenable t hose int erfaces, you can eit her unload t he m odule: # kldunload mac_ifoff Security policy unload: TrustedBSD MAC/ifoff (mac_ifoff)

or leave t he m odule loaded and enable t he int erfaces: # sysctl -w security.mac.ifoff.other_enabled=1 security.mac.ifoff.other_enabled: 0 -> 1

Perhaps you have a syst em whose int erfaces you'd like t o disable at boot up unt il you explicit ly enable t hem . I f t hat 's t he case, add t his line t o / boot / loader.conf: mac_ifoff_load="YES"

- 297 -

6.5.4 See Also • • • • • • •

man 4 mac man mac_seeotheruids man mac_ifoff man sysctl The Trust edBSD proj ect ( ht t p: / / www.t rust edbsd.org/ ) The sysctl sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ configt uningsysct l.ht m l) The MAC sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ m ac.ht m l)

- 298 -

Hack 58 Use mtree as a Built-in Tripwire

W h y con figu r e a t h ir d- pa r t y file in t e gr it y ch e ck e r w h e n you a lr e a dy ha ve m t r e e ? I f you care about t he securit y of your server, you need file int egrit y checking. Wit hout it , you m ay never know if t he syst em has been com prom ised by a root kit or an act ive int ruder. You m ay never know if your logs have been m odified and your ls and ps ut ilit ies replaced by Troj aned equivalent s. Sure, you can download or purchase a ut ilit y such as tripwire, but you already have t he m t ree ut ilit y [ H a ck # 5 4 ] ; why not use it t o hack your own cust om ized file int egrit y ut ilit y? mtree list s all of t he files and t heir propert ies wit hin a specified direct ory st ruct ure. That result ing list is known as a specificat ion. Once you have a specificat ion, you can ask mtree t o com pare it t o an exist ing direct ory st ruct ure, and mtree will report any differences. Doesn't t hat sound like a file int egrit y checking ut ilit y t o you?

6.6.1 Creating the Integrity Database Let 's see what happens if we run mtree against / usr/ bin: # cd /usr/bin # mtree -c -K cksum,md5digest,sha1digest,ripemd160digest -s 123456789 \ > /tmp/mtree_bin mtree: /usr/bin checksum: 2126659563

Let 's pick apart t hat synt ax in Figure 6- 2.

Ta ble 6 - 2 . m t r e e com m a nd syn t a x Com m a nd

Ex pla na t ion

-c

This creat es a specificat ion of t he current working direct ory.

-K

This specifies a keyword. I n our case, it 's cksum.

md5digest, sha1digest,ripemd160digest

Here, I 've specified t he t hree crypt ographic checksum s underst ood by mtree. This is how it det ect s file m odificat ions: any change t o a file will result in a different hash. While it m ay be m at hem at ically feasible for an at t acker t o bypass one crypt ographic hash, it 's darn near im possible for her t o bypass all t hree crypt ographic hashes.

-s

This gives t he num eric seed t hat is used t o creat e t he specificat ion's checksum . Rem em ber t hat seed t o verify t he specificat ion.

- 299 -

Ta ble 6 - 2 . m t r e e com m a nd syn t a x Com m a nd

Ex pla na t ion This redirect s t he result s t o t he file / t m p/ m t ree_bin inst ead of st dout .

>

I f you run t hat com m and, it will perk along for a second or t wo, t hen writ e t he value of t he checksum t o your screen j ust before giving your prom pt back. That 's it ; you've j ust creat ed a file int egrit y dat abase. Before we t ake a look at t hat dat abase, t ake a m om ent t o record t he seed you used and t he checksum you received. Not e t hat t he m ore com plex t he seed, t he harder it is t o crack t he checksum . Those t wo num bers are im port ant , so you m ay consider writ ing t hem on a sm all piece of paper and st oring t hem in your wallet . ( Don't forget t o include a hint t o rem ind you why you have t hat piece of paper in your wallet ! ) Now let 's see what t ype of file we've j ust creat ed: # file /tmp/mtree_bin /tmp/mtree_bin: ASCII text

# ls -l /tmp/mtree_bin -rw-r--r--

1 root

wheel

111503 Nov 23 11:46 /tmp/mtree_bin

I t 's an ASCI I t ext file, m eaning you can edit it wit h an edit or or print it direct ly. I t 's also fairly large, so let 's use head t o exam ine t he first bit of t his file. Here I 'll ask for t he first 15 lines: # head -n 15 /tmp/mtree_bin #

user: dru

#

machine: genisis

#

tree: /usr/bin

#

date: Sun Nov 23 11:46:21 2003

# . /set type=file uid=0 gid=0 mode=0555 nlink=1 flags=none .

type=dir mode=0755 nlink=2 size=6656 time=1065005676.0 CC

nlink=3 size=78972 time=1059422866.0 cksum=1068582540 \ md5digest=b9a5c9a92baf9ce975eee954994fca6c \ sha1digest=a2e4fa958491a4c2d22b7f597f05885bbe8f6a6a \ ripemd160digest=33c74b4200c9507b4826e5fc8621cddb9e9aefe2

- 300 -

Mail

nlink=3 size=72964 time=1059422992.0 cksum=2235502998 \ md5digest=44739ae79f3cc89826f6e34a15f13ed7 \ sha1digest=a7b89996ffae4980ad87c6e7c56cb207af41c1bd \

The specificat ion st art s wit h a nice sum m ary sect ion. I n m y exam ple, t he user t hat creat ed t he specificat ion was dru. Not e t hat I used t he su ut ilit y t o becom e t he superuser before creat ing t he specificat ion, but m y login shell knew t hat I was st ill logged in as t he user dru. The sum m ary also shows t he syst em nam e, genisis, t he direct ory st ruct ure in quest ion, / usr/ bin, and t he t im e t he specificat ion was creat ed. The /set type=file line shows t he inform at ion mtree records by default . Not ice t hat it keeps t rack of each file's uid, gid, m ode, num ber of hard links, and flags. Then, each file and subdirect ory in / usr/ bin is list ed one at a t im e. Since I used -K t o specify t hree different crypt ographic hashes, each file has t hree separat e hashes or digest s.

6.6.2 Preparing the Database for Storage Once you've creat ed a specificat ion, t he last place you want t o leave it is on t he hard drive. I nst ead, sign t hat file, encrypt it , t ransfer it t o a different m edium ( such as a floppy) , and place it in a secure st orage area. To sign t he file: # md5 /tmp/mtree_bin MD5 (/tmp/mtree_bin) = e05bab7545f7bdbce13e1bb04a043e47

You m ay wish t o redirect t hat result ing fingerprint t o a file or a print er. Keep it in a safe place, as you'll need it t o check t he int egrit y of t he dat abase. Next , encrypt t he file. Rem em ber, right now it is in ASCI I t ext and suscept ible t o t am pering. Here I 'll encrypt t he file and send t he newly encrypt ed file t o t he floppy m ount ed at / floppy: # openssl enc -e -bf -in /tmp/mtree_bin -out /floppy/mtree_bin_enc enter bf-cbc encryption password: Verifying - enter bf-cbc encryption password:

The synt ax of t he openssl com m and is fairly st raight forward. I decided t o encrypt enc -e wit h t he Blowfish -bf algorit hm . I t hen specified t he input file, or t he file t o be encrypt ed. I also specified t he out put file, or t he result ing encrypt ed file. I was t hen prom pt ed for a password; t his sam e password will be required whenever I need t o decrypt t he dat abase. Once I verify t hat t he encrypt ed file is indeed on t he floppy, I m ust rem em ber t o rem ove t he ASCI I t ext version from t he hard drive: # rm /tmp/mtree_bin

- 301 -

The ult ra- paranoid, experienced hacker would zero out t hat file before rem oving it using dd if=/dev/zero of=/tmp/mtree_bin bs=1024k count=12.

I 'll t hen st ore t he floppy in a secure place, such as t he safe t hat cont ains m y backup t apes.

6.6.3 Using the Integrity Database Once you have an int egrit y dat abase, you'll want t o com pare it periodically t o t he files on your hard drive. Mount t he m edia cont aining your encrypt ed dat abase, and t hen decrypt it : # openssl enc -d -bf -in /floppy/mtree_bin_enc -out /tmp/mtree_bin enter bf-cbc encryption password:

Not ice t hat I used basically t he sam e com m and I used t o encrypt it . I sim ply replaced t he encrypt swit ch ( -e) wit h t he decrypt swit ch ( -d) . The encrypt ed file is now t he input , and t he plain t ext file is now t he out put . Not e t hat I was prom pt ed for t he sam e password; if I forget it , t he decrypt ion will fail. Before using t hat dat abase, I first want t o verify t hat it s fingerprint hasn't been t am pered wit h. Again, I sim ply repeat t he md5 com m and. I f t he result ing fingerprint is t he sam e, t he dat abase is unm odified: # md5 /tmp/mtree_bin MD5 (/tmp/mtree_bin) = e05bab7545f7bdbce13e1bb04a043e47

Next , I 'll see if any of m y files have been t am pered wit h on m y hard drive: # cd /usr/bin # mtree -s 123456789 < /tmp/mtree_bin mtree: /usr/bin checksum: 2126659563

I f none of t he files have changed in / usr/ bin, t he checksum will be t he sam e. I n t his case it was. See why it was im port ant t o record t hat seed and checksum ? What happens if a file does change? I haven't built world on t his syst em in a while, so I suspect I have source files t hat haven't m ade t heir way int o / usr/ bin yet . Aft er som e poking about , I not ice t hat / usr/ src/ usr.bin has a bluet oot h direct ory cont aining t he source for a file called bt sockst at . I 'll inst all t hat binary: # cd /usr/src/usr.bin/bluetooth/btsockstat # make # make install

# ls -F /usr/bin | grep btsockstat

- 302 -

btsockstat*

Now let 's see if mtree not ices t hat ext ra file: # cd /usr/bin # mtree -s 123456789 < /tmp/mtree_bin . changed modification time expected Wed Oct

1 06:54:36 2003

found Sun Nov 23 16:10:32 2003 btsockstat extra mtree: /usr/bin checksum: 417306521

Well, it didn't fool mtree. That out put is act ually quit e useful. I know t hat btsockstat was added as an extra file, and I know t he dat e and t im e it was added. Since I added t hat file m yself, it is an easy m at t er t o resolve. I f I hadn't and needed t o invest igat e, I have a t im e t o assist m e in m y research. I could t alk t o t he adm inist rat or who was responsible at t hat dat e and t im e, or I could see if t here were any net work connect ions logged during t hat t im e period. Also not e t hat t his addit ion result ed in a new checksum . Once t he changes have been resolved, I should creat e a new dat abase t hat represent s t he current st at e of / usr/ bin. To recap t he necessary st eps: 1. 2. 3. 4.

Use mtree -c t o creat e t he dat abase. Use md5 t o creat e a fingerprint for t he dat abase. Use openssl t o encrypt t he dat abase. Move t he dat abase t o a rem ovable m edia, and ensure no copies rem ain on disk.

6.6.4 Deciding on Which Files to Include When you creat e your own int egrit y dat abase, ask yourself, " Which files do I want t o be aware of if t hey change?" The answer is usually your binaries or applicat ions. Here is a list of com m on binary locat ions on a FreeBSD syst em : • • • • • • • •

/ bin / sbin / usr/ bin / usr/ sbin / usr/ local/ bin / usr/ X11R6/ bin / usr/ com pat / linux/ bin / usr/ com pat / linux/ sbin

The sbin direct ories are especially im port ant because t hey cont ain syst em binaries. Most port s will inst all t o / usr/ local/ bin or / usr/ X11R6/ bin. The second quest ion t o ask yourself is " How oft en should I check t he dat abase?" The answer will depend upon your circum st ances. I f t he m achine is a publicly accessible server, you m ight consider t his as part of your daily m aint enance plan. I f t he syst em 's soft ware

- 303 -

t ends t o change oft en, you'll also want t o check oft en, while you can st ill rem em ber when you inst alled what soft ware.

6.6.5 See Also •

man mtree

- 304 -

Hack 59 Intrusion Detection with Snort, ACID, MySQL, and FreeBSD

H ow t h e a le r t a dm in ist r a t or ca t ch e s t h e w or m . Wit h t he current clim at e of corporat e force reduct ions and t he onslaught of new, fast spreading viruses and worm s, t oday's adm inist rat ors are faced wit h a daunt ing challenge. Not only is t he adm inist rat or required t o fix problem s and keep t hings running sm oot hly, but in som e cases he is also responsible for keeping t he net work from becom ing worm food. This oft en ent ails m onit oring t he t raffic going t o and from t he net work, ident ifying infect ed nodes, and loading num erous vendor pat ches t o fix associat ed vulnerabilit ies. To get a bet t er handle on t hings, you can deploy an I nt rusion Det ect ion Syst em ( I DS) on t he LAN t o alert you t o t he exist ence of all t he nast iness associat ed wit h t he dark side of t he com put ing world. This hack will show you how t o im plem ent a very effect ive and st able I DS using FreeBSD, MySQL, Snort , and t he Analysis Console for I nt rusion Dat abases ( ACI D) . While t hat m eans inst alling and configuring a few applicat ions, you'll end up wit h a feat ure- rich, searchable I DS capable of generat ing cust om alert s and displaying inform at ion in m any cust om izable form at s.

6.7.1 Installing the Software We'll assum e t hat you already have FreeBSD 4.8- RELEASE or newer inst alled wit h plent y of disk space. The syst em is also fully pat ched and t he port s collect ion is up- t o- dat e. I t also helps t o be fam iliar wit h FreeBSD and MySQL com m ands. 6 .7 .1 .1 I n st a ll PH P4 , Apa ch e , a n d M ySQL We'll st art by inst alling PHP4, Apache, and t he MySQL client . As t he superuser: # cd /usr/ports/www/mod_php4 # make install clean

When t he PHP configuration options screen appears, choose t he GD Library Support opt ion. Leave t he ot her default select ions, and choose OK. The build it self will t ake a while because it m ust inst all Apache and t he MySQL client in addit ion t o PHP. 6 .7 .1 .2 I n st a ll M ySQL- se r ve r You'll also need t he MySQL server, which is a separat e port . To ensure t his port inst alls correct ly, t em porarily set t he syst em host nam e t o localhost:

- 305 -

# hostname localhost

# cd /usr/ports/databases/mysql40-server # make install clean

This one will also t ake a while. 6 .7 .1 .3 M or e in st a lla t ion s There are a few ot her port s t o inst all. The next t hree applicat ions are used by ACI D t o creat e graphs of t he out put . ACI D support s bar graphs ( as shown in Figure 6- 3) , line graphs ( Figure 6- 4) , and pie chart s ( Figure 6- 5) .

Figu r e 6 - 3 . An ACI D ba r gr a ph

Figu r e 6 - 4 . An ACI D lin e gr a ph

- 306 -

Figu r e 6 - 5 . An ACI D pie ch a r t

We'll need adodb , a dat abase library for PHP: # cd /usr/ports/databases/adodb # make install clean

PHPlot adds a graph library t o PHP so it will support chart s: # cd /usr/ports/graphics/phplot # make install clean

- 307 -

JPGraph adds m ore support t o PHP for graphs: # cd /usr/ports/graphics/jpgraph # make install clean

Finally, we m ust inst all ACI D and Snort . St art by m odifying snort's Makefile t o include MySQL support : # cd /usr/local/ports/security/snort # vi Makefile

Change: CONFIGURE_ARGS= --with-mysql=no

t o: CONFIGURE_ARGS= --with-mysql=yes

Save your changes and exit . Finally, inst all acid, which will also inst all snort using your m odified Makefile: # cd /usr/ports/security/acid # make install clean

6.7.2 Configuring Now t hat we've inst alled all t he necessary pieces for our I DS, it 's t im e t o configure t hem t o work t oget her. 6 .7 .2 .1 Con figu r e Apa ch e a nd PH P You'll need t o m ake t wo changes t o Apache's configurat ion file, / usr/ local/ et c/ apache/ ht t pdconf. First , search for #ServerName, rem ove t he hash m ark ( #) , and change www.example.com t o your act ual server nam e. Then, for securit y reasons, change ServerSignature On t o ServerSignature Off. This prevent s t he server from providing inform at ion such as HTTP server t ype and version. Most adm ins who run I DSs on t heir net works like t o keep t heir presence som ewhat hidden, since t here are exploit s/ t ools writ t en t o defeat I DS det ect ion. 6 .7 .2 .2 Con figu r e PH P Aft er inst alling PHP, you will not ice t wo sam ple configurat ion files in / usr/ local/ et c, php.inidist and php.ini- recom m ended. As t he nam e suggest s, t he lat t er is t he recom m ended PHP

- 308 -

4- st yle configurat ion file. I t cont ains set t ings t hat m ake PHP " m ore efficient , m ore secure, and [ encourage] cleaner coding." Since our focus is securit y, I recom m end using t his file. Configuring PHP is as sim ple as copying t he sam ple configurat ion file t o / usr/ local/ et c/ php.ini: # cd /usr/local/etc # cp php.ini-recommended php.ini

6 .7 .2 .3 Con figu r e M ySQL MySQL support s several configurat ions. Use m y- sm all.cnf or m y- m edium .cnf if you have less t han 64 M of m em ory, m y- large.cnf if you have 512 M, and m y- huge.cnf if you have 12 G of m em ory. Lat er, if you find your syst em running out of swap space, you can st op mysql and copy one of t he sm aller * .cnf files t o fix t he problem . I n m y exam ple, I 'll copy over m y- large.cnf: # cp /usr/local/share/mysql/my-large.cnf /etc/my.cnf

Next , set up t he init ial dat abases and inst all t he server: # /usr/local/bin/mysql_install_db # /usr/local/etc/rc.d/mysql-server.sh start

You can use t he sockstat com m and t o confirm t hat t he MySQL server is running. You should see MySQL list ening on port 3306: # sockstat | grep mysql

USER

COMMAND

PID

FD PROTO

LOCAL ADDRESS

FOREIGN ADDRESS

mysql

mysqld

16262 5

tcp4

*:3306

*:*

mysql

mysqld

16262 6

stream /tmp/mysql.sock

Then, set t he password for t he root MySQL user. You'll have t o use t he FLUSH PRIVILEGES com m and t o t ell MySQL t o reload all of t he user privileges, or t he server will cont inue using t he old ( blank) password unt il it rest art s: # /usr/local/bin/mysql -u root

Welcome to the MySQL monitor.

Commands end with ; or \g.

Your MySQL connection id is 1 to server version: 4.0.16-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

- 309 -

mysql>SET PASSWORD FOR root@localhost=PASSWORD(' your_password_here '); mysql>FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec)

Then, you can creat e t he snort dat abase: mysql>CREATE DATABASE snort; Query OK, 1 row affected (0.00 sec)

Now we can creat e a MySQL user wit h sufficient perm issions t o access t he new snort dat abase. Do not use t he MySQL root user! By creat ing a new user who has access t o only one dat abase, we've lim it ed t he dam age an at t acker could do if he ever gained access t o t his account . MySQL uses t he GRANT com m and t o give users access t o dat abases. You can cont rol which t ypes of st at em ent s t he user can issue, as well as t he net work host s from which t he user can access MySQL. localhost is a nice, safe set t ing, as we only need t o access t he dat abase from t he local m achine. Again, t his rest rict s t he dam age t hat an at t acker could do from anot her com prom ised host . mysql> GRANT INSERT,SELECT ON snort.* to snort_user_here @localhost \ IDENTIFIED BY ' snort_users_password '; Query OK, 0 rows affected (0.00 sec)

mysql> GRANT INSERT,SELECT,CREATE,DELETE on snort.* \ to snort_user_here @localhost IDENTIFIED BY ' snort_users_password '; Query OK, 0 rows affected (0.01 sec)

- 310 -

mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec)

mysql> quit Bye

6 .7 .2 .4 Con figu r e Snor t First you'll need t o download t he lat est sources from ht t p: / / www.snort .org ( current ly v2.0.5) . Aft er unpacking, use t he create_mysql file t o creat e t he necessary t ables in t he snort dat abase. That 's all t he configurat ion you need; you can now sim ply delet e t he unpacked direct ory. # tar xvfz snort-2.0.5.tar.gz # cd snort-2.0.5/contrib # cp create_mysql /tmp # /usr/local/bin/mysql -p < /tmp/create_mysql snort Enter password:

Enter the MySQL root password here

# cd /usr/local/etc # cp snort.conf-sample snort.conf # vi snort.conf

Scroll down unt il you reach t he # output database: log, mssql, dbname=snort user=snort password=test line. I nsert t he following lines beneat h it : output database: log, mysql, user=mysql_user_name password=mysql_users_ password dbname=snort host=localhost output database: alert, mysql, user=mysql_user_name password=mysql_users_ password dbname=snort host=localhost

Now page down t oward t he bot t om of t he file and select t he t ypes of rules you want t o m onit or for. Keep in m ind t hat t he m ore rules you use, t he m ore work snort will have t o do, using up CPU cycles and m em ory t hat m ight be bet t er used elsewhere. For exam ple, if you don't want t o m onit or X11 or Oracle on any com put er on your net work, com m ent out t hose rules. When you're done, save your changes and exit . Finish by creat ing t he snort log direct ory:

- 311 -

# cd /var/log # mkdir snort

6 .7 .2 .5 Con figu r e ACI D St art by t ight ening t he perm issions of t he configurat ion file: # chmod 644 /usr/local/www/acid/acid_conf.php Have a good read t hrough t he Securit y sect ion of / usr/ local/ www/ acid/ README when you're configuring ACI D. I t cont ains m any good point ers t o ensure your configurat ion is secure.

Then, change t he sect ion t hat cont ains alert_dbname = "snort_log"; t o include t he appropriat e ent ries: $alert_dbname

= "snort";

$alert_host

= "localhost";

$alert_port

= "";

$alert_user

= "mysql_snort_user";

$alert_password = "mysql_snort_users_password";

Leave t he Archive param et ers alone, unless you want t o creat e a separat e dat abase for snort t o st ore archived alert m essages in. To do t his, you'll need t o log int o MySQL, creat e an archive dat abase, set t he appropriat e perm issions, and run t he mysql_create script again as described earlier. The Snort and ACI D docum ent at ion describe t his in m ore det ail. You do need t o t ell ACI D where t o find som e of t he libraries inst alled earlier. I n part icular, change: $ChartLib_path = "";

t o: $ChartLib_path = "/usr/local/share/jpgraph";

6.7.3 Running ACID I t 's t im e t o st art Apache: # /usr/local/sbin/apachectl start /usr/local/sbin/apachectl start: httpd started

- 312 -

Then, link t he ACI D web direct ory. Of course, for securit y reasons, I recom m end giving t he link nam e som et hing ot her t han acid. # cd /usr/local/www/ # ln -s /usr/local/www/acid /usr/local/www/snort

Point your web browser t o ht t p: / / localhost / snort / acid_m ain.php and click t he Set up link. Click t he Creat e ACI D AG but t on t o creat e t he ext ended t ables t hat ACI D will use. When it finishes, you should see som et hing sim ilar t o t he following: Successfully created 'acid_ag'

Successfully created 'acid_ag_alert'

Successfully created 'acid_ip_cache'

Successfully created 'acid_event'

Now click t he Main page link t o be t aken t o ACI D's m ain display page. At t his point you m ight ask, " Where are t he alert s?" There aren't any—we didn't st art snort!

6.7.4 Running Snort First , t ry st art ing snort m anually t o m ake sure it works. Use t he -i swit ch t o specify t he net work int erface t hat will be m onit oring t raffic. I n m y case, it is xl0. # cd /usr/local/etc # /usr/local/bin/snort -c snort.conf -i xl0 database: using the "alert" facility 1458 Snort rules read... 1458 Option Chains linked into 146 Chain Headers 0 Dynamic rules +++++++++++++++++++++++++++++++++++++++++++++++++++

Rule application order: ->activation->dynamic->alert->pass->log

--=

= Initialization Complete =

=--

-*> Snort! /dev/null Password:Password:

Here we have no cached credent ials, so sudo prom pt s us for our password. But since t here are t wo sudo com m ands in t he pipeline, we get t wo password prom pt s, one right aft er t he ot her. When we ent er our password and press Ret urn, not hing happens—our cursor st ays put on t he next line. We are act ually at t he second password prom pt , but t here is no indicat ion of t his. Ent ering our password again will get us out of t he m yst eriously hung pipeline.

6.9.2 sudo Configuration Gotchas sudo is very flexible. The / usr/ local/ et c/ sudoers file has rich sem ant ics t o im plem ent a nearly infinit e set of policies t hat can range from very open t o very rest rict ive. Of course, open policies are easier t o underst and and im plem ent t han t he rest rict ive ones, because t here are so m any ways t o subvert m any seem ingly rest rict ive policies.

- 323 -

The earlier exam ples of sudo lim it at ions assum ed t hat all t he com m ands used were aut horized for our use in t he sudoers file. However, bot h cat and tee are dangerous com m ands t hat could allow a user t o easily t ake cont rol of a syst em . ( Consider sudo tee /etc/spwd.db < myevilspwd.db.) This underlines t he generic risk of enabling com m ands wit h sudo. I t is difficult t o analyze all t he possible ways a part icular com m and could be m isused t o subvert a closed securit y policy. The m ore com m ands you enable wit h sudo, t he harder t his t ask becom es. I n general, beware of com m ands t hat are capable of m odifying files, such as edit ors, dd, cat, and tee, or t hose t hat allow shells t o be run from wit hin t hem , such as emacs and vi. vim provides an rvim variant t hat disallows shell escapes. This variant is inst alled t o / usr/ local/ bin/ rvim when you build t he port / usr/ port s/ edit ors/ vim .

You can t ry rest rict ing what argum ent s can be given t o dangerous com m ands, but beware of alt ernat e m et hods for supplying t hose argum ent s. For exam ple, t he following configurat ion ent ry recent ly cam e up on t he sudo- users m ailing list : Cmnd_Alias

PASSWD

= /usr/bin/passwd, !/usr/bin/passwd root

This works great if t he user t ypes passwd root: % sudo passwd root Sorry, user test is not allowed to execute '/usr/bin/passwd root' as root on ****.

Consider, t hough: % sudo passwd -l root Changing local password for root New Password:

Oops! The addit ion of t he -l flag causes t he pat t ern in t he sudoers file not t o m at ch t he equivalent com m and. The m oral is: t o rest rict param et ers in sudoers, you m ust disallow all perm ut at ions of argum ent s and swit ches t hat you deem undesirable. man sudoers warns about anot her danger: It is generally not effective to "subtract" commands from ALL using the '!' operator.

A user can trivially circumvent this by copying the

desired command to a different name and then executing that. ple:

- 324 -

For exam-

bill

ALL = ALL, !SU, !SHELLS

Doesn't really prevent bill from running the commands listed in SU or SHELLS since he can simply copy those commands to a different name, or use a shell escape from an editor or other program.

Therefore, these

kind of restrictions should be considered advisory at best (and reinforced by policy).

6.9.3 Shell Access with sudo Aut horizing shell access wit h sudo obviously opens your securit y policy t o t he largest possible ext ent , since any available com m and can t hen be run in t he root - enabled shell. This m ay be exact ly what you want , but you also lose sudo's audit t rail, since subsequent com m ands issued from t he shell are not logged. One way t o allow shell access t o t rust ed user s wit hout losing t he audit t rail is t o use sudoscript [ H a ck # 6 2 ] .

6.9.4 See Also • • • • •

man man man The The

sudo sudoers passwd sudo web sit e ( ht t p: / / www.court esan.com / sudo/ ) Sudo- users m ailing list archive ( ht t p: / / www.sudo.ws/ piperm ail/ sudo- users/ )

- 325 -

Hack 62 sudoscript

su do ca n he lp e nfor ce st r ict se cu r it y policie s, bu t w ha t a bou t sit u a t ion s in w h ich you don 't w a n t t o r e st r ict w ha t com m a n ds you r u se r s r u n ? Maybe you're looking for a way t o keep t rack of what your sysadm in t eam does as root, so you can quickly find out what happened when som et hing goes wrong. Even if you're t he only adm inist rat or, it 's possible t o m ake a bad error as root wit hout realizing it . An audit t rail allows you t o go back and see exact ly what you did t ype during t hat 3: 00 AM hacking session. As m ent ioned in [ H a ck # 6 1 ] , giving access t o a shell wit h sudo m eans t hat you lose your audit t rail t he m om ent t he root shell execut es. One answer t o t his problem is sudoscript. Anot her scenario where sudoscript is useful is one sim ilar t o t he sit uat ion t hat caused m e t o writ e sudoscript in t he first place. I was a sysadm in in a sm all st art up whose engineers all had t he root password. The I T crew all used sudo, but t hey had t ried wit hout success t o convince t he engineers t o use it . Upon invest igat ion, I discovered t hat t he principal reason for t his was t he prohibit ion on running shells wit h sudo. I n fact , t he sysadm ins used t he " everyt hing- but - shells" m et hod t he sudoers m anpage warns against [ H a ck # 6 1 ] .

I t quickly becam e clear t hat I wasn't going t o be able t o argue t hat sudo, as im plem ent ed, was equivalent t o having a root shell; posit ions had hardened long before I showed up. So, I wrot e sudoscript t o bring t hese engineers back int o t he I T depart m ent 's support ed circle. I t worked, and having t he audit t rail saved m y bacon several t im es.

6.10.1 sudoscript Overview sudoscript is a pair of Perl script s. One is called sudoshell , or j ust ss. Cont rary t o it s nam e, sudoshell is not a shell like tcsh or bash. I nst ead, it is a front end script t hat uses aut horizat ion from sudo t o run as root and runs script(1) on a FI FO ( nam ed pipe) m anaged by t he second script . That script is a daem on, called sudoscriptd . I t t akes dat a from t he FI FO opened by sudoscript and t ags it wit h t he user's nam e, PI D, and a t im est am p before writ ing it t o a log file. This log file, / var/ log/ sudoscript , is m anaged by t he daem on and rot at ed if it s size exceeds 2 MB. The effect of all t his is a root shell t hat saves it s t erm inal input and out put in a log file. FreeBSD provides sudoscript in t he port s collect ion in / usr/ port s/ securit y/ sudoscript . Download OpenBSD and Net BSD port s from ht t p: / / egbok.com / sudoscript / .

- 326 -

6.10.2 Is sudoscript Secure? The answer is yes and no. The answer is " yes" because sudoscript doesn't confer any privilege of it s own; it relies on sudo for t hat . For t hat reason, program m ing or archit ect ure errors in sudoscript ( which I have worked hard t o avoid) shouldn't increase t he securit y risk t o a syst em . The user of sudoscript already has t he privilege t o do anyt hing at all on t he syst em . The answer is " no" if you expect t he audit t rail provided by sudoscript t o be bullet proof. I t isn't . For one t hing, an xterm will produce a shell t hat is not audit ed. Addit ionally, t he FI FO t hat t he script s use m ust be writ able by t he effect ive user running it . I f t hat effect ive user is root, t hen of course t here are m any, m any ways t o avoid t he audit t rail. Sim ply killing t he daem on ( but not sudoshell) would do t he t rick nicely, for exam ple. The m oral is: don't give sudoscript t o users you don't t rust wit h root. I f you have t o give it t o such users, t hough, it is probably bet t er t han not hing.

6.10.3 Using sudoscript Build sudoscript from source in t he port s t ree or inst all it from a binary package. ( Not e t hat bot h are m isnom ers wit h respect t o sudoscript, since it is pure Perl. These m echanism s inst all t he script s and support ing files.) I f you want t o enable only root shells, sudoscript configurat ion is easy. Add an ent ry like t he following t o / usr/ local/ et c/ sudoers: Cmnd_Alias

SS

= /usr/local/bin/sudoshell, /usr/local/bin/ss

You can t hen grant sudoscript access t o chosen users t hrough t he usual m echanism s. For exam ple: %wheel

ALL=SS

joe

joesbox=SS

Now when a user runs ss: % ss The sudoscriptd doesn't appear to be running! Would you like me to start it for you? (requires root sudo privilege)? yes This will be a one-off startup of the daemon. You may have to arrange for it to be started when the system starts, if that's what you want. See the INSTALL file in the distribution for details. sudoscriptdwaiting for the daemon ..done Script started, output file is /var/run/sudoscript/ssd.test_root_1667/test1667.fifo #

- 327 -

The I NSTALL file m ent ioned lives in / usr/ local/ share/ doc/ sudoscript - version/ , along wit h lot s of ot her docum ent at ion. As shown in t he exam ple, sudoshell will st art sudoscriptd if it isn't running already. You probably want t o add sudoscriptd t o t he syst em st art up, which you can do by renam ing / usr/ local/ rc.d/ sudoscript d.sh.sam ple t o / usr/ local/ rc.d/ sudoscript d.sh. Unfort unat ely, t his script isn't a t rue rc- st yle st art up script in t he m anner of SysV init, in t hat it doesn't have start and stop t arget s; however, t his will change in t he next release. ( As of t his writ ing, sudoscript is at Version 2.1.1.) The im pat ient can m odify t he st art up script using [ H a ck # 8 6 ] .

sudoscript can enable shells as users ot her t han root. This could be handy for audit ing act ivit y of t he dba user, for inst ance. I f you want t o use t his feat ure, you m ust also add a Unix group called ssers. I f t his group exist s when sudoscriptd st art s, it will m ake som e changes t o t he files in / var/ run/ sudoscript ( where t he FI FOs live) t o accom odat e group access t o t hose files. This has securit y im plicat ions in t hat anyone in t he ssers group will have access t o t he FI FOs being used by any ot her concurrent user of sudoscript. Bot h t he user t hat will run ss and t he user ss will enable m ust be in t he ssers group. To get nonroot shells t o work, you also have t o change your sudoers ent ries like so: Host_Alias

DBBOXES

= db1,db2,db3

Cmnd_Alias

SS

= /usr/local/bin/sudoshell, \ /usr/local/bin/ss

Cmnd_Alias

SSASDBA

= /usr/local/bin/sudoshell -u dba, \ /usr/local/bin/ss -u dba

%wheel

ALL=SS

joe

joesbox=SS

datamonkey

DBBOXES=(dba) SSASDBA

Once t he ssers group and t he preceding ent ries in are place: % id uid=1004(datamonkey) gid=1004(datamonkey) groups=1004(datamonkey), 92(ssers) % ss -u dba Password: Script started, output file is /var/run/sudoscript/ssd.datamonkey_dba_2223/datamonkey2223.fifo bash-2.05b$ id

- 328 -

uid=1005(dba) gid=1005(dba) groups=1005(dba), 92(ssers)

6.10.4 The sudoscript Log File The sudoscript log file lives in / var/ log/ sudoscript . I t cont ains ent ries like t he following: Mon Dec 22 00:32:19 New logger for datamonkey with pid 2223 Mon Dec 22 00:32:19 datamonkey:2223 Script started on Mon Dec 22 00:32:19 2003 Mon Dec 22 00:32:25 datamonkey:2223 bash-2.05b$ id Mon Dec 22 00:32:25 datamonkey:2223 uid=1005(dba) gid=1005(dba) groups=1005(dba), 92(ssers) Mon Dec 22 00:49:09 datamonkey:8603 bash-2.05b$ vi .bashrc

(Tons and tons of garbage)

Mon Dec 22 00:49:54 datamonkey:8603 bash-2.05b$ exit Mon Dec 22 00:49:54 datamonkey:8603 Mon Dec 22 00:49:54 datamonkey:8603 Script done on Mon Dec

22 00:49:54 2003

Mon Dec 22 00:49:54 logger (datamonkey,8603) caught signal. Exiting

This looks pret t y bad! The problem is t hat t he script com m and fait hfully st ores all t he input and out put in t he shell, including all t he escape codes t hat t he t erm inal em ulat or t urns int o t hings like cursor m ovem ent and screen refreshes. The problem is part icularly acut e when t he user ent ers a full screen edit or, such as vi. There are t wo approaches t o t his problem t hat help t urn t he gibberish int o useful dat a. First , t his sed script from Unix Power Tools, Third Edit ion ( O'Reilly) will rem ove sim ple escape codes from script out put . #!/bin/sh # Public domain.

# Put CTRL-M in $m and CTRL-H in $b. # Change \010 to \177 if you use DEL for erasing. eval `echo m=M b=H | tr 'MH' '\015\010'`

exec sed "s/$m\$// :x

- 329 -

s/[^$b]$b// t x" $*

Run t he previous out put t hrough t his script . You'll see som et hing like: Mon Dec 22 00:32:19 New logger for datamonkey with pid 2223 Mon Dec 22 00:32:19 datamonkey:2223 Script started on Mon Dec 22 00:32:19 2003 Mon Dec 22 00:32:25 datamonkey:2223 bash-2.05b$ id Mon Dec 22 00:32:25 datamonkey:2223 uid=1005(dba) gid=1005(dba) groups=1005(dba), 92(ssers) Mon Dec 22 00:49:09 datamonkey:8603 bash-2.05b$ vi .bashrc

(Still tons of garbage)

Mon Dec 22 00:49:54 datamonkey:8603 ESC[Mon Dec 22 00:49:54 datamonkey:8603 bash-2.05b$ exit Mon Dec 22 00:49:54 datamonkey:8603 Mon Dec 22 00:49:54 datamonkey:8603 Script done on Mon Dec

22 00:49:54 2003

Mon Dec 22 00:49:54 logger (datamonkey,8603) caught signal. Exiting

That 's a m ore int elligible version of t he out put , but t he vi session is st ill scram bled. We can t ake advant age of t he fact t hat we probably are running t he sam e t erm inal em ulat or as t he user. I f we snip out j ust t he vi session from t he log and t hen cat it t o t he screen, we get : This is a normal line in a file Why does this look so bad??

~ ~ .. many more ~ lines.. ~ ~ ~ :q

- 330 -

That 's recognizable as a vi screen. I n fact , our screen has been updat ed several t im es, once for every t im e t he screen was refreshed in t he original session. The final display shows t he final st at e of t he vi session. Why not clean t his up in t he logging daem on? Because inform at ion is invariably lost when you do t hat kind of t hing. I t 's bet t er t o clean up aft er t he log file is writ t en. I n case you filt er out som et hing im port ant , you st ill have t he original log t o fall back on.

6.10.5 See Also • • • • •

•

man sudoscript man sudoscriptd man sudoshell The sudoscript web sit e ( ht t p: / / egbok.com / sudoscript / ) The Sudoscript - user m ailing list subscript ion link ( ht t p: / / list s.sourceforge.net / m ailm an/ list info/ sudoscript - user) The Problem of PORCMOLSULB ( ht t p: / / egbok.com / sudoscript / PORCMOLSULB.ht m l)

- 331 -

Hack 63 Restrict an SSH server

Con t r ol you r ssh scr ipt s by pla cin g t h e m in a j a il. Using SSH increases t he securit y of file t ransfers and net work logins. Many net work t asks, however, don't really need t he shell associat ed wit h a user account —rem ot e backups, for exam ple. Aft er all, a shell brings wit h it com m ands and an ent ry point int o a syst em 's direct ory st ruct ure. That 's som ewhat scary when you consider t hat m any of your SSH t asks are script ed. Configuring a rest rict ed SSH shell such as scponly can m it igat e t his risk. Not only does it provide nonint eract ive ( read script ed) logins int o t he SSH server, it lim it s t he set of available com m ands. Addit ionally, it provides a chroot opt ion, allowing you t o rest rict t he scponly user account t o it s own direct ory st ruct ure.

6.11.1 Installing scponly Before inst alling t his port , read t hrough t he available opt ions in it s Makefile: # cd /usr/ports/shells/scponly # more Makefile

Depending on t he script s you plan on using, consider disabling wildcard processing ( which can help prevent accident s like rm -R *) . You can also enable rsync support , which is ideal if you're using rsnapshot for backups [ H a ck # 3 5 ] . I f you want t o rest rict t he account t o it s own direct ory, prevent ing your script s from accessing anyt hing else on t he SSH server, include t he chroot opt ion. Once you've chosen your desired opt ions, pass t hem t o t he make com m and. Here I 'll enable chroot support : # make -DWITH_SCPONLY_CHROOT install I f you include t he chroot opt ion, do not use t he clean t arget at t he end of your make com m and. make clean will rem ove t he work/ direct ory, which cont ains a script t hat will set up t he chroot for you.

Toward t he end of t he inst allat ion, you'll see t his m essage: Run following script to setup chroot cage: /usr/ports/shells/scponly/work/scponly-3.8/setup_chroot.sh

Before running t his script , choose a new nam e for t he user account you wish t o rest rict . The script will abort if you use an exist ing user account .

- 332 -

Here I 'll creat e a chroot for an account nam ed backup: # cd work/scponly-3.8/ # chown +x setup_chroot.sh # ./setup_chroot.sh Next we need to set the home directory for this scponly user. please note that the user's home directory MUST NOT be writable by the scponly user. this is important so that the scponly user cannot subvert the .ssh configuration parameters.

for this reason, an "incoming" subdirectory will be created that the scponly user can write into. if you want the scponly user to automatically change to this incoming subdirectory upon login, you can specify this when you specify the user's home directory as follows:

set the home dir to /chroot_path//incoming

when scponly chroots, it will only chroot to chroot_path and afterwards, it will chdir to incoming. enter the home directory you wish to set for this user: /usr/home/rembackup/ Install for what username? backup ls: /lib/libnss_compat*: No such file or directory creating

/usr/home/rembackup/incoming directory for uploading files

6.11.2 Testing the chroot The script will have creat ed t he following direct ory st ruct ure for you: # ls -l /usr/home/rembackup total 10 drwxr-xr-x

2 root

wheel

512 Jan 22 12:37 bin/

drwxr-xr-x

2 root

wheel

512 Jan 22 12:38 etc/

drwxr-xr-x

2 backup wheel

512 Jan 22 12:38 incoming/

- 333 -

drwxr-xr-x

2 root

wheel

512 Jan 22 12:37 lib/

drwxr-xr-x

7 root

wheel

512 Jan 22 12:37 usr/

# ls -l /usr/home/rembackup/bin/ total 1868 -rwxr-xr-x

1 root

wheel

88808 Jan 22 12:37 chmod*

-rwxr-xr-x

1 root

wheel

14496 Jan 22 12:37 echo*

-rwxr-xr-x

1 root

wheel

72240 Jan 22 12:37 ln*

-rwxr-xr-x

1 root

wheel

567772 Jan 22 12:37 ls*

-rwxr-xr-x

1 root

wheel

-rwxr-xr-x

1 root

wheel

-rwxr-xr-x

1 root

wheel

-rwxr-xr-x

1 root

wheel

-rwxr-xr-x

1 root

wheel

73044 Jan 22 12:37 mkdir* 437684 Jan 22 12:37 mv* 80156 Jan 22 12:37 pwd* 439812 Jan 22 12:37 rm* 69060 Jan 22 12:37 rmdir*

# ls -l /usr/home/rembackup/usr/bin/ total 48 -rwxr-xr-x

1 root

wheel

7016 Jan 22 12:37 chgrp*

-rwxr-xr-x

1 root

wheel

7688 Jan 22 12:37 groups*

-rwxr-xr-x

1 root

wheel

7688 Jan 22 12:37 id*

-rwxr-xr-x

1 root

wheel

22616 Jan 22 12:37 scp*

# ls -l /usr/home/rembackup/usr/sbin/ total 8 -rwxr-xr-x

1 root

wheel

7016 Jan 22 12:37 chown*

There you have it ; t hese are t he only com m ands t hat account can use during an SSH session. You can also verify t hat t he specified user account was creat ed for you. I 'll check for t hat backup account : # grep backup /etc/master.passwd backup:*:1015:1015::0:0:User \

- 334 -

&:/usr/home/rembackup//incoming:/usr/local/sbin/scponlyc

Not ice t hat t he account is rest rict ed t o t he scponlyc shell. The t railing c indicat es t hat t his is a chroot.

6.11.3 Now What? Now t hat you have a rest rict ed account , t est it wit h one of your SSH script s. Don't forget t o set up your aut hent icat ion m et hod. Eit her set a password on t he account or configure keybased aut hent icat ion. You can use t his hack in conj unct ion wit h [ H a ck # 3 8 ] and [ H a ck # 3 9 ] .

6.11.4 See Also • •

man scponly The scponly hom e page ( ht t p: / / www.sublim at ion.org/ scponly/ )

- 335 -

Hack 64 Script IP Filter Rulesets

On e fir e w a ll r u le se t isn 't a lw a ys e n ou gh. As a firewall adm inist rat or, you know t hat it t akes a bit of creat ive genius t o creat e a ruleset t hat best reflect s your net work's securit y needs. Things can get m ore int erest ing if t hose needs vary by t im e of day. For exam ple, you m ay need t o allow I nt ernet access bet ween business hours but ban it during t he evening hours. This is easy t o do wit h t wo rulebases, a couple of script s, and t rust y old cron.

6.12.1 Limiting Access with IP Filter I have a FreeBSD firewall/ rout er guarding m y hom e net work. I also happen t o have a daught er who would spend her life online if she were allowed. There's a sim ple solut ion t o rest rict ing her access t o t he I nt ernet t o cert ain t im es of t he day wit hout having t o use a proxy. I use FreeBSD's IP Filter as m y firewall soft ware. My norm al set of firewall rules, / et c/ ipf.rules, allows unrest rict ed access t o t he I nt ernet . Here's t he sect ion of t hat rulebase t hat cont rols m y daught er's access: # --------------------------comment area begin-----------------------------# Internal Interface: ed0 # Allow internal traffic to flow freely. # -------------------------- comment area end -----------------------------pass in

on ed0 all

pass out on ed0 all

Not e t hat t his is not m y ent ire rulebase, j ust t he sect ion cont rolling t he int erface, ed0, connect ed t o t he port ion of t he net work cont aining m y daught er's com put er. Also not e t hat I did not use t he norm al pass in quick on ed0 all or pass out quick on ed0 all. This is because t he use of t he word quick in IP Filter t ells t he program not t o look any furt her for rules applying t o t he flow of t raffic on an int erface. I f t hat were t he case, t his hack would not work. I saved a copy of m y unrest rict ed rulebase as / et c/ ipf.rules.allow for safekeeping. This will be m y first rulebase. # cp /etc/ipf.rules /etc/ipf.rules.allow

I next edit ed a copy of t he original rulebase file, / et c/ ipf.rules, t o block Nat asha's com put er ( I P 10.0.0.3) from accessing t he out side world while st ill allowing her t o do hom ework: # --------------------------comment area begin------------------------------

- 336 -

# Internal Interface: ed0 # Allow internal traffic to flow freely. # -------------------------- comment area end -----------------------------pass in

on ed0 all

pass out on ed0 all

# --------------------------block Natasha's computer-----------------------block in

on ed0 from any to 10.0.0.3

block out on ed0 from any to 10.0.0.3

I saved t his rule file as / et c/ ipf.rules.block, m y second rulebase. This second ruleset will effect ively block her from surfing and using t he usual plet hora of m essaging program s.

6.12.2 Switching Rules on a Schedule To im plem ent t hese rest rict ions at a specific t im e, I wrot e a sm all script : #!/bin/sh

# copy the restrictive rules to the default ipfilter rulebase cp /etc/ipf.rules.block /etc/ipf.rules

# cause ipfilter to re-read and apply the new rulebase /sbin/ipf -Fa -f /etc/ipf.rules

Not ice t hat t his is a very sim ple Bourne shell script . As t he com m ent s st at e, it copies t he second, rest rict ive rulebase t o t he rulebase used by IP Filter. I t t hen t ells IP Filter t o reread and apply t he newly copied rulebase. I saved t his script as / usr/ local/ bin/ block.sh and m ade it execut able: # chmod 751 /usr/local/bin/block.sh

From t here, I used cron t o schedule t he rest rict ion. First , I open up t he crontab edit or: # crontab -e

and t hen add t he line: # minute, hour, all days, all weeks, on these days, script to run

- 337 -

0

21

*

*

0-4

/usr/local/bin/block.sh

which will effect ively shut down access t o t he out side world st art ing at 9: 00 PM, Sunday t hrough Thursday ( i.e., school night s) . To allow access t o t he I nt ernet in t he m orning, I need anot her script : #!/bin/sh

# copy the non-restrictive rules to the default ipfilter rulebase cp /etc/ipf.rules.allow /etc/ipf.rules

# cause ipfilter to re-read and apply the new rulebase /sbin/ipf -Fa -f /etc/ipf.rules

This script is very sim ilar t o t he first one, except t hat it copies over t he non- rest rict ive rulebase. I saved t his file as / usr/ local/ bin/ allow.sh and m ade it execut able: # chmod 751 /usr/local/bin/allow.sh

Once again, I launched crontab -e t o add t he following line: # minute, hour, all days, all weeks, on these days, script to run 0

7

*

*

1-5

/usr/local/bin/allow.sh

This will allow access t o resum e at 7: 00 AM, Monday t o Friday. Obviously t here are no rest rict ions on t he weekends.

6.12.3 Hacking the Hack While I 've successfully used t his hack at hom e for several years, it is easy t o see how t he sam e logic could apply t o schedule m ult iple rulebases t o suit any net work's needs. This gives an adm inist rat or m uch m ore flexible cont rol over t raffic, wit hout t he overhead of addit ional firewall soft ware.

6.12.4 See Also • •

man crontab The I P Filt er HOWTO ( ht t p: / / www.obfuscat ion.org/ ipf/ )

- 338 -

Hack 65 Secure a Wireless Network Using PF

Pr ot e ct you r pr iva t e w ir e le ss ne t w or k fr om u na u t h or ize d u se . The abundance of 802.11 wireless net works has raised an im port ant quest ion. How can you secure a wireless net work so t hat only recognized syst em s can use it ? Wireless Encrypt ion Prot ocol ( WEP) and MAC access list s offer som e prot ect ion against unaut horized users; however, t hey can be difficult t o m aint ain. Wit h OpenBSD's PF, we can m aint ain t ables of recognized client s and updat e t hose t ables wit h a single shell com m and. Known client s can access t he I nt ernet ; unknown client s will only ever see a web page inform ing t hem t hat t his is a privat e net work. For t his hack, we will use dhcpd, PF, and Apache.

6.13.1 DHCP Configuration We'll use a sim ple DHCP configurat ion in / et c/ dhcpd.conf like t his: shared-network GUEST-NET { max-lease-time 300; default-lease-time 120;

option

domain-name-servers 192.168.0.1;

option

routers 192.168.0.1;

subnet 192.168.0.0 netmask 255.255.255.0 { range 192.168.0.101 192.168.0.254; } }

I n t his case, we're using t he subnet 192.168.0.0/ 24. Our firewall and NAT gat eway is 192.168.0.1, and it 's also configured as t he DNS server for our net work. We've allocat ed a range of I P addresses ( 192.168.0.101 t o 192.168.0.254) for dist ribut ion on a first - com e, first - served basis t o any host t hat request s an address via DHCP. Anybody t hat connect s t o our net work will be able t o request a valid I P address in t hat range. The securit y will com e from our PF configurat ion.

6.13.2 PF Configuration OpenBSD has an excellent FAQ on PF, along wit h an exam ple of how t o writ e a ruleset for a hom e or sm all office net work. We'll use t his exam ple as a t em plat e.

- 339 -

We'll st art wit h t he sam ple PF configurat ion t hat allows any host on t he int ernal int erface ( represent ed by t he m acro $int_if) full access t o t he I nt ernet . Then, we will m odify t he rules in / et c/ pf.conf so t hat only aut horized host s have access and set up a web server t o respond t o request s from unaut horized host s. We will also allow unaut horized host s direct access t o our DNS server, t o sim plify our rules and t o avoid m ore com plex split - horizon DNS configurat ion. First , let 's creat e t he t able for aut horized host s and m acros for t he web server and t he DNS server: auth_server = "127.0.0.1 port 8080" dns_server

= "192.168.0.1"

table { 192.168.0.1, 192.168.0.11 };

These lines go near t he t op of / et c/ pf.conf, before any queue, NAT, or filt er rules. We've init ialized t he t able t o cont ain t he addresses of our NAT gat eway and one ot her host , 192.168.0.11, a st at ically configured box we'd like t o have access t o as well. While PF has a ruleset loaded, we can add a host t o t he t able on t he fly: # pfctl -t authorized_hosts -Tadd 192.168.0.101

We can also delet e a host : # pfctl -t authorized_hosts -Tdelete 192.168.0.102

and list all t he aut horized host s: # pfctl -t authorized_hosts -Tshow

Now we need t o m odify t he filt er rules so only our aut horized host s have access. These rules allow any host on our net work t o have access: pass in

on $int_if from $int_if:network to any

pass out on $int_if from any

keep state

to $int_if:network keep state

We'll change t hem like t his t o use our t able: pass in

on $int_if from to any keep state

pass out on $int_if from any to keep state

Right aft er t hose rules, we'll add t he following rules t o allow unaut horized host s t o access our web server and DNS server: pass in

on $int_if proto tcp from ! to $auth_server

pass in

on $int_if proto {tcp, udp} from any to $dns_server port domain \

- 340 -

keep state

Now any host in t he authorized_hosts t able will have full access t o t he I nt ernet . Any ot her host s will only be able t o lookup nam es and reach t he web server. We'll add som e sim ple rules so unaut horized users will see a rej ect ion page if t hey t ry t o go t o any web sit e. I n t he NAT sect ion, we'll add t his rule: rdr on $int_if proto tcp from ! to any port www -> \ $auth_server

This rule redirect s any unknown host at t em pt ing t o access a rem ot e m achine on t he www port t o t he web server t hat will ret urn t he rej ect ion page. We could inst all a web server on t he firewall box or on som e separat e m achine. I n m y case, I 'll run Apache on t he firewall, list ening at 127.0.0.1 and port 8080, so it won't be confused wit h any ot her web servers I 'm running.

6.13.3 Apache Configuration Apache is inst alled by default wit h OpenBSD, so we'll reconfigure it t o list en on port 8080 of t he gat eway ( wit h I P address 127.0.0.1) and ret urn t he sam e page for every URL request ed. ( Apache is also available in t he FreeBSD port s collect ion and Net BSD packages collect ion.) First , we'll enable Apache wit h t he httpd_flags param et er in / et c/ rc.conf. Next , we need t o edit Apache's configurat ion file, / var/ www/ conf/ ht t pd.conf. Find t he Listen direct ive and add 127.0.0.1:8080. Next , creat e a VirtualHost ent ry like t his:

ServerAdmin none DocumentRoot /var/www/auth ErrorDocument 404 /index.html

This t ells Apache t o list en t o t he appropriat e port and I P address. For every incom ing request , Apache will t ry t o serve a page beneat h t he given direct ory. Any t im e it can't find a page, it will serve t he index.ht m l page inst ead. We don't have eit her yet , so creat e t he direct ory / var/ www/ aut h and place an index.ht m l like t his in it :

Unauthorized -- This is a private network

- 341 -

Unauthorized

This is a private network and you are not authorized to use it.

6.13.4 Putting it All Together St art or rest art dhcpd, pf, and Apache like t his, where [interfaces] is t he list of int erfaces on which you provide DHCP: # kill `cat /var/run/dhcpd.pid`; dhcpd -q [interfaces]

# pfctl -f /etc/pf.conf # apachectl stop && apachectl start

Congrat ulat ions! When a new host connect s t o your net work, it should request an address wit h DHCP. I f so, it will receive an address in t he range of 192.168.0.101 t o 192.168.0.254. I f t he assigned address is not already in t he authorized_hosts t able, any t im e t hat host at t em pt s t o load a web page it will receive your Unauthorized page. The firewall will silent ly discard any packet s dest ined for any ot her port s out side of your net work. I f you want t o allow a new host t o use your net work, j ust use pfctl t o add it t o t he t able. To m ake t he change perm anent , add t he address or a range of addresses t o t he t able definit ion in / et c/ pf.conf, or even creat e an ext ernal file list ing allowed addresses. See t he PF FAQ sect ion on t ables for m ore.

6.13.5 Security Concerns This t echnique only cont rols t he abilit y of host s on your net work t o rout e packet s t hrough your firewall. I t will not prot ect ot her host s on t he sam e subnet from unaut horized access, so t hey should have reasonable local firewall rules. A wise approach m ight be t o build a firewall wit h t hree int erfaces: one ext ernal and t wo int ernal. One int ernal subnet would host your regular m achines, and t he ot her subnet would allow guest access wit h t his t echnique, separat ing t he subnet s wit h addit ional PF rules.

6.13.6 Hacking the Hack Running t he web server on t he firewall is a sim ple approach. However, you can redirect t o anot her host , such as a dedicat ed aut hent icat ion server. For sim plicit y, t his server should not be on t he $int_if:network subnet ; if it is, t he redirect ion becom es m ore com plicat ed. The PF FAQ has a sect ion devot ed t o port forwarding in t his m anner.

- 342 -

I used Apache because it is inst alled by default wit h OpenBSD and because it s configurat ion is t rivial in t his case. Alm ost any HTTP server will do t he j ob, t hough.

6.13.7 See Also • •

OpenBSD's PF FAQ ( ht t p: / / www.openbsd.org/ faq/ pf/ ) NoCat .net 's NoCat Aut h, aut hent icat ion soft ware for open wireless nodes ( ht t p: / / nocat .net / )

- 343 -

Hack 66 Automatically Generate Firewall Rules

Ea sily pr ot e ct a ny Fr e e BSD w or k st a t ion w it h a fu lly con figu r e d fir e w a ll. You know t he im port ance of being prot ect ed by a firewall. You know where t o look in t he m anpages for det ails. Given enough t im e and t rouble, you could writ e a firewall configurat ion for any sit uat ion. They're all reasonably sim ilar, t hough, so why not generat e t he configurat ion by answering a few quest ions? That 's t he purpose of t he I PFilt er set up script : t o generat e configurat ion rules for t ypical SOHO firewalls using FreeBSD and I PFilt er. Even novice users can ret ain t he full benefit s of a firewall wit hout first having t o learn synt ax. I n fact , wit h t his script , you should be able t o set up a t ypical firewall wit h no FreeBSD configurat ion knowledge at all. Even if you're not a novice user, t his is a great script t o refer friends t o as t hey discover FreeBSD. Now you can rest easy in t he t hought t hat your friends are prot ect ed—and you didn't even have t o find t he t im e t o show t hem how t o set up t heir syst em s.

6.14.1 What the Script Does The script uses a sim ple quest ion and answer t ext int erface. I t has four m ain part s:

Net work set t ings and I PFilt er firewall and I PNAT configurat ion This configures int ernal and ext ernal net work card int erface I P address set t ings eit her m anually or via DHCP. I t creat es st at eful firewall rules on t he ext ernal net work int erface and configures NAT t o provide I nt ernet connect ion sharing on t he int ernal net work int erface.

ADSL PPPOE configurat ion This prom pt s for a login nam e, password, and Et hernet NI C t o generat e t he / et c/ ppp/ ppp.conf file. I t t hen insert s t he required PPP variables in / et c/ rc.conf. This st art s userland PPP at boot up.

DHCP server configurat ion This checks for t he inst allat ion of t he I SC DHCP server. I f it 's not inst alled, t he script offers t o inst all t he lat est version from t he port s syst em or via a precom piled package. Once inst alled, t he script will configure t he DHCP server by prom pt ing for t he addresses of t he I SP's DNS servers, t he address of t he int ernal NI C t o use as t he default gat eway, and t he I P address range and subnet m ask t o use for t he int ernal LAN.

- 344 -

Serial console set up Answer " yes" t o t his sect ion of t he script if you plan on running t he firewall headless [ H a ck # 2 6 ] .

6.14.2 Installation The easiest way t o inst all t he script is t o download it t o t he syst em t hat will becom e t he firewall. I prefer t he fetch com m and: % fetch http://www.roq.com/bsd/ipfilterscript.tar.gz

I f net working isn't configured on t hat syst em yet , you can copy t he file from anot her device, such as a USB flash key: # mount -t msdos /dev/da0s1 /mnt # cp /mnt/ipfilterscript.tar.gz /tmp/

Once you have t he script , ext ract it and run it : # tar -zxf ipfilterscript.tar.gz # ./ipfilter.pl ###################################################################### 1: Would you like to setup PPPoE DSL connection (Choose 1) 2: Setup IP configuration, Firewalling and NAT (Choose 2) or 3: Setup a DHCP server (Choose 3 and hit enter) 4: Setup serial console support 5: Exit ######################################################################

I f you use ADSL wit h PPPoE, choose 1 and press Ent er. I f you have ADSL but use it wit h a st at ic I P, inst ead choose 2, which com bines I P configurat ion, Firewalling, and NAT set up. Choosing 3 will inst all and configure a DHCP server. First , however, configure your net work, as t he script will at t em pt t o download and inst all t he DHCP server.

6.14.3 Example Usage For t his exam ple, I will choose 2 for I P configurat ion. The script list s m y t hree Et hernet cards, rl0, xl0, and rl0, t wo of which I haven't configured. rl0: flags=8843 mtu 1500 inet6 fe80::202:44ff:fe36:8259%rl0 prefixlen 64 scopeid 0x1 inet 10.0.0.5 netmask 0xff000000 broadcast 10.255.255.255

- 345 -

ether 00:02:44:36:82:59 media: Ethernet autoselect (10baseT/UTP) status: active xl0: flags=8802 mtu 1500 options=3 ether 00:50:da:89:bc:9f media: Ethernet 10baseT/UTP (10baseT/UTP ) rl1: flags=8802 mtu 1500 ether 00:02:44:04:14:2c media: Ethernet autoselect (10baseT/UTP) status: no carrier lo0: flags=8049 mtu 16384 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 ##################################################################### Choose your external Nic, eg "fxp0" . If you are firewalling for a PPPoE / ADSL setup use "tun0" #####################################################################

At t he m om ent , I have only one Et hernet card plugged int o som et hing. Only rl0 has active st at us, so it is plugged int o m y ADSL m odem . I 'll configure it wit h a st at ic I P address by t yping in rl0 and pressing Ent er. The script now asks for m y int ernal net work card, which is rl1. ####################################################################### choose your internal Nic, eg "rl0" ####################################################################### rl1 ####################################################################### Internal nic IP, Recommended "192.168.1.1" . Hit "ENTER" for recommended defaults #######################################################################

- 346 -

Now t he script needs t o know t he I P address of t he gat eway device, behind which all of m y int ernal m achines live. The default s are fine, so I can sim ply press Ent er for t he next few quest ions. Setting Internal nic IP to 192.168.1.1 ####################################################################### Internal nic Netmask, Just hit enter for 255.255.255.0 ####################################################################### Setting Internal nic Netmask to 255.255.255.0

When asked for m y ext ernal I P, I t ype it in m anually since I am set t ing up a st at ic I P connect ion: ####################################################################### External nic IP, or type "DHCP" for DHCP, for connections like ADSL type "NONE" for no dhclient on external nic ####################################################################### 10.6.1.2 Setting External nic IP to 10.6.1.2 ####################################################################### External nic netmask, eg 255.255.255.0 ####################################################################### 255.255.255.254 Setting External Netmask to 255.255.255.254 ####################################################################### Do you want to enter a gateway default IP address? if you ISP provided you with a default gateway choose Yes Y/N, default = no y What is your gateway IP for your firewall machine to route to, (eg: 111.1.1.1) 10.6.1.1 ####################################################################### Do you want statefull firewall or just allow everything and rely on IPNAT to protect you, I recommend firewalling :) Choose: "y" for statefull firewall or "n" for allow everything

- 347 -

####################################################################### y ####################################################################### Do you want to forward any ports from the firewall to a internal host ip? n ####################################################################### Do you want IP Filter to log denied packets? Y/N, default = yes y #### Denied packets will be logged to /var/log/firewall.log #### ####################################################################### Do you want to install a /etc/ipfrestart script so you can easily reset your rules? Handy if you are trying out new rulesets. Y/N, default = yes y ####################################################################### Do you want ftp active mode supprt? when ftping out behind a basic NAT firewall, active mode ftp wont work. This is because normal active mode ftp actually initiates a FTP connection from the server back to YOU! and requires more then basic nat to work. The day FTP is gone and fully replaced by something more secure like SSH's sftp will be a day when the internet is large degree more secure. Choose: "y" to switch on active ftp support (recommended) or "n" y Going to write the data to these files /etc/rc.conf /etc/ipf.rules /etc//etc/ipnat.rules /etc/newsyslog.conf

hit ctrl+c to abort All done, type "reboot" for changes to take effect

- 348 -

######################################################################## Settings for internal machines behind the firewall: Gateway: 192.168.1.1 Netmask: 255.255.255.0 DNS: (Your ISPS DNS) Clients IP: 192.168.1.2 or higher ########################################################################

Finally, t he script writ es t he necessary inform at ion t o t he required configurat ion files. When I reboot, t he syst em is fully configured t o access t he I SP and provide NAT and DHCP services t o t he int ernal LAN, and it will prot ect all packet s t hrough it s firewall.

6.14.4 See Also • •

The I PFilt erscript web sit e ( ht t p: / / www.roq.com / bsd/ ) The I PFilt er web sit e ( ht t p: / / coom bs.anu.edu.au/ ~ avalon/ )

- 349 -

Hack 67 Automate Security Patches

Ke e p u p- t o- da t e w it h se cu r it y pa t che s. We all know t hat keeping up- t o- dat e wit h securit y pat ches is im port ant . The t rick is com ing up wit h a workable plan t hat ensures you're aware of new pat ches as t hey're released, as well as t he st eps required t o apply t hose pat ches correct ly. Michael Vince creat ed quickpatch t o assist in t his process. I t allows you t o aut om at e t he port ions of t he pat ching process you'd like t o aut om at e and m anually perform t he st eps you prefer t o do yourself.

6.15.1 Preparing the Script quickpatch requires a few dependencies: perl, cvsup, and wget. Use which t o det erm ine if you already have t hese inst alled on your syst em : % which perl cvsup wget /usr/bin/perl /usr/local/bin/cvsup wget: Command not found.

I nst all any m issing dependencies via t he appropriat e port ( / usr/ port s/ lang/ perl5, / usr/ port s/ net / cvsup- wit hout - gui, and / usr/ port s/ ft p/ wget , respect ively) . Once you have t he dependencies, download t he script from ht t p: / / roq.com / proj ect s/ quickpat ch and untar it : % tar xzvf quickpatch.tar.gz

This will produce an execut able Perl script nam ed quickpatch.pl. Open t his script in your favorit e edit or and review t he first t wo screens of com m ent s, up t o t he #Stuff you probably don't want to change line. Make sure t hat t he $release line m at ches t he t ag you're using in your cvs- supfile [ H a ck # 80] : # The release plus security patches branch for FreeBSD that you are # following in cvsup. # It should always be a long the lines of RELENG_X_X , example RELENG_4_9 $release='RELENG_4_9';

The next few pat hs are fine as t hey are, unless you have a part icular reason t o change t hem :

- 350 -

# Ftp server mirror from where to fetch FreeBSD security advisories $ftpserver="ftp.freebsd.org"; # Path to store patcher program files $patchdir="/usr/src/"; # Path to store FreeBSD security advisories $advdir="/var/db/advisories/"; $advdirtmp="$advdir"."tmp/";

I f you're planning on applying t he pat ches m anually and, when required, rebuilding your kernel yourself, leave t he next sect ion as is. I f you're brave enough t o aut om at e t he works, m ake sure t hat t he following pat hs accurat ely reflect your kernel configurat ion file and build direct ories: # Path to your kernel rebuild script for source patches that require kernel #rebuild $kernelbuild="/usr/src/buildkernel"; #$kernelbuild="cd /usr/src ; make buildkernel KERNCONF=GENERIC && make #installkernel KERNCONF=GENERIC ; reboot"; # Path to your system recompile scipt for patches that require full # operating system recompile $buildworld="/usr/src/buildworld"; #$buildworld="cd /usr/src/ ; make buildworld && make installworld ; reboot"; #Run patch command after creation, default no $runpatchfile="0"; # Minimum advisory age in hours. This is to make sure you don't patch # before your local cvsup server has had a # chance to recieve the source change update to your branch, in hours $advisory_age="24";

Review t he em ail account s so t he appropriat e account receives not ificat ions: # Notify email accounts, eg: qw([email protected] root@localhost); @emails = qw(root);

- 351 -

6.15.2 Running the Hack Run t he script wit hout any argum ent s t o see t he available opt ions: # /.quickpatch.pl # Directory /var/db/advisories/ does not exist, creating # Directory /var/db/advisories/tmp/ does not exist, creating Quickpatch - Easy source based security update system "./quickpatch.pl updateadv" to download / update advisories db "./quickpatch.pl patch" or "./quickpatch.pl patch > big_patch_file" to create patch files "./quickpatch.pl notify" does not do anything but email you commands of what it would do "./quickpatch.pl pgpcheck" to PGP check advisories

Before applying any pat ches, it needs t o know which pat ches exist . St art by downloading t he advisories: # ./quickpatch.pl updateadv

This will connect t o ft p: / / ft p.freebsd.org/ pub/ FreeBSD/ CERT/ advisories and download all of t he advisories t o / var/ db/ advisories. The first t im e you use t his com m and, it will t ake a while. However, once you have a copy of t he advisories, it t akes only a second or so t o com pare your copies wit h t he FTP sit e and, if necessary, download any new advisories. Aft er downloading t he advisories, see if your syst em needs pat ching: # ./quickpatch.pl notify #

I f t he syst em is fully pat ched, you'll receive your prom pt back. However, if t he syst em is behind in pat ches, you'll see out put sim ilar t o t his: # ./quickpatch.pl notify ###################################################################### ####### FreeBSD-SA-04%3A02.shmat.asc ####### Stored in file /var/db/advisories/tmp/FreeBSD-SA-04%3A02.shmat ####### Topic: shmat reference counting bug ####### Hostname: genisis - 20/2/2004 11:57:30 ####### Date Corrected: 2004-02-04 18:01:10

- 352 -

####### Hours past since corrected: 382 ####### Patch Commands cd /usr/src # patch < /path/to/patch ### c) Recompile your kernel as described in and reboot the system. /usr/src/buildkernel ## Emailed root

I t looks like t his syst em needs t o be pat ched against t he " schm at reference count ing bug." While running in notify m ode, quickpatch em ails t his inform at ion t o t he configured address but neit her creat es nor inst alls t he pat ch. To creat e t he pat ch, use: # ./quickpatch.pl patch ######################################################### ####### FreeBSD-SA-04%3A02.shmat.asc ####### Stored in file /usr/src/FreeBSD-SA-04%3A02.shmat ####### Topic: shmat reference counting bug ####### Hostname: genisis - 21/2/2004 10:41:54 ####### Date Corrected: 2004-02-04 18:01:10 ####### Hours past since corrected: 405 ####### Patch Commands cd /usr/src # patch < /path/to/patch ### c) Recompile your kernel as described in # and reboot the #system. /usr/src/buildkernel

# file /usr/src/FreeBSD-SA-04%3A02.shmat /usr/src/FreeBSD-SA-04%3A02.shmat: Bourne shell script text executable

- 353 -

This m ode creat es t he pat ch as a Bourne script and st ores it in / usr/ src. However, it is up t o you t o apply t he pat ch m anually. This m ay suit your purposes if you int end t o review t he pat ch and read any not es or caveat s associat ed wit h t he act ual advisory.

6.15.3 Automating the Process One of t he advant ages of having a script is t hat you can schedule it s execut ion wit h cron. Here is an exam ple of a t ypical cron configurat ion for quickpatch.pl; m odify t o suit your own purposes. Rem em ber t o creat e your logging direct ories and touch your log files before t he first run. # Every Mon, Wed, and Fri at 3:05 do an advisory check and download any # newly released security advisories 5

3

*

*

1,3,5

root

/etc/scripts/quickpatch.pl updateadv > \

/var/log/quickpatch/update.log 2>1

# 20 minutes later, check to see if any new advisories are ready for use # and email the patch commands to the configured email address 25

3

*

*

1,3,5

root

/etc/scripts/quickpatch.pl notify >> \

/var/log/quickpatch/notify.log 2>&1

# 24 hours later patch mode is run which will run the patch commands if # no one has decided to interfere. 25

3

*

*

2,4,6

root

/etc/scripts/quickpatch.pl patch >> \

/var/log/quickpatch/patch.log 2>&1

6.15.4 See Also • •

The quickpatch.pl web sit e ( ht t p: / / roq.com / proj ect s/ quickpat ch) The FreeBSD Securit y Advisories page ( ht t p: / / www.freebsd.org/ securit y/ index.ht m l# adv)

- 354 -

Hack 68 Scan a Network of Windows Computers for Viruses

Regardless of t he size of your net work, t he cost of annual subscript ions for ant ivirus soft ware can quickly becom e a pain in t he . . . checkbook. Using FreeBSD's st rengt h as a net work server, how hard could it be t o hack an easier and cheaper way t o adm inist er t he ant ivirus bat t le? The solut ion I found uses a com binat ion of FreeBSD and Clam AV and Sharit y- Light , bot h of which are found in t he port s collect ion. As seen in [ H a ck # 1 9 ] , Sharit y- Light can m ount Windows shares. Once t he shares are m ount ed, Clam AV will scan t hem for viruses.

6.16.1 Preparing the Windows Systems For t he syst em s you wish t o virus scan, share t heir drives as follows: 1. Open My Com put er and right - click on t he drive you wish t o share. Select Sharing from t he list of opt ions t hat appear. I f Sharing is not available, you will need t o act ivat e file sharing in t he Net work set t ing in Cont rol Panel. Use Help if you're unsure of where t o find t his set t ing.

2. I n t he Sharing t ab of t he Propert ies window, assign a nam e t o t he new share. I 'll use cdrive in t his exam ple. Choose a nam e t hat is bot h useful t o you and not already in use. ( I f a share already exist s, click on New Share.) 3. Unless your net work is com plet ely closed t o t he out side world, click on Perm issions and lim it t he access t o your user. You should only need read access for scanning purposes. 4. I f you need furt her assist ance, search for " sharing" in Windows Help. ( Click on t he St art but t on and select Help.) Once you've configured t he Windows syst em s for sharing, it 's t im e t o prepare t he FreeBSD syst em .

6.16.2 Preparing the FreeBSD System I nst all and configure Sharit y- Light [ H a ck # 1 9 ] . Rem em ber t o edit / et c/ host s t o reflect t he Net BI OS nam es of t he Microsoft syst em s. Then, creat e a m ount point . Since I 'll be aut om at ing t he process lat er on wit h a script , I need only one m ount point . For now, I 'll t est t he required st eps using one syst em : # mkdir /mnt/winshare

- 355 -

# shlight //winbox1/cdrive /mnt/winshare -U algould -P pwd Using port 1653 for NFS.

Here, I 've m ount ed t he cdrive share locat ed on winbox1 t o t he / m nt / winshare m ount point . This part icular share has a usernam e and password.

6.16.3 Installing and Running the Virus Scanner Clam AV is a GPL ant ivirus applicat ion t hat can be used alone or as a daem on in conj unct ion wit h m ail server t ools such as milter or pop3vscan ( bot h are available in t he port s collect ion) . Alt hough Clam AV can det ect and rem ove files t hat have been cont am inat ed wit h viruses, it does not disinfect t hese files. First , inst all Clam AV from t he port s syst em : # cd /usr/ports/security/clamav # make install clean

The Clam AV port inst alls several execut ables, including clamd, clamdscan, clamscan, freshclam, and sigtool. Each of t hese com m ands has a m anpage, as does clam av.conf, t he configurat ion file. For t he purposes of t his proj ect , we will be using only clamscan and freshclam. Since we will not be act ivat ing clamd, we do not need t o change t he configurat ion file. To updat e Clam AV's virus dat abase, execut e freshclam: # freshclam Current working dir is /usr/local/share/clamav Checking for a new database - started at Tue Dec 30 14:55:43 2003 Connected to clamav.elektrapro.com. Reading md5 sum (viruses.md5): OK viruses.db is up to date. Reading md5 sum (viruses2.md5): OK Downloading viruses.db2 ........... done Database updated (containing in total 11983 signatures). Database updated from clamav.elektrapro.com.

Once you've updat ed t he virus definit ions, use clamscan t o scan for viruses. You don't need t o be t he superuser, but you m ust be able t o read t he files and direct ories t hat you're scanning. Here's what happens when I scan an arbit rary file in m y hom e direct ory: % clamscan todo.txt

- 356 -

todo.txt: OK

----------- SCAN SUMMARY ----------Known viruses: 11982 Scanned directories: 0 Scanned files: 1 Infected files: 0 Data scanned: 0.00 Mb I/O buffer size: 131072 bytes Time: 0.241 sec (0 m 0 s)

One file scanned and no viruses found—good. When we scan t he Windows share, however, we will want t o scan direct ories recursively ( using t he -r opt ion) and log t he result ing report t o a file ( using t he -l filename opt ion) . To scan t he Windows share m ount ed at / m nt / winshare and save t he scan report t o / var/ log/ clam scan.log, execut e: # clamscan -l /var/log/clamscan.log -r /mnt/winshare

At t his point , t housands of filenam es fly by t he console, ending in a report sim ilar t o t he one shown earlier, which is saved t o / var/ log/ clam scan.log. clamscan will creat e t he report file if it does not exist . I f t he report file exist s, it will append t he new report t o t he exist ing file. You can review t he report wit h any t ext edit or. By default , clamscan only report s t hat a file has been infect ed—it is up t o you t o rem ove t he virus.

6.16.4 Automating the Process Scanning a single share is nice, but it would be even bet t er t o scan all of t he com put ers in t he net work at night . Since I can m ount and scan a share wit hout being prom pt ed for addit ional inform at ion, I can aut om at e t hese com m ands in a script . I want each Windows syst em t o be m ount ed, scanned, and unm ount ed in t urn, and I want each syst em t o have it s own scan report log. Since I also want t o put t he report logs in a clam scan direct ory in / var/ log, I need t o creat e t he direct ory. While I 'm at it , I 'll creat e t he script file and m ake it readable and execut able only by root : # mkdir /var/log/clamscan # touch /root/scanscript # chmod u+x,go-rwx /root/scanscript

Next , I 'll use m y favorit e edit or t o add t he com m ands t o / root / scanscript :

- 357 -

# more /root/scanscript #! /bin/sh # /root/scanscript # Sequentially mount Windows shares, scan them for viruses and unmount them.

# update virus databases freshclam

# winbox1 shlight //winbox1/cdrive /mnt/winshare -U algould -P pwd clamscan -l /var/log/clamscan/winbox1 -r /mnt/winshare unshlight /mnt/winshare

# winbox2 shlight //winbox2/cdrive /mnt/winshare -U algould -P pwd clamscan -l /var/log/clamscan/winbox2 -r /mnt/winshare unshlight /mnt/winshare

# winbox3 shlight //winbox3/cdrive /mnt/winshare -U algould -P pwd clamscan -l /var/log/clamscan/winbox3 -r /mnt/winshare unshlight /mnt/winshare

Now I can execut e t he script at will or schedule it s execut ion using cron. As wit h any ant ivirus scanning policy, execut e t he script when users will be least affect ed and t he scanned com put ers are up and running.

6.16.5 See Also man clamscan, man freshclam, man clamd, man clamdscan, man clamav.conf, man sigtool, The Sharit y- Light README and FAQ ( / usr/ local/ share/ doc/ Sharit y- Light / ) , The Sharit y- Light web sit e ( ht t p: / / www.obdev.at / product s/ sharit y- light / ) , The Clam AV web sit e ( ht t p: / / clam av.elekt rapro.com / )

- 358 -

Chapter 7. Going Beyond the Basics I nt roduct ion Sect ion 69. Tune FreeBSD for Different Applicat ions Sect ion 70. Traffic Shaping on FreeBSD Sect ion 71. Creat e an Em ergency Repair Kit Sect ion 72. Use t he FreeBSD Recovery Process Sect ion 73. Use t he GNU Debugger t o Analyze a Buffer Overflow Sect ion 74. Consolidat e Web Server Logs Sect ion 75. Script User I nt eract ion Sect ion 76. Creat e a Trade Show Dem o

- 359 -

Introduction Have you ever wondered what m odificat ions a web or m ail adm inist rat or m akes t o her servers? Maybe you're curious about what policies ot her adm inist rat ors use t o im plem ent bandwidt h cont rol? How do busy adm inist rat ors m anage t he log dat a from a server farm ? Perhaps you've cont em plat ed using t he Expect script ing language. However, t here's a good chance you've never t hought of using eesh, a t ot ally undocum ent ed but useful script ing ut ilit y. This chapt er also includes t wo hacks on t he em ergency repair process, as m any users prefer t o hope t hat t hey'll never need an em ergency repair kit . I nst ead, learn t o overcom e your fear of t he inevit able and m ast er t he art of repairing before t he em ergency.

- 360 -

Hack 69 Tune FreeBSD for Different Applications

Kn ow how t o t u ne a nd w h a t t o t u n e on you r Fr e e BSD syst e m As an adm inist rat or, you want t o t une your server syst em s so t hey work at peak efficiency. How do you know what t o t une? The answer depends heavily upon t he syst em 's funct ion. Will t he syst em perform a lot of sm all net work t ransact ions? Will it perform a sm all num ber of large t ransact ions? How will disk operat ions fact or in? How you answer t hese and ot her quest ions det erm ines what you need t o do t o im prove t he perform ance of your syst em s. This hack st art s wit h general opt im izat ions and t hen looks at funct ion- specific t unables.

7.2.1 Optimizing Software Compiling A good place t o st art is wit h soft ware com piling, as you want t o com pile soft ware and updat es as efficient ly as possible. Whenever you com pile, your com piler m akes assum pt ions about your hardware in order t o creat e binaries. I f you have an x86- com pliant CPU, for exam ple, your com piler will creat e binaries t hat can run on any CPU from a 386 onward. While t his allows port abilit y, it won't t ake advant age of any new abilit ies of your CPU, such as t he ext ended MMX, SSE, SSE2, or 3DNow! inst ruct ion set s. This is also why using precom piled binaries on your syst em is a surefire way t o reduce your overall perform ance. To ensure t hat soft ware will be com piled efficient ly, updat e your com piler flags in / et c/ m ake.conf . This file does not exist on new syst em s, but you can copy it from / usr/ share/ exam ples/ et c/ default s/ m ake.conf. St art by edit ing t he CPUTYPE= line t o reflect your CPU t ype; you'll find support ed t ypes list ed as com m ent s j ust before t his line. While t his will t ake advant age of your CPU's feat ures, t he disadvant age is t hat your com piled binaries m ay not run on different CPU t ypes. However, if all of your syst em s run t he sam e CPU plat form , any opt im izat ions you m ake t o shared binaries will affect all of your syst em s equally well. Next , change t he CFLAGS line t o CFLAGS= -O2 -pipe -funroll-loops. The -pipe opt ion can significant ly decrease t he am ount of t im e it t akes t o com pile soft ware, by using pipes t o com m unicat e bet ween com piler processes inst ead of t em porary files, but at t he expense of using slight ly m ore m em ory. The -funroll-loops saves one CPU regist er t hat would ot herwise be t ied up in t racking t he it erat ion of t he loop, but at t he expense of m aking a slight ly larger binary. The m ake.conf file also cont ains a line for CXXFLAGS. These opt ions are sim ilar t o t he CFLAGS options but apply t o C+ + code.

7.2.2 Kernel Optimizations I n your kernel configurat ion, add t he following line aft er t he machine i386 line:

- 361 -

makeoptions

COPTFLAGS="-O2 -pipe -funroll-loops -ffast-math"

This is sim ilar t o t he CLAGS opt ion in / et c/ m ake.conf, except t hat it opt im izes kernel com pilat ion. See [ H a ck # 5 4 ] for inst ruct ions on how t o st rip and com pile a kernel.

You can also add t his line: TOP_TABLE_SIZE=number

where number is a prim e num ber t hat is at least t wice t he num ber of lines in / et c/ passwd. This st at em ent set s t he size of t he hash t hat top uses. Set t he following opt ion if you have an AMD K5/ K6/ K6- 2 or Cyrix 6x86 chip. I t enables cache writ e allocat ion for t he L1 cache, which is disabled by default for t hese chips. options

CPU_WT_ALLOC

This opt ion will disable NFS server code, so include it when you know t hat you will not be act ing as an NFS server: options

NFS_NOSERVER

Anot her way of saving kernel m em ory is t o define t he m axim um num ber of swap devices, as shown in t he next exam ple. Your kernel needs t o allocat e a fixed am ount of bit m apped m em ory so t hat it can int erleave swap devices. I set t he num ber t o 1 on m y workst at ion and 2 on m y servers. I f I need t o add m ore t o a server, I can easily creat e anot her part it ion. options

NSWAPDEV=number

I f you plan on com piling all your requisit es int o t he kernel ( NI C driver, I PF/ I PFW, et c.) and won't be loading any of t hese opt ions as m odules, you can include t his line t o skip m odule com piling. This saves significant ly on t he t im e t aken t o com pile a kernel ( som et im es reducing it by t wo- t hirds) . makeoptions

MODULES_OVERRIDE=""

By default , all kernel opt ions are com piled as m odules. This allows you t o use kldload t o load a m odule even t hough it isn't specified in your kernel configurat ion file. The advant age of MODULES_OVERRIDE is t he decrease in kernel com pilat ion t im e. The disadvant age is t hat you'll need t o recom pile your kernel if you ever need t o add addit ional funct ionalit y, since you will have lost t he abilit y t o load t he kernel m odule separat ely.

- 362 -

7.2.3 Optimizing Network Performance Most m odern net work cards and swit ches support t he abilit y t o aut o- negot iat e t he com m unicat ion speed. While t his reduces adm inist rat ion, it com es at t he cost of net work t hroughput . I f your swit ch, server, or workst at ion is set t o use aut o- negot iat ion, it will st op t ransferring net work t raffic every few m om ent s t o renegot iat e it s speed. I f your net work driver support s it , you can set net work speed wit h ifconfig at runt im e or in / et c/ rc.conf at boot t im e. Here is an exam ple: % grep fxp0 /etc/rc.conf ifconfig_fxp0="inet x.x.x.x netmask x.x.x.x media 100BaseTX mediaopt full-duplex" Read t he m anpage for your NI C driver t o see whet her it support s mediaopt. For exam ple, if your NI C is rl0, read man 4 rl.

Next , you can enable DEVICE_POLLING in your kernel, which changes t he m et hod by which dat a t ravels from your net work card t o t he kernel. Wit hout t his set t ing, frequent int errupt calls m ay never free t he kernel. This is known as livelock and can leave your m achine unresponsive. Those of us unfort unat e enough t o be on t he wrong side of cert ain denial- ofservice at t acks know about t his. The DEVICE_POLLING opt ion causes t he kernel t o poll t he net work card at cert ain predefined t im es, during idle loops, or on clock int errupt s. This allows t he kernel t o decide when it is m ost efficient t o poll a device for updat es and for how long, and ult im at ely result s in a significant increase in perform ance. To t ake advant age of DEVICE_POLLING, you need t o com pile t wo opt ions int o your kernel: options DEVICE_POLLING and options HZ=1000. The lat t er opt ion slows t he clock int errupt s t o 1,000 t im es per second, which prevent s t he kernel from polling t oo oft en. Once you've recom piled your kernel, you'll st ill need t o enable t he feat ure. Add t his line t o / et c/ sysct l.conf: kern.polling.enable=1

The DEVICE_POLLING opt ion does not work wit h SMP- enabled kernels by default . I f you are com piling an SMP kernel wit h DEVICE_POLLING, first rem ove t he following lines from / usr/ src/ sys/ kern/ kern_poll.c: #ifdef SMP #include "opt_lint.h" #ifndef COMPILING_LINT #error DEVICE_POLLING is not compatible with SMP #endif

- 363 -

#endif

7.2.4 Optimizing Mail Servers Mail servers t ypically have a very large num ber of net work connect ions, during which t hey t ransfer a sm all am ount of dat a for a short period of t im e before closing t he connect ion. I n t his case, it is useful t o have a large num ber of sm all net work buffers. Net work connect ions have t wo buffers, one for sending and one for receiving. The size of t he buffer dict at es how quickly dat a will funnel t hrough t he net work and, in t he event of a net work delay, how m uch dat a can back up t he server for t hat connect ion before t here is a problem . Having a net work buffer t hat is t oo sm all will cause a dat a backlog as t he CPU wait s for t he net work t o clear, which causes great er CPU overhead. Having a net work buffer t hat is t oo large wast es m em ory by using t he buffer inefficient ly. Finding a balance is t he key t o t uning. I find t hat m ult iplying t he num ber of est ablished connect ions by 32 leaves m e wit h room t o breat he in t he event t hat I see an abnorm ally high surge of t raffic. I 've com e t o t his num ber over t im e t hrough t rial and error. So, if you expect t o have a peak of 128 servers sending you m ail, having 8,192 net work buffer clust ers would be good ( 128 2 per connect ion 32) . Also, rem em ber t hat connect ions can t ake up t o t wo full m inut es or m ore t o close com plet ely. I f you expect m ore t han 128 em ails in any given t wo- m inut e period, increase t he num ber accordingly. Anot her im port ant value t o cont rol is t he m axim um num ber of socket s. St art wit h t he sam e num ber of socket s as t here are net work buffers, and t hen t une as appropriat e. You can find out how m any net work buffer clust ers are in use wit h t he com m and netstat m. You can specify t he values you want in / boot / loader.conf. For exam ple: kern.ipc.nmbclusters=8192 kern.ipc.maxsockets=8192

As wit h any perform ance t uning, m onit or your syst em aft er m aking changes. Did you go overboard or underest im at e what you would need? Always check and adj ust accordingly.

7.2.5 Optimizing File Servers File servers generally have longer- lived and less frequent net work connect ions t han t hose on m ail servers. They usually t ransfer larger files. To det erm ine t he opt im al num ber of net work buffer clust ers, consider how m any client s you have. Mult iplying t he num ber of net work buffers by t wo is good pract ice, t hough som e adm ins prefer t o m ult iply by four t o accom m odat e m ult iple file t ransfers. I f you have 128 client s connect ing t o t he file server, set t he num ber of net work buffer clust ers t o 1,024 ( 128 2 per connect ion 4) .

7.2.6 Optimizing Web Servers I f you have m ore t han one elem ent on your web page ( for exam ple, m ult iple im ages or fram es) , expect web browsers t o m ake m ult iple connect ions t o your web server. I t 's com m on t o see four connect ions per page served. Also count any dat abase or net work connect ions m ade in server- side script ing.

- 364 -

Web servers go t hrough periods of highs and lows. While you m ight serve 100 pages per m inut e on average, at your low you m ight serve 10 pages per m inut e and at peak over 1,000 pages per m inut e. At a peak of 1,000 pages per m inut e, your clust ers and socket s should be around 16,384 ( 1,000 pages 2 per connect ion 4 connect ions 2 for growt h) .

7.2.7 See Also • • • •

•

•

man tuning man gcc ( t he GCC m anpage, which explains CPU com piling opt im izat ions) man ifconfig " Tuning FreeBSD for different applicat ions" ( ht t p: / / silverwrait h.com / papers/ freebsdt uning.php) " Opt im izing FreeBSD and it s kernel" ( ht t p: / / silverwrait h.com / papers/ freebsdkernel.php) Not es on t uning Apache servers at ht t p: / / www.bolt hole.com / uuala/ webt uning.t xt

- 365 -

Hack 70 Traffic Shaping on FreeBSD

Alloca t e ba n dw idt h for cr u cia l se r vice s. I f you're fam iliar wit h your net work t raffic, you know t hat it 's possible for som e syst em s or services t o use m ore t han t heir fair share of bandwidt h, which can lead t o net work congest ion. Aft er all, you have only so m uch bandwidt h t o work wit h. FreeBSD's dummynet m ay provide a viable m et hod of get t ing t he m ost out of your net work, by sharing bandwidt h bet ween depart m ent s or users or by prevent ing som e services from using up all your bandwidt h. I t does so by lim it ing t he speed of cert ain t ransfers on your net work—also called t raffic shaping.

7.3.1 Configuring Your Kernel for Traffic Shaping To t ake advant age of t he t raffic shaping funct ionalit y of your FreeBSD syst em , you need a kernel wit h t he following opt ions: options IPFIREWALL options DUMMYNET options HZ=1000

dummynet does not require t he HZ opt ion, but it s m anpage st rongly recom m ends it . See [ H a ck # 6 9 ] for m ore about HZ and [ H a ck # 5 4 ] for det ailed inst ruct ions about com piling a cust om kernel. The t raffic- shaping m echanism delays packet s so as not t o exceed t he t ransfer speed lim it . The delayed packet s are st ored and sent lat er. The kernel t im er t riggers sending, so set t ing t he frequency t o a higher value will sm oot h out t he t raffic by providing sm aller delays. The default value of 100 Hz will t rigger sends every 10 m illiseconds, producing burst y t raffic. Set t ing HZ=1000 will cause t he t rigger t o happen every m illisecond, result ing in less packet delay.

7.3.2 Creating Pipes and Queues Traffic shaping occurs in t hree st ages: 1. Configuring t he pipes 2. Configuring t he queues 3. Divert ing t raffic t hrough t he queues and/ or pipes Pipes are t he basic elem ent s of t he t raffic shaper. A pipe em ulat es a net work link wit h a cert ain bandwidt h, delay, and packet loss rat e. Queues im plem ent weight ed fair queuing and cannot be used wit hout a pipe. All queues connect ed t o a pipe share t he bandwidt h of t hat pipe in a cert ain configurable proport ion.

- 366 -

The m ost im port ant param et er of a pipe configurat ion is it s bandwidt h. Set t he bandwidt h wit h t his com m and: # ipfw pipe 1 config bw 120kbit/s This is a sam ple com m and run at t he com m and prom pt . However, as t he hack progresses, we'll writ e t he act ual dummynet policy as rules wit hin an ipfw rulebase.

This com m and creat es pipe 1 if it does not already exist , assigning it 120 kilobit s per second of bandwidt h. I f t he pipe already exist s, it s bandwidt h will be changed t o 120 Kbps. When configuring a queue, t he t wo m ost im port ant param et ers are t he pipe num ber it will connect t o and t he weight of t he queue. The weight m ust be in t he range 1 t o 100, and it default s t o 1. A single pipe can connect t o m ult iple queues. # ipfw queue 5 config pipe 1 weight 20

This com m and inst ruct s dummynet t o configure queue 5 t o use pipe 1, wit h a weight of 20. The weight param et er allows you t o specify t he rat ios of bandwidt h t he queues will use. Queues wit h higher weight s will use m ore bandwidt h. To calculat e t he bandwidt h for each queue, divide t he t ot al bandwidt h of t he pipe by t he t ot al weight s, and t hen m ult iply each weight by t he result . For exam ple, if a 120 Kbps pipe sees act ive t raffic ( called flows) from t hree queues wit h weight s 3, 2, and 1, t he flows will receive 60 Kbps, 40 Kbps, and 20 Kbps, respect ively. I f t he flow from t he queue wit h weight 2 disappears, leaving only t he flows wit h weight s 3 and 1, t hose will receive 90 Kbps and 30 Kbps, respect ively. ( 120 / ( 3+ 1) = 30, so m ult iply each weight by 30.) The weight concept m ay seem st range, but it is rat her sim ple. Queues wit h equal weight s will receive t he sam e am ount of bandwidt h. I f queue 2 has double t he weight of queue 1, it has t wice as m uch bandwidt h. Queues t hat have no t raffic are not t aken int o account when dividing t raffic. This m eans t hat in a configurat ion wit h t wo queues, one wit h weight 1 ( for unim port ant t raffic) and t he ot her wit h weight 99 ( for im port ant business t raffic) , having bot h queues act ive will result in 1% / 99% sharing, but if t here is no t raffic on t he 99 queue, t he unim port ant t raffic will use all of t he bandwidt h.

7.3.3 Using Masks Anot her very useful opt ion is t o creat e a m ask by adding mask mask-specifier at t he end your config line. Masks allow you t o t urn one flow int o several flows; t he m ask will dist inguish t he different flows. The default m ask is em pt y, m eaning all packet s fall int o t he sam e flow. Using mask all would m ake all connect ions significant , m eaning t hat every TCP or UDP connect ion would appear as a separat e flow. When you apply a m ask t o a pipe, each of t hat pipe's flows act s as a separat e pipe. Yet , each of t hose flows is an exact clone of t he original pipe, in t hat t hey all share t he sam e param et ers. This m eans t hat t he t hree act ive flows from our exam ple pipe will use 360 Kbps, or 120 Kbps each.

- 367 -

For a queue, t he flows will act as several queues, each wit h t he sam e weight as t he original one. This m eans you can use t he m ask t o share a cert ain bandwidt h equally. For our exam ple wit h t hree flows and t he 120 Kbps pipe, each flow will get a t hird of t hat bandwidt h, or 40 Kbps. This hack assum es t hat you will int egrat e t hese rules in your firewall configurat ion or t hat you are using ipfw only for t raffic shaping. I n t he lat t er case, having t he IPFIREWALL_DEFAULT_TO_ACCEPT opt ion in t he kernel will great ly sim plify your t ask. I n t his hack, we som et im es lim it only incom ing or out going bandwidt h. Wit hout t his opt ion, we would have t o allow t raffic in bot h direct ions, t raffic t hrough t he loopback int erface, and t hrough t he int erface we will not lim it . However, you should consider disabling t he IPFIREWALL_DEFAULT_TO_ACCEPT opt ion, as it will drop packet s t hat your policy does not specifically allow. Addit ionally, enabling t he opt ion m ay cause you t o accept pot ent ially m alicious t raffic you hadn't considered. The exam ple configurat ions in t his hack were t est ed wit h an ipf- based firewall t hat had an explicit deny rule at t he end. When int egrat ing t raffic shaping int o an exist ing ipfw firewall, keep in m ind t hat an ipfw pipe or ipfw queue rule is equivalent t o " ipfw accept aft er slow down . . . " if t he sysctl net.inet.ip.fw.one_pass is set t o 1 ( t he default ) . I f t he sysctl is set t o 0, t hat rule is j ust a delay in a packet 's pat h t o t he next rule, which m ay well be a deny or anot her round of shaping. This hack assum es t hat t he default behavior of t he pipe and queue com m ands is t o accept or an equivalent act ion.

7.3.4 Simple Configurations There are several ways of lim it ing bandwidt h. Here are som e exam ples t hat assum e an ext ernal int erface of ed0: # only outgoing gets limited ipfw pipe 1 config bw 100kbits/s

ipfw add 1 pipe 1 ip from any to any out xmit ed0

To lim it bot h incom ing and out going t o 100 and 50 Kbps, respect ively: ipfw pipe 1 config bw 100kbits/s ipfw pipe 2 config bw 50kbits/s

ipfw add 100 pipe 1 ip from any to any in

recv ed0

ipfw add 100 pipe 2 ip from any to any out xmit ed0

To set a lim it at ion on t ot al bandwidt h ( incom ing plus out going) : ipfw pipe 1 config bw 100kbits/s

- 368 -

ipfw add 100 pipe 1 ip from any to any in

recv ed0

ipfw add 100 pipe 1 ip from any to any out xmit ed0

I n t his exam ple, each host get s 16 Kbps of incom ing bandwidt h ( out going is not lim it ed) : ipfw pipe 1 config bw 16kbits/s mask dst-ip 0xffffffff

ipfw add 100 pipe 1 ip from any to any in recv ed0

7.3.5 Complex Configurations Here are a couple of real- life exam ples. Let 's st art by lim it ing a web server's out going t raffic speed, which is a configurat ion I have used on one of m y servers. The server had som e FreeBSD I SO files, and I did not want it t o hog all t he out going bandwidt h. I also want ed t o prevent people from gaining an unfair advant age by using download accelerat ors, so I chose t o share t he t ot al out going bandwidt h equally am ong 24- bit net works. # pipe configuration, 2000 kilobits maximum ipfw pipe 1 config bw 2000kbits/s

# the queue will be used to enforce the /24 limit mentioned above ipfw queue 1 config pipe 1 mask dst-ip 0xffffff00

# with this mask, only the first 24 bits of the destination IP # address are taken into consideration when generating the flow ID

# divert outgoing traffic from the web server (at 1.1.1.1) ipfw add queue 1 tcp from 1.1.1.1 80 to any out

Anot her real- life exam ple involves lim it ing incom ing t raffic by depart m ent . This configurat ion lim it s t he incom ing bandwidt h for a sm all com pany behind a 1 Mbps connect ion. Before t his was applied, som e users were using peer- t o- peer client s and download accelerat ors, and t hey were hogging alm ost all t he bandwidt h. The solut ion was t o im plem ent som e weight ed sharing bet ween depart m ent s and let t he depart m ent s t ake care of t heir own hogs. # Variables we will use # External interface EXTIF=fxp0

- 369 -

# My IP address ME=192.168.1.1

# configure the pipe, 95% of total incoming capacity ipfw pipe 1 config bw 950kbits/s

# configure the queues for the departments # departments 1 and 2 heavy net users ipfw queue 1 config pipe 1 weight 40 ipfw queue 2 config pipe 1 weight 40

# accounting, they shouldn't use the network a lot ipfw queue 3 config pipe 1 weight 5

# medium usage for others ipfw queue 4 config pipe 1 weight 20

# incoming mail (SMTP) to this server, HIGH priority ipfw queue 10 config pipe 1 weight 100

# not caught by the previous categories - VERY LOW bandwidth ipfw queue 11 config pipe 1 weight 1

# classify the traffic # only incoming traffic is limited, outgoing is not affected. ipfw add 10 allow ip from any to any out xmit via $EXTIF

# department 1 ipfw add 100 queue 1 ip from any to 192.168.0.16/28 in via $EXTIF

# department 2

- 370 -

ipfw add 200 queue 2 ip from any to 192.168.0.32/28 in via $EXTIF

# accounting ipfw add 300 queue 3 ip from any to 192.168.0.48/28 in via $EXTIF

# mail ipfw add 1000 queue 10 ip from any to $ME 25 in via $EXTIF

# others ipfw add 1100 queue 11 ip from any to any in via $EXTIF

The incom ing lim it is set t o 95% of t he t rue available bandwidt h. This will allow t he shaper t o delay som e packet s. I f t his were not t he case and t he pipe had t he sam e bandwidt h as t he physical link, all of t he delay queues for t he pipe would have been em pt y. The ext ra 5% of bandwidt h on t he physical link fills t he queues. The shaper chooses packet s from t he queues based on weight , passing t hrough packet s from queues wit h a higher weight before packet s from queues wit h lower weight . dummynet can lim it incom ing or out going bandwidt h in m ult iple ways. Pairing it wit h well t hought out ipfw rules can produce good result s when your requirem ent s are not ext rem ely com plex. However, keep in m ind t hat dummynet cannot guarant ee bandwidt h or qualit y of service.

7.3.6 See Also • • • •

man dummynet man ipfw man ipf " Using Dum m ynet for Traffic Shaping on FreeBSD" ( ht t p: / / www.bsdnews.org/ 02/ dum m ynet .php)

- 371 -

Hack 71 Create an Emergency Repair Kit

Th e Boy Scou t a n d syst e m a dm in ist r a t or m ot t o: " Be pr e pa r e d!" As a good adm inist rat or, you back up on a regular basis and periodically perform a t est rest ore. You creat e im ages [ H a ck # 2 3 ] of im port ant servers so you can quickly recreat e a syst em t hat is t aken out of com m ission. Are you prepared if a syst em sim ply refuses t o boot ? Som e part s of your drives are as im port ant as your dat a, yet few backup program s back t hem up. I 'm t alking about your part it ion t able and your boot blocks. Pret end for a m om ent t hat t hese som ehow becom e corrupt ed. The good news is t hat your operat ing syst em and all of your dat a st ill exist . The bad news is t hat you can no longer access t hem . Fort unat ely, t his is recoverable, but only if you've done som e preparat ory work before t he disast er. Let 's see what 's required t o creat e an em ergency repair kit .

7.4.1 Inventory of the Kit When you inst all a syst em , part icularly a server, invest som e t im e preparing for an em ergency. On a FreeBSD syst em , your kit should include: •

• •

•

The original inst all CD ( or t wo floppies cont aining kern.flp and m fsroot .flp or one floppy cont aining boot .flp) A floppy cont aining addit ional drivers, drivers.flp A fixit floppy, fixit .flp ( or a CD cont aining t he live filesyst em ; t his will be t he second, t hird, or fourt h CD in a set , but not t he first CD) A print out of your part it ion t able, / et c/ fst ab, and / var/ run/ dm esg.boot

Place t hese it em s in an envelope and st ore it in a secure locat ion wit h your backup t apes. Make a not e on t he envelope of t he syst em t o which t his kit should apply, along wit h t he version of t he operat ing syst em . I deally, you should have t wo copies of bot h your em ergency kit and backup m edia. St ore t he second copy off- sit e.

7.4.2 Preparing the Floppies Regardless of how you inst all a syst em , t ake a few m inut es t o download t he * .flp files found in t he floppies direct ory. This is especially im port ant if you use cvsup t o upgrade a syst em , as you can go m ont hs or years wit hout t he inst allat ion CD- ROM or floppy m edia. Your aim is t o t est t hese floppies on your syst em before a disast er st rikes. The last t hing you want t o be doing in an em ergency is scurrying around creat ing floppies only t o find t hat an essent ial driver is m issing. Here, I 'll connect t o t he m ain FreeBSD FTP server and download t he files for an i386, 5.1RELEASE syst em : # ftp ftp.freebsd.org Trying 62.243.72.50... Connected to ftp.freebsd.org.

- 372 -

220 Name (ftp.freebsd.org:dlavigne6): anonymous 331 Guest login ok, send your complete e-mail address as password. Password: ftp> cd pub/FreeBSD/releases/i386/5.1-RELEASE/floppies 250 CWD command successful. ftp> binary 200 Type set to I. ftp> mget *.flp mget boot.flp [anpqy?]? a Prompting off for duration of mget.

ftp> bye 221 Goodbye.

I find it convenient t o creat e a floppies direct ory wit h subdirect ories for each version of FreeBSD I have running in m y net work. I t hen download t he appropriat e * .flp files t o t he appropriat e subdirect ory so t hey are available when I wish t o creat e an em ergency repair kit for a new syst em . Once you have all five files, you can decide which ones you'll need for your part icular syst em . To perform an em ergency repair, you'll need som e way t o load your version of t he operat ing syst em int o m em ory so you can access t he ut ilit ies on t he fixit floppy and rest ore what ever dam age has happened t o your own operat ing syst em . There are several ways t o load an operat ing syst em . The first approach is t o boot direct ly from t he inst all CD- ROM, assum ing it is boot able and your BI OS support s t his. I f t his is your scenario, you don't need boot .flp, kern.flp, or m fsroot .flp. I f boot ing from t he CD- ROM isn't an opt ion, you can use eit her boot .flp or bot h kern.flp and m fsroot .flp. boot .flp is basically t he cont ent s of t he ot her t wo floppies placed ont o one floppy. The kicker is t hat you need a floppy capable of holding 2.88 MB of dat a. Depending upon your hardware, you m ay or m ay not need drivers.flp. I f t he inst aller det ect ed all of your hardware, you won't need t his floppy. Ot herwise, you will. Finally, if you don't have a CD cont aining t he live filesyst em , you'll need fixit .flp, as t his floppy cont ains t he act ual repair ut ilit ies. Use dd t o t ransfer t hese files t o floppies. Repeat t his for each * .flp file you require, using a different floppy for each file: # dd if=fixit.flp of=/dev/fd0

- 373 -

Label each floppy wit h it s nam e and version of FreeBSD and writ e prot ect t he floppies.

7.4.3 The Rest of the Kit Before t est ing your floppies, print som e im port ant syst em inform at ion—you won't rem em ber all of t hese det ails in an em ergency. First , you'll want a copy of your filesyst em layout : # more /etc/fstab # Device

Mountpoint

FStype

Options

Dump

Pass#

/dev/ad0s1b

none

swap

sw

0

0

/dev/ad0s1a

/

ufs

rw

1

1

/dev/ad0s1e

/tmp

ufs

rw

2

2

/dev/ad0s1f

/usr

ufs

rw

2

2

/dev/ad0s1d

/var

ufs

rw

2

2

/dev/acd0

/cdrom

cd9660

ro,noauto

0

0

proc

/proc

procfs

rw

0

0

linproc

/compat/linux/proc

linprocfs

rw

0

0

/dev/fd0

/floppy

msdos

rw,noauto

0

0

Here, I 've j ust sent t he out put t o a pager for viewing. Depending upon how print ing is set up on your syst em , redirect t hat out put eit her direct ly t o lpr or t o a file t hat you can send t o a print er. Not ice t hat all of m y hard drive part it ions st art wit h / dev/ ad0s1. The nam e of your hard drive is needed in order t o view t he part it ion t able, or what FreeBSD calls t he disklabel: # bsdlabel ad0s1 # /dev/ad0s1: 8 partitions: #

size

offset

fstype

[fsize bsize bps/cpg]

a:

524288

0

4.2BSD

2048 16384 32776

b:

1279376

524288

swap

c: 30008097

0

unused

0

0 # "raw" part, don't edit

d:

524288

1803664

4.2BSD

2048 16384 32776

e:

524288

2327952

4.2BSD

2048 16384 32776

f: 27155857

2852240

4.2BSD

2048 16384 28512

- 374 -

Once you have a print out of your disklabel, com plet e your kit by print ing t he cont ent s of / var/ run/ dm esg.boot . This file cont ains your st art up m essages, including t he result s of t he kernel probing your hardware.

7.4.4 Testing the Recovery Media Now you're ready t o t est t hat your kit works before sealing t he envelope and sending it off for secure st orage. First , boot t he syst em using eit her your CD- ROM or t he em ergency floppies. Once t he kernel has loaded and probed your hardware, t he screen will ask: Would you like to load kernel modules from the driver floppy? I f you choose yes, you will be asked t o insert t he drivers.flp floppy and will be present ed wit h a list of m odules t o choose from : cd9660.ko if_awi.ko if_fwe.ko if_sk.ko if_sl.ko if_sn.ko

Taking a look at t hose m odules, aren't you glad you're t est ing your kit before an em ergency? While t he m odules don't have t he m ost descript ive nam es, it 's easy t o find out what each m odule represent s if you have access t o a working syst em . For exam ple, t he m odules t hat begin wit h if are int erfaces. To see what t ype of int erface if_awi.ko is: % whatis awi awi(4)

- AMD PCnetMobile IEEE 802.11 PCMCIA wireless network driver

You can whatis each nam e; j ust don't include t he beginning if or t he t railing .ko. I f you do need any of t hese drivers, save yourself som e grief and writ e yourself a not e explaining which drivers t o choose off of t he drivers.flp. The lucky bloke who has t o repair t he syst em will t hank you for t his bit of hom ework. Once you exit from t his m enu, you'll be prom pt ed t o rem ove t he floppy. You'll t hen be present ed wit h t he sysinstall Main Menu screen. Choose Fixit from t he m enu and insert fixit .flp. You should be prom pt ed t o press Alt F4, and you should t hen see a Good Luck! screen wit h a Fixit# prom pt . Excellent , your floppy is good and your repair kit is com plet e. Type exit t o ret urn t o t he m enu and exit your way out of t he inst all ut ilit y. I f t his had been an act ual em ergency, you'd definit ely want t o read t he next hack [ H a ck # 72] .

7.4.5 See Also man bsdlabel, The Em ergency Rest ore Procedure sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO88591/ books/ handbook/ backup- basics.ht m l)

- 375 -

Hack 72 Use the FreeBSD Recovery Process

Le a r n h ow t o u se you r e m e r ge n cy r e pa ir k it be for e t h e e m e r ge n cy . Now t hat you have an em ergency repair kit , it's wort h your while t o do a dry run so you know ahead of t im e what opt ions will be available t o you. You m ay even decide t o m odify your kit as a result of t his t est . Let 's go back t o t hat sysinst all Main Menu screen [ H a ck # 7 1 ] and see what happens when you choose Fixit. You'll be present ed wit h t he following opt ions: Please choose a fixit option There are three ways of going into "fixit" mode: - you can use the live filesystem CDROM/DVD, in which case there will be full access to the complete set of FreeBSD commands and utilities, - you can use the more limited (but perhaps customized) fixit floppy, - or you can start an Emergency Holographic Shell now, which is limited to the subset of commands that is already available right now.

X Exit

Exit this menu (returning to previous)

2 CDROM/DVD

Use the "live" filesystem CDROM/DVD

3 Floppy

Use a floppy generated from the fixit image

4 Shell

Start an Emergency Holographic Shell

I f you choose t he Shell opt ion, you'll find t hat t hey weren't kidding when t hey warned you'd be lim it ed t o a subset of com m ands. Nearly all of t he com m ands you know and love will result in a not found error m essage. This is why you went t o t he t rouble of eit her creat ing t hat fixit floppy or purchasing/ burning a CD- ROM/ DVD t hat cont ains t he live filesyst em .

7.5.1 Using the fixit Floppy Let 's see what you can repair wit h t he fixit floppy. When you choose t hat opt ion, follow t he prom pt s: insert t he floppy, t hen press Alt F4. Do m ake not e of t he m essage you receive: +-----------------------------------------------------------------------+ | You are now running from FreeBSD "fixit" media.

|

| --------------------------------------------------------------------- | | When you're finished with this shell, please type exit.

- 376 -

|

| The fixit media is mounted as /mnt2.

|

|

|

| You might want to symlink /mnt/etc/*pwd.db and /mnt/etc/group

|

| to /etc/ after mounting a root filesystem from your disk.

|

| tar(1) will not restore all permissions correctly otherwise!

|

|

|

| Note: you might use the arrow keys to browse through the

|

| command history of this shell.

|

+-----------------------------------------------------------------------+

Good Luck!

Fixit#

I t 's not a bad idea t o creat e t hose sym links now, before you forget . You'll have t o m ount your root slice first , so refer t o your / et c/ fst ab print out for t he proper nam e of t hat slice. I n t his exam ple, / is on / dev/ ad0s1a. I 'll m ount it wit h t he read- writ e opt ion: Fixit# mount -o rw /dev/ad0s1a /mnt Fixit#

I f your com m and is successful, you'll receive t he prom pt back. A quick ls t hrough / m nt should convince you t hat you now have access t o t he hard disk's root filesyst em . I f your com m and is not successful, run fsck_ffs unt il t he filesyst em is clean, t hen m ount t he filesyst em : Fixit# fsck_ffs /dev/ad0s1 ** /dev/ad0s1 ** Last Mounted on /mnt ** Phase 1 - Check blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 821 files, 27150 used, 99689 free (985 frags, 12338 blocks, 0.8% fragmentation)

- 377 -

Fixit# mount -u -o rw /dev/ad0s1 /mnt

Now for t hose sym links: Fixit# ln -f -s /mnt/etc/*pwd.db /etc Fixit# ln -f -s /mnt/etc/group /etc

Not e t hat you need t o include t he force ( -f) swit ch when you m ake your sym bolic ( -s) links. You need t o overwrit e t he exist ing link t hat links mnt2, or t he fixit floppy, t o / et c. You inst ead want t o link t he files on your hard drive ( / m nt ) t o / et c. You'll also not ice t hat while in t he Fixit# prom pt , t he up arrow will recall hist ory, but t ab com plet ion does not work. At t hat Fixit# prom pt , you have t wo com m and set s available t o you. The first is t hat lim it ed com m and set t hat com es wit h t he sysinstall ut ilit y. Not e t hat t hese are t he only com m ands available at t hat holographic shell prom pt : Fixit# ls stand -sh*

gunzip*

route*

[*

gzip*

rtsol*

arp*

help/

sed*

boot_crunch*

hostname*

sh*

camcontrol*

ifconfig*

slattach*

cpio*

minigzip*

sysinstall*

dhclient*

mount_nfs*

test*

dhclient-script*

newfs*

tunefs*

etc/

ppp*

usbd*

find*

pwd*

usbdevs*

fsck_ffs*

rm*

zcat*

The second com m and set is on t he floppy it self, m ount ed as m nt 2: Fixit# ls mnt2/stand bsdlabel*

dd*

fixit_crunch*

mount_cd9660*

sleep*

cat*

df*

ftp*

mount_msdosfs*

swapon*

chgrp*

disklabel*

kill*

mv*

sync*

chmod*

dmesg*

ln*

reboot*

tar*

chown*

echo*

ls*

restore*

telnet*

- 378 -

chroot*

ex*

mkdir*

rm*

umount*

clri*

expr*

mknod*

rmdir*

vi*

cp*

fdisk*

mount*

rrestore*

view*

You'll also find a m inim al set of not es in: Fixit# ls stand/help

One of t he first t hings you'll not ice, especially if you t ry t o read one of t hose help docum ent s, is t he lack of a pager. You won't have any luck wit h more or less. However, cat and view are available for viewing files. I f you've never used view before, rem em ber t o t ype :q t o quit t he viewer. Also not e t hat all of t he rest ore ut ilit ies are on hand, unless you've used pax as your backup ut ilit y.

7.5.2 Using the Live Filesystem Let 's pause here for a m om ent and com pare t he fixit floppy t o t he live filesyst em . There's one CD m arked as live in a purchased set . I f you burn your own I SO im ages, t he second im age for your release will cont ain t he live filesyst em . For exam ple, here is t he list ing for ft p: / / ft p.freebsd.org/ pub/ FreeBSD/ I SO- I MAGES/ 5.1- RELEASE/ : 5.1-RELEASE-i386-disc1.iso

630048 KB

06/05/03

00:00:00

5.1-RELEASE-i386-disc2.iso

292448 KB

06/05/03

00:00:00

5.1-RELEASE-i386-miniinst.iso

243488 KB

06/05/03

00:00:00

1 KB

06/05/03

00:00:00

CHECKSUM.MD5

disc1.iso is t he inst all CD, and disc2.iso is t he live filesyst em CD. There are several advant ages t o using t he live filesyst em . First , you don't have t o m ake any floppies. I n fact , your ent ire kit can be as sim ple as t his one CD and your print out s specific t o t hat syst em . Second, t he CD is boot able, so you can reach t hat Fixit# prom pt in under a m inut e. Third, you have t he ent ire built - in com m and set available t o you. When you ent er t he Fixit screen, you'll see t he sam e welcom e m essage as before. This t im e, it is t he CD t hat is m ount ed as / m nt 2, which is really a link t o / dist : Fixit# ls -l /mnt2 lrwxr-xr-x

1 root

wheel

5 Dec

8 08:22 /mnt2@ -> /dist

Fixit# ls /dist .cshrc

boot/

etc/

root/

tmp/

.profile

boot.catalog

floppies/

rr_moved/

usr/

- 379 -

COPYRIGHT

cdrom.inf

mnt/

sbin/

bin/

dev/

proc/

sys@

var/

A quick ls /dist/bin and ls /dist/sbin will display all of t he com m ands t hat com e wit h a FreeBSD syst em . There isn't a lim it ed com m and set wit h t he live filesyst em .

7.5.3 Emergency Repair Now t hat I 've shown you t he various ways t o ent er t he Fixit facilit y, you're probably wondering what you should be doing at t hat prom pt . FreeBSD is quit e robust and is usually capable of boot ing your hard drive t o som e sort of prom pt . However, if t he disk fails com plet ely or is som ehow incapable of boot ing t o a prom pt , t he fixit facilit y is one of your opt ions. From here, you can run fsck on your various filesyst em s, which m ay fix t he problem . You can see which filesyst em s are st ill m ount able, allowing you t o assess t he ext ent of t he dam age. I f som e files were dam aged, you can rest ore t hose files from backup. I f it t urns out t hat t he drive is dam aged beyond repair, you can rest easy in t he fact t hat you have a print out of your hardware and part it ioning schem e, a floppy cont aining any necessary drivers, and a backup of all of your dat a. Above all, you were prepared.

7.5.4 See Also •

The Backup Basics sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ backupbasics.ht m l)

- 380 -

Hack 73 Use the GNU Debugger to Analyze a Buffer Overflow

You don 't h a ve t o be a pr ogr a m m e r t o use a de bu gge r . As an end user, you m ay not realize t hat you have t he abilit y t o analyze securit y exploit s. Aft er all, t he organizat ion t hat dist ribut es your operat ing syst em of choice or t he provider of a given applicat ion will deal wit h securit y issues and m ake updat es available. However, keep in m ind t hat Securit y Officers apply t he sam e t ools and t echniques t hat end users use for debugging program s. Knowing how t o analyze a problem will help you t o t roubleshoot any m isbehaving process in a Unix environm ent .

7.6.1 An Example Exploit Analyzing a m alfunct ioning process st art s wit h basic inform at ion, such as error m essages and ret urn values. Som et im es t hose aren't enough, t hough. Som e error m essages are unclear. I n t he case of securit y vulnerabilit ies, t here m ay not be an error code or ret urn value, because t he program m ay crash or m isbehave silent ly. The BSDs provide several t ools t o analyze a program 's execut ion. You can m onit or syst em calls wit h ktrace and resources wit h fstat. You can run a debugger such as GDB, t he GNU Debugger, and wat ch your operat ing syst em 's int ernal operat ion. I n som e cases, a program m ust run in a part icular environm ent , which m ay m ake it difficult t o analyze due t o t he lim it at ions of som e t ools. For exam ple, a telnetd advisory from 2001 ( ht t p: / / www.cert .org/ advisories/ CA- 2001- 21.ht m l) affect ed m ost Unix operat ing syst em s. This part icular vulnerabilit y cam e t o light when a group called TESO released an exam ple exploit for it . On Unix syst em s, telnetd runs as root , so t hat once t he syst em aut hent icat es t he user, t he process has t he privileges required t o set t he user I D of t he login shell t o t hat of t he user who logged in. This m eans t hat a rem ot e ent it y who can cause telnetd t o m isbehave by sending it carefully designed input could execut e processes as root on your syst em . On m ost Unix syst em s, telnetd does not run as a st andalone daem on. Since logins are relat ively infrequent ( on t he syst em t im escale com pared t o t housands of int errupt s per second) , t he inetd service st art s telnetd as needed. This is a sim ple exam ple of t he dat a st ream sufficient t o crash vulnerable telnetds using perl and nc ( net cat ) : % perl -e 'print "\377\366"x512' |

nc testhost telnet

This was t he exam ple I used t o diagnose t he problem and t est t he fix. I f you run t his com m and against an im pervious Telnet daem on, you'll see t he following out put :

- 381 -

% perl -e 'print "\377\366"x512' | nc testhost telnet

[Yes]

[Yes]

[Yes]

The [Yes] m essage will repeat 512 t im es because t he charact ers you sent , \377\366, represent t he Telnet prot ocol's " ARE YOU THERE" cont rol m essage, and you asked t he quest ion 512 t im es. I f you run t his com m and against a vulnerable telnetd, t he out put can vary. I n som e cases, your connect ion m ay close before you get 512 [Yes] responses because telnetd crashed. I n ot her cases, you m ay receive seem ingly random out put from port ions of t he telnetd m em ory space. These bot h indicat e t hat t he program did som et hing it was not supposed t o, due t o t he specific input you gave it .

7.6.2 Using the GNU Debugger I n order t o fix t he problem , we need t o find out where t he execut able did som et hing incorrect ly. We would like t o run t he program under t he cont rol of GDB, but we cannot st art telnetd from t he com m and line t he way we usually would when debugging m ost execut ables. Norm ally, GDB is invoked in one of t hree ways. First , t o run a program and debug it , t ype: % gdb programname

GNU gdb 5.3nb1 Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.

Type "show warranty" for details.

This GDB was configured as "i386--netbsdelf"...(no debugging symbols found)... (gdb) run

- 382 -

I f t his is your first t im e using gdb, t ype help at t he (gdb) prom pt . Type quit when you are finished using t he debugger.

Second, t o exam ine t he core file of a program t hat has already crashed, use: % gdb programname

programname .core

Third, t o exam ine a program t hat is already running, t ype: % gdb programname

processid

I n t he case of telnetd, we cannot use t he first m et hod, because inetd m ust st art telnetd in order t o at t ach it t o a net work socket and operat e properly. We cannot use t he second m et hod, because processes t hat run wit h root privileges do not leave core files, since t he program 's m em ory im age could cont ain sensit ive dat a. That leaves t he t hird m et hod. At t aching t o a running process is problem at ic because telnetd isn't running unt il som eone connect s. We'll need t o m odify our at t ack script : % perl -e 'sleep 30; print "\377\366"x512' |

nc testhost telnet

Now nc opens a socket t o t he t est host , inetd spawns a telnetd in response, and perl wait s for 30 seconds before sending t he at t ack st ring. I n anot her t erm inal, on t he t est host , we say: % ps -ax | grep telnetd 27857 ??

S

0:00.05 telnetd

27859 pd

S+

0:00.02 grep telnetd

% gdb /usr/libexec/telnetd 27857 GNU gdb[...] Attaching to program `/usr/libexec/telnetd', process 27857

- 383 -

From here we can allow telnetd t o crash and observe t he exact t ype of error t hat caused t he crash. I f we've built telnetd wit h debugging inform at ion, GDB will even display t he line of source code t he program was execut ing when it crashed. Now we can use our favorit e debugging t echniques and eit her insert debugging m essages or use GDB and set breakpoint s and wat chpoint s t o discover at what point t he program went off course. We can t hen det erm ine what changes t o m ake t o correct t he error and prevent t he exploit . I f you're not a program m er, you can save t he inform at ion and send it t o t he developers.

7.6.3 Hacking the Hack We were fort unat e in t his exam ple because we had det ails of t he exploit . That m ade it easy t o experim ent and t ry different approaches. I n m any cases, however, you won't know t he det ails of an exploit , and you m ay only know t hat t here is a problem because of error m essages in your logs. You can use tcpdump t o capt ure t he t raffic on t he relevant port . Once you can correlat e t he t im est am p of t he log's error m essage wit h som e of your tcpdump t raffic, you can t ake t he dat a sent in an at t ack and creat e a Perl script t o resend it . You can t hen apply t he t echniques already described t o analyze and correct t he problem .

7.6.4 See Also • • • • •

man ktrace man fstat man gdb The Net cat web sit e; see t he Read Me file ( ht t p: / / www.at st ake.com / research/ t ools/ net work_ut ilit ies) The " Debugging wit h GDB" t ut orial ( ht t p: / / www.delorie.com / gnu/ docs/ gdb/ gdb_t oc.ht m l)

- 384 -

Hack 74 Consolidate Web Server Logs

Au t om a t e log pr oce ssin g on a w e b fa r m . As t he adm inist rat or of m ult iple web servers, I ran across a few logging problem s. The first was t he need t o collect logs from m ult iple web servers and m ove t hem t o one place for processing. The second was t he need t o do a real- t im e tail on m ult iple logs so I could wat ch for specific pat t erns, client s, and URLs. As a result , I wrot e a series of Perl script s collect ively known as logproc. These script s send t he log line inform at ion t o a single log host where som e ot her log analysis t ool can work on t hem , solving t he first problem . They also m ult icast t he log dat a, let t ing you wat ch live log inform at ion from m ult iple web servers wit hout having t o wat ch individual log files on each host . A prim ary goal is never t o lose log inform at ion, so t hese script s are very careful about checking exit codes and such. The basic m odel is t o feed logs t o a program via a pipe. Apache support s t his wit h it s st andard logging m echanism , and it is t he only web server considered in t his hack. I t should be possible t o m ake t he syst em work wit h ot her web servers—even servers t hat can only writ e logs t o a file—by using a nam ed pipe. I 've used t hese script s on product ion sit es at a few different com panies, and I 've found t hat t hey handle high loads quit e well.

7.7.1 logproc Described Download logproc from ht t p: / / www.pet erson.at h.cx/ ~ j lp/ soft ware/ logproc.t ar.gz. Then, ext ract it : % gunzip logproc.tar.gz % tar xvf logproc.tar % ls -F logproc ./

../

logserver.bin/

webserver.bin/

% ls -F logserver.bin ./

apache_rrd*

cleantmp*

logwatch*

../

arclogs*

collect*

meter*

% ls -F webserver.bin ./

../

batcher*

cleantmp*

copier*

- 385 -

mining/

As you can see, t here are t wo part s. One runs on each web server and t he ot her runs on t he log server. The logs are fed t o a process called batcher t hat runs on t he web server and writ es t he log lines t o a bat ch file as t hey are received. The bat ch file st ays sm all, cont aining only five m inut es' wort h of logs. Each com plet ed bat ch file m oves off t o a holding area. A second script on each web server, t he copier , t akes t he com plet ed bat ch files and copies t hem t o t he cent ralized log host . I t t ypically runs from cron. On t he log host , t he collect process, also run from cron, collect s t he bat ches and sort s t he log lines int o t he appropriat e daily log files. The syst em can also m onit or log inform at ion in real t im e. Each batcher process dum ps t he log lines as it receives t hem out t o a m ult icast group. List ener processes can ret rieve t hose log lines and provide real- t im e analysis or m onit oring. See t he sam ple logwatch script included wit h logproc for det ails.

7.7.2 Preparing the Web Servers First , creat e a hom e direct ory for t he web server user. I n t his case, we'll call t he user www. Make sure t hat www's hom e direct ory in / et c/ m ast er.passwd point s t o t hat sam e locat ion, not t o / nonexist ent . I f necessary, use vipw t o m odify t he locat ion in t he password file. # mkdir ~www # chown www:www ~www

Next , log in as t he web server user and creat e a public/ privat e SSH keypair: # su www % ssh-keygen -t dsa

Creat e t he direct ories used by t he log processing t ools, and copy t he script s over: % cd ~www % mkdir -p bin logs/{work,save}/0 logs/tmp logs/work/1

% cp $srcdir/logproc/webserver.bin/* bin/

Exam ine t hose script s, and edit t he variables list ed in Table 7- 1 t o reflect your sit uat ion.

Ta ble 7 - 1 . Va r ia ble s a n d va lu e s for logpr oc's w e b se r ve r scr ipt s Scr ipt batcher

Va r ia ble

Va lu e

$loguser

The nam e of t he web server user

$mcast_if

The nam e of t he int erface t hat can reach t he log host

$logroot

The hom e direct ory of t he web server user

- 386 -

Ta ble 7 - 1 . Va r ia ble s a n d va lu e s for logpr oc's w e b se r ve r scr ipt s Scr ipt

Va r ia ble

cleantmp $logroot copier

Va lu e The hom e direct ory of t he web server user

$loghost

The nam e of t he host where t he logs will collect

$logroot

The hom e direct ory of t he web server user

$loghost_logroot The direct ory on t he collect or host where t he logs will be collect ed $loghost _loguser

The user on t he log host who owns t he logs

$scp_prog

The full pat h t o t he scp program , plus any addit ional opt ions

$ssh_prog

The full pat h t o ssh, plus any opt ions

Then, m ake sure you have sat isfied all of t he dependencies for t hese program s: # perl -wc batcher; perl -wc cleantmp; perl -wc copier

The only dependency you likely won't have is IO::Socket::Multicast. I nst all it via t he / usr/ port s/ net / p5- I O- Socket - Mult icast port on FreeBSD syst em s or from t he CPAN sit e ( ht t p: / / www.cpan.org/ ) . Next , configure ht t pd.conf t o log t o t he batcher in parallel wit h norm al logging. Not e t hat t he batcher com m and line m ust include t he inst ance ( site, virtual, secure) and t ype ( access, error, ssl) of logging: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" "%{User-Agent}i\" \ \"%{Cookie}i\" %v" full CustomLog "|/home/www/bin/batcher site access" full ErrorLog

"|/home/www/bin/batcher site error"

You can adj ust t he LogFormat direct ive as necessary t o log t he inform at ion you or your log sum m arizat ion soft ware needs. Finally, rest art Apache and verify t hat t he bat chers are creat ing bat ches: # apachectl configtest # apachectl graceful # cd $wwwhome/logs/ # ls tmp

Should list error log files for each batcher instance

# ls work/0

Should list the working batches for each batcher instance

# ls save/0 a

Verify that batches have moved into the save directory after

- 387 -

five-minute batch interval # ls work/0

and that new batches are currently being created

7.7.3 Preparing the Log Host St art by creat ing a log user t o receive t he logs, com plet e wit h a hom e direct ory. Becom e t he log user and copy t he public key from t he web server int o ~ log/ .ssh/ aut horized_keys2. Then, as t he log user, creat e t he direct ories t he log collect ion t ools use: # su log % cd ~log % mkdir -p bin web/{work,save}/{0,1} web/tmp web/{current,archive}

7.7.4 Testing the Configuration From a web server ( as t he web server's user) , ssh t o t he log host m anually t o verify t he configurat ion of t he authorized_keys2: # su www % ssh loghost -l loguser date I f your com m and fails, check t hat t he perm issions on t hat file are set t o 600.

Then, run copier m anually t o verify t hat t he log files act ually m ake it t o t he log server. Wat ch your run out put on t he web server, t hen check t hat save/ 0 on t he log server cont ains t he newly copied logs. Once you're sat isfied wit h t hese m anual t est s, schedule a cron j ob t hat copies and cleans up log files. These j obs should run as t he web server user: # crontab -e -u www

----------------------------- cut here ----------------------------# copy the log files down to the collector host every 15 minutes 0,15,30,45 * * * * /home/www/bin/copier

# clean the tmp directory once an hour 0 * * * * /home/www/bin/cleantmp ----------------------------- cut here -----------------------------

- 388 -

Finally, wait unt il t he next copier run and verify t hat t he bat ches appear on t he log host .

7.7.5 Configuring Scripts on the Log Host You should now have several bat ches sit t ing in save/ 0 in t he log t ree. Each bat ch cont ains t he log lines collect ed over t he bat ch int erval ( by default , five m inut es) and has a filenam e indicat ing t he inst ance ( site, virtual, secure) , t ype ( access, error, ssl) , web server host , t im est am p indicat ing when t he bat ch was originally creat ed, and PI D of t he batcher process t hat creat ed each bat ch. Now, inst all t he log processing script s int o bin/ : # cp $srcdir/collector/{arclogs,cleantmp,collect} bin/

Edit t hem t o have valid pat hs for t heir new locat ion and any OS dependencies, as shown in Table 7- 2.

Ta ble 7 - 2 . Va r ia ble s a n d va lu e s for logpr oc's log h ost scr ipt s Scr ipt arclogs

Va r ia ble

Va lu e

$logroot

The locat ion of t he logs

$gzip_prog

The full pat h t o t he gzip binary

cleantmp

$logroot

The locat ion of t he logs

collect

$logroot

The locat ion of t he logs

$gzip_prog

The full pat h t o t he gzip binary

Again, m ake sure all dependencies are sat isfied: # perl -wc arclogs; perl -wc cleantmp; perl -wc collect

I f you don't have Time::ParseDate, t hen inst all it from t he / usr/ port s/ devel/ p5- Tim em odules port on FreeBSD or from CPAN. Run collect m anually as t he log user t o verify t hat t he log bat ches get collect ed and t hat log dat a ends up in t he appropriat ely dat ed log file. Once you're sat isfied, aut om at e t hese t asks in a cron j ob for t he log user: # crontab -e -u log

----------------------------- cut here ----------------------------# run the collector once an hour 0 * * * * /home/log/bin/collect

# clean the tmp directory once an hour

- 389 -

0 * * * * /home/log/bin/cleantmp ----------------------------- cut here -----------------------------

Wait unt il t he next collect run and verify t hat t he bat ches are properly collect ed. Com pare t he collect ed log files wit h t he cont ent s of your old logging m echanism 's log file on t he web servers. Make sure every hit m akes it int o t he collect ed log files for t he day. You m ight want t o run bot h logging m echanism s for several days t o get a good feel t hat t he syst em is working as expect ed.

7.7.6 Viewing Live Log Data The log server program s provide addit ional t ools for m onit oring and sum m arizing live log dat a. On a t radit ional single web server environm ent , you can always tail t he log file t o see what 's going on. This is no longer easy t o do, because t he logs are now writ t en in sm all bat ches. ( Of course, if you have m ult iple web servers, m ult iple tail processes would have t o run on each web server.) The batcher process helps wit h t his by m ult icast ing t he logs out t o a m ult icast group. Use t he logwatch t ool on t he log server t o view t he live log dat a: % cd ~log/bin % ./logwatch

On a high- volum e web sit e, t here is likely t o be t oo m uch dat a t o scan m anually. logwatch accept s argum ent s t o specify which t ype of log dat a you want t o see. You can also specify a Perl regular expression t o lim it t he out put . The meter script wat ches t he log dat a on t he m ult icast st ream , in real t im e, and sum m arizes som e inform at ion about t he log dat a. I t also st ores inform at ion in an RRDTool ( ht t p: / / www.rrdt ool.org/ ) dat abase. The m ining direct ory cont ains a checklog script t hat produces a " t op t en client s" and " t op t en vhost s" report . Alt ernat ively, you can feed t he collect ed log files t o your exist ing web server log processing t ools.

7.7.7 See Also •

The logproc web sit e ( ht t p: / / www.pet erson.at h.cx/ ~ j lp/ soft ware/ logproc.t ar.gz)

- 390 -

Hack 75 Script User Interaction

Use a n e x pe ct scr ipt t o h e lp u se r s ge n e r a t e GPG k e ys. There are occasions when you can t ake advant age of Unix's flexibilit y t o cont rol som e ot her t ool or syst em t hat is less flexible. I 've used Unix script s t o updat e dat abases on userunfriendly m ainfram e syst em s when t he alt ernat ive was an expensive m ainfram eprogram m ing service cont ract . You can use t he sam e approach in reverse t o let t he user int eract wit h a t ool, but wit h a const rained set of choices. The Expect script ing language is ideal for creat ing such int eract ive script s. I t is available from Net BSD pkgsrc as pkgsrc/ lang/ t cl- expect or pkgsrc/ lang/ t k- expect , as well as from t he FreeBSD port s and OpenBSD packages collect ions. We'll use t he com m and- line version for t his exam ple, but keep in m ind t hat expect-tk allows you t o provide a GUI front end t o a com m and- line process if you're willing t o writ e a m ore com plex script . I n t his case, we'll script t he generat ion of a GPG key. I nst all GPG from eit her pkgsrc/ securit y/ gnupg or t he appropriat e port or package.

7.8.1 The Key Generation Process During t he process of generat ing a GPG key, t he program asks t he user several quest ions. We m ay wish t o im pose const raint s so t hat a set of users ends up wit h keys wit h sim ilar param et ers. We could t rain t he users, but t hat would not guarant ee correct result s. Script ing t he generat ion m akes t he process easier and elim inat es errors. First , let 's look at a t ypical key generat ion session: % gpg --gen-key gpg (GnuPG) 1.2.4; Copyright (C) 2003 Free Software Foundation, Inc. This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the file COPYING for details.

Please select what kind of key you want: (1) DSA and ElGamal (default) (2) DSA (sign only) (4) RSA (sign only) Your selection? 4 What keysize do you want? (1024) 2048 Requested keysize is 2048 bits Please specify how long the key should be valid.

- 391 -

0 = key does not expire

= key expires in n days

w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 0 Key does not expire at all Is this correct (y/n)? y

You need a User-ID to identify your key; the software constructs the user id from Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) "

Real name:

Let 's pause t here t o consider t he elem ent s we can const rain. You probably want t o specify t he crypt ographic algorit hm and key lengt h for all users consist ent ly, based on your securit y and int eroperabilit y requirem ent s. I 'll choose RSA signing and encrypt ion keys, but GPG doesn't provide a m enu opt ion for t hat . I 'll have t o creat e t he signing key first and t hen add t he encrypt ion subkey.

7.8.2 A Simple Script Here's an expect script t hat would duplicat e t he session shown so far: #!/usr/pkg/bin/expect -f

set timeout -1 spawn gpg --gen-key match_max 100000 expect "(4) RSA (sign only)" expect "Your selection? " send "4\r" expect "What keysize do you want? (1024) " send "2048\r"

- 392 -

expect "Key is valid for? (0) " send -- "0\r" expect "Key does not expire at all" expect "Is this correct (y/n)? " send -- "y\r" expect "Real name: "

The script begins by sett ing timeout t o infinit e, or -1, so expect will wait forever t o m at ch t he provided input . Then we spawn t he process t hat we're going t o cont rol, gpg --gen-key. match_max set s som e buffer size const raint s in byt es, and t he given value is far m ore t han we will need. Aft er t he init ial set t ings, t he script sim ply consist s of st rings t hat we expect from t he program and st rings t hat we send in reply. This m eans t hat t he script will answer all of t he quest ions GPG asks unt il Real name: , wit hout wait ing for t he user's input . Not e t hat in several places we expect t hings besides t he prom pt . For exam ple, before responding t o t he Your selection? prom pt , we verify t hat t he version of GPG we have execut ed st ill has t he sam e m eaning for t he fourt h opt ion, by expect ing t hat t he t ext of t hat m enu choice is st ill RSA (sign only). I f t his were a real, product ion- ready script , we should print a warning m essage and t erm inat e t he script if t he value does not m at ch our expect at ions, and perhaps include a check of t he GPG version num ber. I n t his sim ple exam ple, t he script will hang, and you m ust break out of it wit h Ct rl- c.

7.8.3 Adding User Interaction There are several ways of handling t he fields we do want t he user t o provide. For t he great est degree of cont rol over t he user experience, we could use individual expect com m ands, but here we will t ake a sim pler approach. Here's som e m ore of t he script : interact "\r" return send "\r" expect "Email address: " interact "\r" return send "\r" expect "Comment: " interact "\r" return send "\r" expect "Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? " interact "\r" return send "\r" expect "Enter passphrase: "

- 393 -

interact "\r" return send "\r" expect "Repeat passphrase: " interact "\r" return send "\r"

The interact com m and allows t he user t o int eract direct ly wit h t he spawned program . We place a const raint t hat t he user's int eract ion ends as soon as t he user presses t he Ent er key, which sends t he carriage ret urn charact er, \r. At t hat point , t he interact com m and ret urns and t he script resum es. Not e t hat we have t o send t he \r from t he script ; expect int ercept ed t he carriage ret urn and GPG did not see it .

7.8.4 Handling Incorrect Input Again, a correct script would have a m ore com plex flow of execut ion and allow for cases where t he spawned program rej ect s t he user's input wit h an error m essage. For exam ple, t he Real Name field m ust be m ore t han five charact ers long. I f a user t ypes less t han five charact ers, GPG will prom pt him t o ret ype his usernam e. However, t he expect script j ust shown will not accept t he new user input , because it is now wait ing for t he Email address: prom pt . Alt ernat ively, we could replace t hese t hree lines: interact "\r" return send "\r" expect "Email address: "

wit h: interact -o "Email address: " return send_user "Email address: "

I nst ead of st opping int eract ion when t he user presses return, we st op int eract ion when t he program out put s t he Email address: prom pt . That 's t he difference bet ween interact and interact -o; t he form er st ops int eract ion based on input from t he user, and t he lat t er on out put from t he program . This t im e, we don't need t o send t he carriage ret urn, because t he user's keypress is passed t hrough t o GPG. However, we do need t o echo t he prom pt , because expect has consum ed it . This m et hod let s GPG handle t he error condit ions for us: Real name: abc Name must be at least 5 characters long Real name: abcde Email address:

- 394 -

7.8.5 Hacking the Hack Aft er GPG receives t he inform at ion it needs t o generat e t he key, it m ight not be able t o find enough high- qualit y random dat a from t he syst em . The script ought t o handle t hat by spawning a process t o generat e m ore syst em act ivit y, such as perform ing a lot of disk act ivit y by running a find across t he ent ire disk. Aft er generat ing t he signing key, t he script could spawn a new inst ance of GPG wit h t he -edit-key opt ion, t o generat e t he desired RSA encrypt ion key. Alt hough t he final script m ay end up execut ing t hree processes, t he whole process is seam less t o t he user. You can hide even m ore of t he gut s by using expect's log_user set t ing t o hide t he out put of t he program s at point s where t he user does not need t o see t hem . You can use a script like t his in conj unct ion wit h any Unix com m and- line program . By com bining expect wit h telnet or ssh, you can cont rol non- Unix syst em s, t hereby leveraging t he flexibilit y of Unix int o a non- Unix dom ain. This even works wit h program s for which you do not have source code, such as cont rol ut ilit ies for com m ercial dat abases or applicat ion soft ware. I n t he case of GPG, we do have source code, so we could m odify t he program , but writ ing an expect script is easier. A carefully designed expect script m ay not require changes when a new version of GPG is released. Source code changes t o GPG would require int egrat ion wit h any new version of GPG.

7.8.6 See Also • • •

man expect The expect web sit e, which includes sam ple script s ( ht t p: / / expect .nist .gov/ ) Exploring Expect , by Don Libes, t he aut hor of expect ( ht t p: / / www.oreilly.com / cat alog/ expect / )

- 395 -

Hack 76 Create a Trade Show Demo

I frequent ly represent Net BSD at t rade shows. I t 's challenging t o at t ract at t ent ion because t here are m any boot hs at a show—people will walk by quickly unless som et hing cat ches t heir eye. You also need t o balance eye- candy wit h funct ionalit y so t hat you can at t ract and keep a visit or's at t ent ion. I needed an ent icing dem o t o run on one of t he com put ers in t he boot h. I want ed t o show off several applicat ions, such as office product ivit y t ools, video, and gam es, and have m usic playing, but t here's only so m uch screen real est at e. Cram m ing all of t hose t hings on t he screen at once would clut t er t he screen, and t he point would be lost . Most X window m anagers have som e concept of virt ual deskt ops, separat e work spaces t hat you can flip bet ween. For exam ple, Enlight enm ent ( pkgsrc/ wm / enlight enm ent ) not only has t he concept of virt ual deskt ops, but as an added bonus for t he t rade show environm ent offers a nice sliding effect as you t ransit ion from one deskt op t o t he next .

7.9.1 Introducing eesh Norm ally in Enlight enm ent , t o swit ch from one virt ual deskt op t o t he next , you m ove t he m ouse point er t o t he edge of t he screen and t hen push past it , or you use a key sequence t o m ove t o an adj acent deskt op. For an unat t ended dem o, we need t o aut om at e t his process. Enlight enm ent provides an undocum ent ed ut ilit y called eesh t hat can cont rol m ost aspect s of t he Enlight enm ent window m anager. You can writ e script s t o m ove windows, resize t hem , or flip bet ween deskt ops. Not e t hat eesh isn't a friendly ut ilit y; it doesn't even produce a prom pt when you run it . Type help for t he m enu or exit t o quit : % eesh help Enlightenment IPC Commands Help commands currently available: use "help all" for descriptions of each command use "help " for an individual description

actionclass

active_network

advanced_focus

autosave

background

border

button

button_show

colormod

configpanel

copyright

current_theme

tc

cursor

default_theme

dialog_ok

dok

dock

dump_mem_debug

exit

q

- 396 -

sfa

focus_mode

sf

fx

geominfo_mode

sgm

goto_area

sa

goto_desktop

sd

group

gc

group_info

gl

group_op

gop

help

?

imageclass

internal_list

il

list_class

cl

list_remember

list_themes

tl

module

move_mode

general_info

smm

nop

Unfort unat ely, t he eesh ut ilit y seem s t o be unt est ed. I t som et im es behaves inconsist ent ly by not accept ing com m ands unt il you ent er t hem a second t im e or by wit hholding out put unt il you press Ent er again. As an exam ple, t here are act ually m ore com m ands t han t hose indicat ed in t he help list ing. Look in t he Enlight enm ent source's ipc.c file for a com plet e list .

7.9.2 Discovering Commands We'll st art our script by m aking sure t hat Enlight enm ent is configured t he way we want for our dem o. We want six work spaces ( 3 by 2) t o display our program s. Wit hin eesh, t ry t he following com m ands: num_areas ? Number of Areas: 2 2 help num_areas Enlightenment IPC Commands Help : num_areas (sna) -------------------------------Change the size of the virtual desktop Use "num_areas " to change the size of the virtual desktop. Example: "num_areas 2 2" makes 2x2 virtual destkops Use "num_areas ?" to retrieve the current setting

num_areas 3 2

Now we have t he num ber of areas we want . areas is t he Enlight enm ent nam e for virt ual deskt ops, since Enlight enm ent also support s m ult iple deskt ops, but t hat 's different . Now we'd like our screen t o display t he first area, so t hat t he program s our script runs will open t here: goto_area 0 0

I f your t erm inal wasn't on t he first area, it j ust m oved off t he screen. Use t he m ouse t o ret urn t o t hat area.

- 397 -

eesh also let s us writ e com m ands on t he com m and line wit h t he -e ( execut e com m and) flag: % eesh -e "goto_area 0 0"

7.9.3 Sample Scripts Now we know enough t o writ e a sim ple dem o script : #!/bin/sh

eesh -e "num_desks 1" eesh -e "num_areas 3 2" sleep 1 eesh -e "goto_area 0 0"

# Configure the default gqmpeg playlist to play your desired music gqmpeg

# Show an interesting avi file. xanim -geometry +50x+10 netbsd3.avi &

# Give the programs time to start, to make sure they # open on the correct area. # Also, lets people watching see what started up. sleep 3 eesh -e "goto_area 1 0"

# Word Processing abiword sampledoc.abw & sleep 2 eesh -e "goto_area 2 0"

# Spreadsheet gnumeric samplesheet.gnumeric &

- 398 -

sleep 2 eesh -e "goto_area 0 1"

# A lively game battleball & sleep 2 eesh -e "goto_area 1 1"

# Web Browsing (of a local hierarchy, in case you don't have net # connectivity at a trade show) firebird file://index.html & sleep 3 eesh -e "goto_area 2 1" sleep 1

# Insert your favorite application here # Leave screen back at page 1. eesh -e "goto_area 0 0"

When you run t he script , t he screen will slide around t o t he various areas and pause a few seconds bet ween program launches. We have m ost of t he t hings we want ed: m usic, video, and applicat ions. The next st ep is t o keep it m oving. Try t he following script : #!/bin/sh while [ 1 ] do eesh -e "goto_area 0 0" sleep 2 eesh -e "goto_area 1 0" sleep 2 eesh -e "goto_area 2 0" sleep 2 eesh -e "goto_area 0 1" sleep 2

- 399 -

eesh -e "goto_area 1 1" sleep 2 eesh -e "goto_area 2 1" sleep 2 done

To st op t he m oving display, you have t o get your keyboard focus int o t he xterm where t he script is running so t hat you can press Ct rl- c. That can be difficult , but we'll address it short ly.

7.9.4 More Complex Scripts For a com plex dem onst rat ion, you can have different set s of t hese script s t hat visit different set s of areas. You can also change t he delay so t hat com plex areas display for a longer period. I also m ade a script t hat clears all of t he viewing areas. That way, when visit ors t o t he boot h play around wit h t he m achine, I can easily reset t o a clean st at e and t hen st art t he dem o again. Since m any of t he ut ilit ies you'll dem onst rat e don't creat e .pid files, I find it easiest t o use pkill, t he " kill process by nam e" ut ilit y. ( FreeBSD provides killall.) I 'll also leave you wit h t wo exam ple script s t hat show how t o ext ract inform at ion about Enlight enm ent 's current set t ings for use in a m ore com plex script . The first script is retitle: #!/bin/sh

WIN=`eesh -ewait "set_focus ?" | sed 's/^focused: //' ` xterm -geometry 47x7+227+419 -fn -*-courier-*-o-*-*-34-*-*-*-*-*-*-* -e \ /home/david/bin/retitle2 $WIN

The second is retitle2: #!/bin/sh WIN=$1 echo "enter new title:" read TITLE eesh -e "win_op $WIN title $TITLE"

Wit h t hese script s and e16keyedit , you can bind a key com binat ion t o change t he t it le of any window. This m akes it m uch easier t o keep t rack of xterms, if you prefer t ask- orient ed t it les.

- 400 -

Now back t o t he cont rol issue. When I first wrot e t his dem o, I used a swit ch wired t o a serial port t o st art and st op t he dem o so t hat keyboard focus did not m at t er. However, wiring swit ches is m ore work t han configuring soft ware, so I found a bet t er way. The e16keyedit ut ilit y, writ t en by Geoff " Mandrake" Harrison and Carst en " Rast er" Hait zler ( t he prim ary developers of Enlight enm ent ) , allows you t o bind funct ion keys and Met a keys t o run program s or perform t he sam e funct ions t hat you can wit h eesh. Using e16keyedit, you can define funct ion keys t o set up t he dem o, clean up t he dem o, and st art and st op t he area rot at ions. Since t he funct ion keys can be bound t o work anywhere wit hin Enlight enm ent , keyboard focus no longer m at t ers. You're ready t o give a fant ast ic dem o! e16keyedit is not part of t he m ain Enlight enm ent dist ribut ion. Download it from SourceForge ( ht t p: / / sourceforge.net / proj ect / showfiles.php?group_id= 2) .

7.9.5 See Also •

The Enlight enm ent web sit e ( ht t p: / / www.enlight enm ent .org/ )

- 401 -

Chapter 8. Keeping Up-to-Date I nt roduct ion Sect ion 77. Aut om at ed I nst all Sect ion 78. FreeBSD from Scrat ch Sect ion 79. Safely Merge Changes t o / et c Sect ion 80. Aut om at e Updat es Sect ion 81. Creat e a Package Reposit ory Sect ion 82. Build a Port Wit hout t he Port s Tree Sect ion 83. Keep Port s Up- t o- Dat e wit h CTM Sect ion 84. Navigat e t he Port s Syst em Sect ion 85. Downgrade a Port Sect ion 86. Creat e Your Own St art up Script s Sect ion 87. Aut om at e Net BSD Package Builds Sect ion 88. Easily I nst all Unix Applicat ions on Mac OS X

- 402 -

Introduction One of t he dist inguishing charact erist ics of t he BSDs is t he ease wit h which you can keep your operat ing syst em source and inst alled software up- t o- dat e. I n fact , each of t he BSDs provides m ult iple alt ernat ives, allowing users t o choose t he approaches t hat best m at ch t heir t im e and bandwidt h requirem ent s. This chapt er provides a plet hora of ways t o m aint ain an updat ed syst em . While m any are writ t en from t he FreeBSD perspect ive, don't let t hat st op you from hacking your own cust om ized Net BSD or OpenBSD solut ions. I n fact , t his chapt er concludes wit h one user dem onst rat ing how t o enj oy t he benefit s of t he BSD port s and packages collect ions on Mac OS X!

- 403 -

Hack 77 Automated Install

I f you 'r e r e spon sible for in st a llin g m u lt iple syst e m s, h ope fu lly you 've discove r e d t h e a r t of a u t om a t ing in st a lls. Most operat ing syst em s have som e sort of script ing m echanism t hat allows you t o predefine t he answers t o t he quest ions asked by t he inst all program . Once you've st art ed t he act ual inst all, you can leave and ret urn t o a fully inst alled syst em . The alt ernat ive is t o sit t here, answering every prom pt when it appears. No, t hank you! Even as a hom e user, it 's well wort h your while t o spend a few m inut es cust om izing t he inst all script t hat com es wit h FreeBSD. Try t his hack once and you'll never want t o sit and wat ch an inst all again.

8.2.1 Preparing the Install Script Before inst alling any syst em , you need t o know t he following: • • • • •

The I P set t ings and host nam e of t he host you're inst alling The FreeBSD nam e of t hat host 's NI C Which dist ribut ions, or part s of t he OS, t o inst all Your desired part it ioning schem e Which packages ( applicat ions) t o inst all

Of course, it 's always a good idea t o record t his inform at ion and include it wit h t he docum ent at ion for t he syst em . FreeBSD's inst all m echanism lives in / st and/ sysinst all. Not surprisingly, man sysinstall describes all of t he script able bit s of t his program . I 'll go over som e useful param et ers, but you'll definit ely want t o skim t hrough t he m anpage t o see if t here are addit ional param et ers suit ed t o your part icular environm ent . FreeBSD also com es wit h a com m ent ed, ready- t o- cust om ize inst all script , locat ed in / usr/ src/ usr.sbin/ sysinst all/ inst all.cfg. Copy t his file, t hen edit t he copy in your favorit e edit or. St art by insert ing your own net work set t ings: # This is the installation configuration file for my test machine, # crate.cdrom.com. # It is included here merely as a sort-of-documented example. # # $FreeBSD: src/usr.sbin/sysinstall/install.cfg,v 1.11 2001/09/06 10:04:27 murray Exp $

# Turn on extra debugging. debug=yes

- 404 -

################################ # My host specific data hostname=crate.cdrom.com domainname=cdrom.com nameserver=204.216.27.3 defaultrouter=204.216.27.228 ipaddr=204.216.27.230 netmask=255.255.255.240 ################################

Replace t he exam ple net work inform at ion wit h t he nam e and I P set t ings associat ed wit h t he specific host you'd like t o inst all. I f you're using DHCP t o obt ain t his inform at ion, fill in t he host nam e line and replace t he ot her lines wit h: tryDHCP=YES

Next , replace t he nam e of t he NI C and t he pat h t o t he FTP sit e. I n t his exam ple, t he NI C is rl0 and I 'm using t he default FTP sit e: ################################ # Which installation device to use _ftpPath=ftp://ftp.freebsd.org/pub/FreeBSD/ netDev=rl0 mediaSetFTP ################################

Next com e t he desired dist ribut ions. ( See man sysinstall for m ore det ails.) I nclude t hem all on t he one dists= line, separat ed by a space: ################################ # Select which distributions we want. dists=bin doc games manpages dict compat4x ports src sbase ssys Xbin Xcfg \ Xdoc Xlib Xman Xset Xfnt Servers/XS3V Xfsrv distSetCustom ################################

- 405 -

Not e t hat distSetCustom allows you t o cust om ize which dist ribut ions t o inst all. I f you'd like t o inst all t he works, use distSetEverything and don't specify any dists=.

The part it ioning schem e sect ion is very im port ant . I f you don't want t o use t he default schem e which uses t he ent ire disk, read t his sect ion of t he m anpage carefully. Also, t he default file gives exam ples for t hree disks. Make sure you rem ove t he exam ples and replace t hem wit h your own part it ioning schem e. The following exam ple is t he equivalent of choosing a for " all," follow ed by a for " aut o default s" : ############################################################# # Set the parameters for the partition editor # ad = IDE, da = SCSI disk=ad0 partition=exclusive diskPartitionEditor

############################################################# # - All sizes are expressed in 512 byte blocks! # - "Size in MB" = sectors * 512 / 1024 / 1024 # - "Number of blocks" = xsize in mb * 1024 * 1024 / 512 # The non-zero value after the mountpoint means enable soft updates

# 256MB UFS ad0s1a ad0s1-1=ufs 524288 /

# 240MB SWAP ad0s1b ad0s1-2=swap 491520 none

# 256MB UFS ad0s1d ad0s1-3=ufs 524288 /var

# 256MB UFS ad0s1e

- 406 -

ad0s1-4=ufs 524288 /tmp

# Rest of FreeBSD partition ad0s1f ad0s1-5=ufs 0 /usr

diskLabelEditor

# runs diskLabelCommit diskPartitionWrite installCommit

Finally, list which applicat ions you would like t o inst all. List each package on it s own line, followed by t he packageAdd com m and: # Install some packages at the end. package=fetchmail-6.2.0 packageAdd package=pine-4.55 packageAdd package=lynx-2.8.5d14 packageAdd

The FreeBSD package list ( ft p: / / ft p.freebsd.org/ pub/ FreeBSD/ releases/ i386/ 5.1RELEASE/ packages/ All) has t he exact nam es of each available package. Replace i386/ 5.1RELEASE wit h your plat form and desired operat ing syst em version.

8.2.2 Test-Drive Now t hat you've creat ed a cust om ized version of inst all.cfg, prepare a freshly form at t ed UFS floppy: # fdformat -f 1440 /dev/fd0 # bsdlabel -w /dev/fd0 fd1440 # newfs /dev/fd0

Once t he floppy is ready, copy inst all.cfg ont o it . On a t est syst em , st art t he inst all process eit her by boot ing from a FreeBSD CD- ROM/ DVD or wit h t he t wo inst all floppies. When you receive t he sysinstall Main Menu screen, choose Load Config. I nsert t he floppy cont aining your cust om ized inst all.cfg and press OK.

- 407 -

Once t he configurat ion file has been loaded, you'll receive t he m essage You may remove the floppy from floppy drive unit A. While t his is m eant t o be an unat t ended inst all, you should be present during your first t est inst all. This will give you t he opport unit y t o ensure t hat your script runs sm oot hly, wit hout hanging at any port ion of t he inst all. I f it does hang, check your inst all.cfg for a t ypo in t hat sect ion. Once t he inst all is com plet e, you'll ret urn t o t he sysinstall Main Menu. At t his point , you can eit her configure t he syst em int eract ively by choosing Configure or use a prepared post - configurat ion script , as found in / usr/ doc/ en_US.I SO8859- 1/ art icles/ pxe/ post . inst all.cfg is not responsible for post - inst all configurat ion.

Once you're happy wit h your floppy, label it wit h your operat ing syst em version. St ore it where you can find it t he next t im e you're ready t o inst all a version of t hat operat ing syst em .

8.2.3 See Also • •

man sysinstall / usr/ src/ usr.sbin/ sysinst all/ inst all.cfg ( t he sam ple inst allat ion configurat ion file)

- 408 -

Hack 78 FreeBSD from Scratch

For t h ose w ho pr e fe r t o w ipe t h e ir disk s cle a n be for e t h e y u pgr a de t he ir syst e m s. Have you ever upgraded your syst em wit h make world? I f you have only one syst em on your disks, you m ay run int o a problem : if t he installworld fails part way t hrough, you m ay end up wit h a broken syst em t hat m ight not even boot . I t 's also possible t hat t he installworld will run sm oot hly, but t he new kernel will not boot . What if you're like m e and believe in t he " wipe your disks when upgrading syst em s" paradigm ? Reform at t ing ensures t here is no old cruft left lying around. I t also m eans you have t o recom pile or reinst all all your port s and packages and t hen redo all your carefully craft ed configurat ion t weaks. FreeBSD From Scrat ch solves all t hese problem s. The st rat egy is sim ple: use a running syst em t o inst all a new syst em under an em pt y direct ory t ree, m ount ing new part it ions in t hat t ree as appropriat e. Many config files can copy st raight across, and mergemaster can t ake care of t hose t hat cannot . You can perform arbit rary post - configurat ion of t he new syst em from wit hin t he old syst em , up t o t he point where you can chroot t o t he new syst em . This upgrade has t hree st ages, where each st age eit her runs a shell script or invokes make:

st age_1.sh Creat es a new boot able syst em under an em pt y direct ory, m erges or copies as m any files as are necessary, and t hen boot s t he new syst em

st age_2.sh I nst alls your desired port s

st age_3.m k Does post - configurat ion for soft ware inst alled in t he previous st age From now on, whenever you feel like an updat e is in order, sim ply t oggle t he part it ions you want t o wipe and reinst all. While com piling t he port s during st age t wo, t he syst em will not be available for it s usual dut ies. I f you run a product ion server, consider t he downt im e caused by st age t wo. I f t im e is an issue, consider using precom piled packages inst ead of port s.

- 409 -

8.3.1 Stage One: System Installation This hack uses several script s and configurat ion files t hat you can download from t he original docum ent 's sit e ( list ed in t his hack's Sect ion 8.3.4 sect ion) . Also, if you keep your docs up- t o- dat e wit h cvsup, t he script s and original docum ent can be found in / usr/ doc/ en_US.I SO8859- 1/ art icles/ fbsd- from - scrat ch. The script for st age one is st age_1.sh. When run wit h exact ly one argum ent : # ./stage_1.sh default

it will read it s configurat ion from st age_1.conf.default and writ e a log t o st age_1.log.default . You'll need t o cust om ize st age_1.conf.default t o m at ch your idea of t he perfect syst em . I have t ried t o com m ent all of t he sect ions you should adapt . I n addit ion t o t he cust om ized sect ions, t he configurat ion script m ust provide four shell funct ions: • • • •

create_file_systems create_etc_fstab copy_files all_remaining_customization

Before you run st age_1.sh, m ake sure you have com plet ed t he usual t asks in preparat ion for make installworld/installkernel: • • •

Configure your kernel config file. Com plet e make buildworld. Com plet e make buildkernel KERNCONF=whatever.

The st age_1.sh script will st op at t he first com m and t hat fails, so you cannot overlook errors. I t will also st op if you use an unset environm ent variable, which is probably due t o a t ypo. Answer no or press Ent er when mergemaster asks if whet her should delet e / var/ t m p/ t em proot .st age1. This direct ory cont ains som e files t hat m ust be copied t o t he new syst em lat er. *** Comparison complete Do you wish to delete what is left of /var/tmp/temproot.stage1? [no] no

Aft er t hat , it will list t he files it inst alled: *** You chose the automatic install option for files that did not exist on your system.

The following were installed for you:

/newroot/etc/defaults/rc.conf ... /newroot/COPYRIGHT

- 410 -

(END)

Type q t o quit t he pager. Then, you'll have t o deal wit h login.conf: *** You installed a login.conf file, so make sure that you run '/usr/bin/cap_mkdb /newroot/etc/login.conf' to rebuild your login.conf database

Would you like to run it now? y or n [n]

The answer does not m at t er, since we will run cap_mkdb in eit her case. You can download t he aut hor's st age_1.conf.default , which you'll need t o m odify subst ant ially. The com m ent s should give you enough inform at ion regarding what t o change. Pay at t ent ion t o t he newfs com m ands. While you cannot creat e new filesyst em s on m ount ed part it ions, t he script will happily erase any unm ount ed part it ions. This can be enough t o ruin your day, so be sure t o m odify t he device nam es t o m at ch your scenario. Running t his script inst alls a syst em t hat , when boot ed, provides inherit ed users and groups, firewalled I nt ernet connect ivit y over Et hernet and PPP, correct t im e zone set t ings and NTP, and m ore m inor configurat ions, such as / et c/ t t ys and / et c/ inet d.conf. Ot her areas of configurat ion will not work unt il st age t wo com plet es. For exam ple, we have copied files t o configure print ing and X11. Print ing, however, needs applicat ions not found in t he base syst em . Sim ilarly, X11 will not run before we have com piled t he server, libraries, and program s.

8.3.2 Stage Two: Ports Installation I t is possible t o inst all precom piled packages at t his st age inst ead of com piling port s. I n t his case, st age_2.sh will be not hing m ore t han a script ed list of pkg_add com m ands. I inst all m y favorit e port s via t he downloadable st age_2.sh script . You can run it m ult iple t im es safely, as it will skip all port s t hat are already inst alled. I t also support s t he dry run opt ion ( -n) , which will show what would be done. Run it like st age_1.sh, wit h exact ly one argum ent t o denot e a config file: # ./stage_2.sh default

This exam ple will read t he list of port s from st age_2.conf.default . The act ual list of port s consist s of lines wit h t wo or m ore space- separat ed words: t he cat egory and t he port , opt ionally followed by an inst allat ion com m and t hat will com pile and inst all t he port . By default , t his is make install. Most of t he t im e, it suffices t o nam e only t he cat egory and port . You can fine- t une som e port s by specifying make variables, as found in t he port 's Makefile: www mozilla make WITHOUT_MAILNEWS=yes WITHOUT_CHATZILLA=yes install

- 411 -

mail procmail make BATCH=yes install

I n fact , you can specify arbit rary shell com m ands, so you are not rest rict ed t o sim ple make invocat ions: java linux-sun-jdk14 yes | make install news inn-stable CONFIGURE_ARGS="--enable-uucp-rnews --enable-setgid-inews" \ make install

Not e t hat t he line for news/inn-stable includes an exam ple of a one- shot shell variable assignm ent t o CONFIGURE_ARGS. The port 's Makefile will use t his as an init ial value and augm ent som e ot her essent ial args. The difference bet ween specifying a make variable on t he com m and line ( as in t he last exam ple) and t he following: news inn-stable make CONFIGURE_ARGS="--enable-uucp-rnews \ --enable-setgid-inews" install

is t hat t he lat t er will override inst ead of augm ent . Be careful t hat your port s do not use an int eract ive inst all; t hey should not t ry t o read from st din. I f t hey do, t hey will read t he next line or lines from your list of port s and get confused. I f st age_2.sh m yst eriously skips a port or st ops processing, t his is likely t he reason.

Finally, t his script will creat e a log file nam ed LOGDI R/ cat egory+ port for each port it inst alls. When you download t he st age_2.sh script , you m ay want t o m odify t hese variables at t he beginning of t he script t o reflect your environm ent : DBDIR="/var/db/pkg" PORTS="/usr/ports" LOGDIR="/home/root/setup/ports.log"; mkdir -p \ ${LOGDIR}

8.3.3 Stage Three: Post-Configuration You inst alled your beloved port s during st age t wo, but som e port s require a lit t le bit of configurat ion. This is t he j ob of st age t hree, t he post - configurat ion st age. I have chosen t o im plem ent st age t hree as a Makefile because t his allows easy select ion of what you want t o configure sim ply by running:

- 412 -

# make -f stage_3.mk target

As wit h st age_2.sh, m ake sure you have st age_3.m k available aft er boot ing t he new syst em , eit her by put t ing it on a shared partit ion or by copying it som ewhere on t he new syst em . Aut om at ing t he inst allat ion of a port m ay prove difficult if it is int eract ive and does not support make BATCH=YES install. For a few port s, t he int eract ion is not hing m ore t han t yping yes when asked t o accept som e license. I f such input is read from t he st andard input , we sim ply pipe t he appropriat e answers t o t he inst allat ion com m and, usually make install. This is how I dealt wit h j ava/ linux- sun- j dk14 in t he previous exam ple. This st rat egy, however, does not work for edit ors/ st aroffice52, which requires t hat X11 is running. The inst allat ion procedure involves a fair am ount of clicking and t yping, so it cannot be aut om at ed like ot her port s can. However, t he following workaround does t he t rick for m e. First , I creat ed a st aroffice package on t he old syst em wit h: # cd /usr/ports/editors/staroffice52 # make package =

==>

Building package for staroffice-5.2_1

Creating package /usr/ports/editors/staroffice52/staroffice-5.2_1.tbz Registering depends:. Creating bzip'd tar ball in '/usr/ports/editors/staroffice52/staroffice-5.2_1.tbz'

During st age t wo, I used pkg_add t o add t his package: # pkg_add /usr/ports/editors/staroffice52/staroffice-5.2_1.tbz

Upgrading Configuration Files Be aware of upgrade issues for config files. I n general, you do not know when and if t he form at or cont ent s of a config file changes. A new group m ay be added t o / et c/ group, or / et c/ passwd m ay gain anot her field. Sim ply copying a config file from t he old t o t he new syst em m ay be enough m ost of t he t im e, but in t hese cases it is not . Unfort unat ely, mergemaster is available only for base syst em files, not for anyt hing inst alled by port s. All you can do is be alert , especially when t he m aj or version num ber bum ps. All act ively m aint ained soft ware program s are prim e candidat es for config file scrut iny. To det ect such silent changes, I keep a copy of t he m odified config files in t he sam e place where I keep st age_3.m k and com pare t he result wit h a make rule. For exam ple, I exam ine Apache's ht t pd.conf in t arget config_apache wit h: # ... automated httpd.conf modifications here ... @if ! cmp -s /usr/local/etc/apache2/httpd.conf httpd.conf; then \ echo "ATTENTION: the httpd.conf has changed. Please examine if"; \ echo "the modifications are still correct. Here is the diff:"; \

- 413 -

diff -u /usr/local/etc/apache2/httpd.conf httpd.conf; \ fi

I f t he diff is innocuous, I can m ake t he m essage go away wit h cp /usr/local/etc/apache2/httpd.conf httpd.conf. See [ H a ck # 9 2 ] for m ore on t his st rat egy.

The downloadable st age_3.m k will give you an idea of how t o aut om at e all reconfigurat ion.

8.3.4 See Also •

" FreeBSD From Scrat ch" ( includes links t o t he script s) at ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ art icles/ fbsd- from scrat ch/ art icle.ht m l

- 414 -

Hack 79 Safely Merge Changes to /etc

Use a t h r e e - w a y m e r ge t o de a l w it h upgr a de d configu r a t ion file s. Even t hough you probably run cvsup on a daily basis, you likely run make world only a few t im es a year, whenever a new version of t he OS is released. The st eps required t o upgrade your syst em are well docum ent ed and fairly st raight forward. That is, it 's easy unt il it 's t im e t o run mergemaster. mergemaster is an im port ant st ep, as it int egrat es changes t o / et c. For exam ple, occasionally a core ut ilit y such as Sendm ail will require a new user or group in / et c/ passwd. Problem s can occur if t hose changes aren't int egrat ed. I f you've used mergemaster before, you know it 's not t he m ost user- friendly ut ilit y out t here. Misint erpret a diff, and you m ight lose your configurat ion file changes or, worse, m iss a necessary change. You m ight even end up blowing away your own users in / et c/ passwd—not t he m ost convenient way t o st art off a new upgrade.

8.4.1 Initial Preparations An alt ernat ive is t o use etcmerge ( / usr/ port s/ sysut ils/ et cm erge) . This ut ilit y does m ost of t he work for you. Unlike t he t wo- way diff used by mergemaster, t his ut ilit y can com pare t he changes bet ween t hree set s of edit s: • • •

The / et c from your original version of FreeBSD Any changes you've m ade t o / et c since t hen The / et c for your new version of FreeBSD Before any upgrade, you definit ely want a fresh, t est ed backup of all of your dat a, including / et c.

Once you've inst alled etcmerge, ensure you have a backup copy of / et c: # tar czvf etc.tgz /etc

Here, I 've saved a copy only t o t he local hard drive. Be sure t o copy it t o anot her locat ion as well, j ust t o be safe: t o anot her syst em , a rem ovable m edia, or even your em ail account . The next st ep is t o locat e a copy of / et c t hat is original t o your current operat ing syst em and save it t o / var/ db/ et c. ( This is a good st ep t o add t o your regim e when you inst all a new syst em .) Assum ing t his isn't a fresh inst all and you've m ade changes t o / et c, you can get t he original, unm odified / et c for your operat ing syst em version at ht t p: / / people.freebsd.org/ ~ eivind/ et c/ . Here, I 've downloaded t he 5.1- RELEASE version and unt arred it t o t he correct place:

- 415 -

# tar -C /var/db -zxvpf etc-5.1-RELEASE.tar.gz # ls /var/db/etc/

So, now you have a copy of t he original / et c, as well as your own cust om ized / et c. You'll receive t he / et c for a newer version of FreeBSD once you've changed your cvs- supfile t o reflect t he newer t ag [ H a ck # 8 0 ] . For exam ple, I 'm current ly running 5.1- RELEASE, so m y cust om supfile cont ains t his line: *default tag=RELENG_5_1_0_RELEASE

When I 'm ready t o upgrade t o 5.2, I 'll change t hat line t o reflect t he new t ag: *default tag=RELENG_5_2_0_RELEASE

My next cvsup will grab t he sources for t he new operat ing syst em version. None of t he changes t o / usr/ src will be int egrat ed unt il you make buildworld and make installworld as per t he inst ruct ions in t he handbook. Sim ply downloading t he changes does not upgrade your operat ing syst em .

Once cvsup has finished downloading all of t he changes, t ake t he t im e t o read / usr/ src/ UPDATI NG, which list s all of t he known got chas for t his release. For exam ple, t here m ay be m andat ory opt ions for t he kernel process of t he upgrade, cert ain st ages m ay require a reboot before t he next st age works, or perhaps direct ory st ruct ures such as / et c have seen m aj or changes. Once you've m ade your necessary preparat ions, ensure t hese st eps have succeeded before using etcmerge: • • • •

make make make make

buildworld buildkernel installkernel installworld

8.4.2 Using etcmerge Now t hat you have a new world, use etcmerge t o int egrat e any changes t o / et c. As per it s m anpage, st art wit h t he init ializat ion st ep: # etcmerge init

The script will perk along for a m om ent or t wo before producing a screen full of lines t hat st art wit h ETCMERGE. Here's t he beginning of t hat out put : ETCMERGE: >>>

Finding classes of files

ETCMERGE: >>>

Working from

- 416 -

ETCMERGE: >>>

Active:

/etc

ETCMERGE: >>>

Reference: /var/db/etc

ETCMERGE: >>>

New:

/root/etc-work/200401191624/etc-new

Not e t he nam e of t he direct ory in t he last line. I t cont ains t he working files t hat are ready for your review. You'll t hen receive lines for different classes—see man etcmerge for a descript ion of each conflict class. Here's a sam ple out put from a syst em I recent ly upgraded: ETCMERGE: >>>> Class 7:

3 conflict(s)

A class 7 conflict m eans a file exist ed for all t hree versions of / et c. Any differences will appear wit h diff- st yle m arkers. This part icular syst em has t hree files cont aining conflict s. Their nam es are in t he file called 7.conflict s: # more /root/etc-work/200401191624/7.conflicts ./manpath.config ./pwd.db ./spwd.db

The et c- m erged subdirect ory cont ains copies of t hose files wit h t he differences m arked. Look t here and exam ine each file list ed as cont aining conflict s: # cd /root/etc-work/200401191624/etc-merged # vi manpath.config Don't send pwd.db or spwd.db t o an edit or—t hese are t he dat abase versions of your password files. I nst ead, use diff t o see if t he conflict is because you've added users or because FreeBSD has added a new user: # diff etc-new/master.passwd /etc/master.passwd

Rem ove t he t wo .db lines from 7.conflict s m anually so etcmerge is aware t hat you've resolved t he conflict s t o your password dat abases.

As you review your own files, t he angle bracket m arkers indicat e which lines have changed. Next t o each angle bracket m arker is t he nam e of t he file cont aining t he conflict ing lines. For exam ple, if t he nam e of t he file includes t he / et c- new direct ory, t he lines in quest ion belong t o t he new version of t he file. Once you've decided which version of t he lines you wish t o keep, rem ove t he angle bracket lines as well as t he unwant ed version of t he lines. Once you're finished your edit s, t his com m and will int egrat e t hem :

- 417 -

# etcmerge install /etc/mail/aliases: 24 aliases, longest 10 bytes, 246 bytes total Install done - removing copies of old /etc/ and old reference. Done. #

Congrat ulat ions! You've successfully upgraded your operat ing syst em while m aint aining your cust om izat ions t o / et c.

8.4.3 See Also • • • • •

[ H a ck # 9 2 ] man mergemaster man etcmerge man build The makeworld sect ion of t he FreeBSD Handbook, which includes direct ions for using mergemaster ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO88591/ books/ handbook/ m akeworld.ht m l)

- 418 -

Hack 80 Automate Updates

Fr e e BSD pr ovide s m a n y t ools t o m a k e soft w a r e upgr a de s a s pa in le ss a s possible . I n fa ct , t he e n t ir e pr oce ss is fully scr ipt a ble . Sim ply ch oose t h e pie ce s you w a n t a n d h ow up- t o- da t e you w a n t t o be . End users and adm inist rat ors alike share a desire t o keep t heir operat ing syst em s and applicat ions as up- t o- dat e as possible. However, if you're an operat ing syst em s vet eran, you're well aware t hat t his desire doesn't always t ranslat e int o foolproof, easy execut ion. For exam ple, do you have t o scour t he far corners of t he I nt ernet t o find t he lat est updat es? Once you find t hem , is it possible t o upgrade safely wit hout overwrit ing t he dependencies required by ot her applicat ions?

8.5.1 Assembling the Pieces The cvsup process provides t he lat est updat es t o t he FreeBSD operat ing syst em , port s collect ion, and docum ent s collect ion. You no longer have t o scour t he I nt ernet looking for t he lat est sources. Sim ply run cvsup! Since our int ent ion is t o script t he whole process, inst all t he cvsup- wit hout - gui port : # cd /usr/ports/net/cvsup-without-gui # make install clean

I f you've never used cvsup before, t ake t he t im e t o read it s sect ion in t he FreeBSD Handbook so you have an overview of how t he process works. When t he inst all finishes, copy / usr/ share/ exam ples/ cvsup/ cvs- supfile t o a locat ion t hat m akes sense t o you ( e.g., / root or / usr/ local/ et c) . Use t he com m ent s in t hat file and t he inst ruct ions in t he handbook t o cust om ize t he file so it reflect s your closest m irror, operat ing syst em ( t ag) , and what you would like t o updat e. Here's m y cvs- supfile. I t uses a Canadian m irror and updat es all sources, port s, and docum ent s on a FreeBSD 5.1- RELEASE syst em : # more /root/cvs-supfile #use the Canadian mirror *default host=cvsup.ca.freebsd.org

#keep these lines as-is! *default base=/usr/local/etc/cvsup *default prefix=/usr

- 419 -

#this is a 5.1-RELEASE system *default tag=RELENG_5_1_0_RELEASE

#keep this line as-is! *default release=cvs delete use-rel-suffix compress

#update all src, ports, and docs src-all ports-all tag=. doc-all tag=. I f you want t o specify which source, port s, and docs t o inst all, see t he handbook for direct ions on creat ing a refuse file.

I f your cvs- supfile includes t he ports-all tag=. line, inst all portupgrade. This port will not only keep t rack of which port s need upgrading, it will also t rack dependencies and aut om at e t he ent ire applicat ion upgrade process: # cd /usr/ports/sysutils/portupgrade # make install clean

We can also t ake advant age of t he fastest-cvsup port . As t he nam e im plies, it looks for t he fast est cvsup m irror: # cd /usr/ports/sysutils/fastest-cvsup # make install clean

8.5.2 An Example Dry Run Wit h t he necessary pieces in place, let 's run t hem from t he com m and line t o see how t hey work. First , use cvsup t o download any changes t o t he operat ing syst em , soft ware, or docum ent s t ree: # cvsup -L2 /root/cvs-supfile Parsing supfile "/root/cvs-supfile" Connecting to cvsup.ca.freebsd.org Connected to cvsup.ca.freebsd.org Server software version: SNAP_16_1f

- 420 -

Negotiating file attribute support Establishing collection information Establishing multiplexed-mode data connection Running Updating collection src-all/cvs Updating collection ports-all/cvs

Updating collection doc-all/cvs

Shutting down connection to server Finished successfully

The -L2 swit ch t urns on verbosit y. Subst it ut e / root / cvs- supfile wit h t he locat ion of your cust om ized cvs- supfile. I t 's rare for src t o change. When it does, it is usually due t o a securit y pat ch. I f you not ice changes t o src, go t o ht t p: / / www.freebsd.org/ securit y/ # adv t o see if t he securit y incident affect s you and how t o apply t he pat ch if it does.

Once cvsup is com plet e, int egrat e t he changes t o t he port s and t he docum ent s t rees. This will t ake care of t he docum ent changes: # cd /usr/doc # make install You need t he docproj - noj adet ex port [ H a ck # 8 9 ] for t his com m and t o succeed.

For t he port s, first updat e your port s index: # cd /usr/ports # make index Generating INDEX-5 - please wait.. Done.

An alt ernat ive is t o inst ead run portsdb -Uu. Not e t hat if you've creat ed a refuse file, eit her com m and will produce a screen or t wo of error m essages. You can safely ignore t hese.

- 421 -

Once your port s t ree is up- t o- dat e, see if any of your inst alled applicat ions need upgrading: # portversion -l " cd pub/FreeBSD/development/CTM/ports-cur ftp> ls

- 433 -



-rw-r--r--

1 110

root

22332066 Jan 23 08:46 ports-cur.5100xEmpty.gz

-rw-r--r--

1 110

root

67953 Jan 24 00:43 ports-cur.5101.gz

-rw-r--r--

1 110

root

14256 Jan 24 16:51 ports-cur.5102.gz

Look t oward t he end of t he list ing for t he large file closest t o t he present dat e. I t will have t he word xEmpty in it s nam e. That file is your st art ing delt a. Download t hat and any subsequent delt as. ftp> get ports-cur.5100xEmpty.gz ftp> get ports-cur.5101.gz ftp> get ports-cur.5102.gz ftp> quit Your first ftp t ransfer will be t he largest and longest , as you are downloading t he elem ent s necessary t o build t he port s t ree st ruct ure. Subsequent sessions will be very quick.

Not e t he .gz ext ension; leave t he files com pressed. CTM will st ill work, and you'll save disk space. Save your delt as t o / usr/ port s, and rem ain in t his direct ory when you use t he ctm com m and. Now t hat you have your st art ing delt as, apply t hem wit h ctm: # ctm ports-cur.5100xEmpty.gz ctm: warning: .ctm_status not found

The first t im e you use ctm, it will com plain about a m issing .ct m _st at us file. Don't worry; it will creat e it for you. Aft er a few seconds, it will send a lot of out put t o st dout . Once t he com m and has finished, you'll have a fully inst alled version of t he port s t ree. That .ct m - st at us file will t ell you t he delt a num ber of t hat port s t ree: # more .ctm-status ports-cur 5100

- 434 -

Then, sim ply apply any subsequent delt as in ascending order. This will correct ly incorporat e all of t he changes t o t he port s t ree. # ctm ports-cur.5101.gz # ctm ports-cur.5102.gz # more .ctm-status ports-cur 5102

That 's it . Whenever you want t o updat e your port s t ree, ftp int o your CTM m irror, download t he delt as cont aining a higher num ber t han your current version, and apply t hem in order. I t 's up t o you whet her t o keep t he com pressed versions of t he files you download. Once you've successfully applied a delt a—as indicat ed by .ct m - st at us—you no longer need t o st ore t hat delt a file. However, if download speed or t im e is an issue, consider keeping a copy of t hat large st art ing delt a, j ust in case you ever want t o recreat e your port s t ree from scrat ch.

8.8.2 Hacking the Hack I f you're t oo lazy or forget ful t o ftp for changes periodically, consider receiving t hem aut om at ically via em ail. Changes occur once or t wice a day. Subscribe t o t he ct m - port s- cur m ailing list t o receive t hem ( ht t p: / / list s.freebsd.org/ m ailm an/ list info/ ct m - port s- cur/ ) . Com plet e t he online subscript ion form , and reply t o t he em ail t hat asks you t o confirm your subscript ion. However, do not subscribe t o t hat m ailing list unt il you've configured your syst em t o handle t hose em ails. Basically, you want t he syst em t o int ercept t hose CTM updat es inst ead of sending t hem direct ly t o your m ailbox. There are t wo ways t o do t his: eit her creat e a sendmail alias or creat e a procmail recipe. See man ctm_rmail for det ailed inst ruct ions. I t 's also a good idea t o verify t he PGP signat ures before applying t hose updat es. You can find det ailed inst ruct ions for t his, as well as for using ctm_rmail t o handle incom ing delt as, in t his m essage from t he ct m - users m ailing list : ht t p: / / list s.freebsd.org/ piperm ail/ ct m users/ 2003- Oct ober/ 000039.ht m l.

8.8.3 See Also • •

man ctm_rmail The CTM sect ion of t he FreeBSD Handbook ( ht t p: / / www.freebsd.org/ doc/ en_US.I SO8859- 1/ books/ handbook/ ct m .ht m l)

- 435 -

Hack 84 Navigate the Ports System

Use bu ilt - in com m a nds t o k e e p a br e a st of t h e Fr e e BSD por t s colle ct ion. What first at t ract ed m e t o FreeBSD—and what has definit ely kept m y at t ent ion since—is t he port s collect ion. Over 10,000 applicat ions are a m ere make install clean away. For a soft ware j unkie like m yself, it is indeed Nerdvana t o no longer scour t he I nt ernet for soft ware or fight m y way t hrough dependency hell j ust t o convince an applicat ion t o inst all. Adm it t edly, it 's easy t o get lost in a sea of port s. How do you choose which applicat ion best suit s your needs? How do you keep t rack of which port s have been inst alled on your syst em ? How do you m ake sure you don't inadvert ent ly delet e a dependency? Read on t o see how t o get t he m ost out of t he built - in ut ilit ies for m anaging port s.

8.9.1 Finding the Right Port You know you want t o inst all som e soft ware t o add funct ionalit y t o your syst em . Wouldn't it be great if you could generat e a list of all t he port s t hat are available for your specific need? Well, you can, and it 's alm ost t oo easy wit h t he built - in port search facilit y. I n t his exam ple, I 'll look for port s dealing wit h VPN soft ware: % cd /usr/ports % make search key=vpn | more Port:

poptop-1.1.4.b4_2

Path:

/usr/ports/net/poptop

Info:

Windows 9x compatible PPTP (VPN) server

Maint:

[email protected]

Index:

net

B-deps:

expat-1.95.6_1 gettext-0.12.1 gmake-3.80_1 libiconv-1.9.1_3

R-deps:

I snipped t he result s for brevit y as t his com m and gives t he det ails of each port associat ed wit h VPNs. The form at of t he out put is quit e useful, as it gives t he nam e of t he port it self, it s locat ion in t he port s t ree, a brief descript ion, t he address of t he m aint ainer, as well as t he build and run dependencies. I f you're only int erest ed in seeing how m any port s are available, pipe t he result s t o grep inst ead of more: % make search key=vpn | grep Port

- 436 -

Port:

poptop-1.1.4.b4_2

Port:

pptpclient-1.3.1

Port:

ike-scan-1.2

Port:

openvpn-1.5.0

Port:

tinc-1.0.2

Port:

vpnd-1.1.0

Perhaps you'd prefer t o know t heir locat ions: % make search key=vpn | grep Path Path:

/usr/ports/net/poptop

Path:

/usr/ports/net/pptpclient

Path:

/usr/ports/security/ike-scan

Path:

/usr/ports/security/openvpn

Path:

/usr/ports/security/tinc

Path:

/usr/ports/security/vpnd

What if you already know t he nam e of t he port you want t o inst all but aren't sure what versions are available? Use search name= inst ead. For exam ple, t his com m and will search for all port s wit h netscape in t heir nam es: % make search name=netscape | grep Port Port:

pt_BR-netscape7-7.02

Port:

netscape-remote-1.0_1

Port:

netscape-wrapper-2000.07.07

Port:

netscape-communicator-4.78

Port:

netscape-navigator-4.78

Port:

linux-netscape-communicator-4.8

Port:

linux-netscape-navigator-4.8

Port:

netscape7-7.1

I f you find t he search facilit y useful, it is a good idea t o updat e your port s index periodically. Becom e t he superuser and issue t he following com m and ( it m ay t ake a while, so don't execut e it if you're in a hurry) : # cd /usr/ports # make index

- 437 -

Finally, if you really want t o fine- t une your search result s, spend a few m om ent s reading t he exam ples in / usr/ port s/ Tools/ script s/ README.port search.

8.9.2 Dealing with Installed Ports You've spent a few m ont hs inst alling soft ware and t rying out new applicat ions. How do you keep t rack of all of t hat soft ware and all of t hose dependencies? pkg_info is your friend. My favorit e pkg_info swit ch is definit ely -x. ( There's not really a m nem onic for t his swit ch; I t end t o t hink of it as " give m e version x." ) I f I st ack it wit h any ot her swit ch, I don't need t o know t he full nam e ( including t he com plet e version num ber) of a port . For exam ple: % pkg_info -xc lynx

will show t he one- line com m ent ( -c) of every applicat ion t hat st art s wit h lynx, regardless of t he version num ber. Besides saving m em ory cells for ot her purposes, it 's an excellent way t o find out if you have m ore t han one version of lynx inst alled. Aft er inst alling a port , it 's useful t o see if t here were any m essages, as t hese oft en cont ain configurat ion inst ruct ions: % pkg_info -xD xmms Information for xmms-esound-1.2.8_2:

Install notice: Xmms supports Gzipped and uncompressed skins.

If you would like to use

Zip format skins you will need to ensure archivers/unzip is installed.

How m any t im es have you inst alled a port and had no clue regarding t he nam e of t he execut able, m uch less t he nam es and locat ions of any configurat ion files or docum ent at ion? Thank goodness for -L, t he file- list ing flag: % pkg_info -xL lynx | more Information for lynx-2.8.4.1d:

Files: /usr/local/man/man1/lynx.1.gz /usr/local/bin/lynx /usr/local/etc/lynx.cfg.default /usr/local/share/doc/lynx/CHANGES

- 438 -

Depending upon t he applicat ion, t he list ing m ay be quit e long. A j udicious pipe t o grep bin, grep man, or grep doc m ay bet t er suit your purposes.

8.9.3 Checking Dependencies Before Uninstalling Before uninst alling an applicat ion, it is always a good idea t o see if any ot her packages require t hat applicat ion as a dependency. For exam ple, you've t yped pkg_info | more and see t he applicat ion ORBit-0.5.17. You t hink t o yourself, " I don't rem em ber inst alling, or even ever using, t his applicat ion. Where did it com e from ? Maybe I should j ust get rid of it ." This com m and will clear up your m ini- m yst ery: % pkg_info -xR ORBit Information for ORBit-0.5.17_1:

Required by: bonobo-1.0.22 flashplugin-mozilla-0.4.10_4

Since t he snipped out put t ook up m ost of a page, it looks like t his applicat ion is useful aft er all. Don't worry; if you did t ry t o uninst all t hat applicat ion, pkg_delete would refuse since it is required by t hose ot her applicat ions. However, it is always nice t o be aware of t hese t hings ahead of t im e. I f you really do want t o force t he uninst all of an applicat ion, use -F ( force) wit h pkg_delete.

8.9.4 Checking the Disk Space Your Ports Use What happens if you go a lit t le inst all- crazy and end up wit h m ore applicat ions t han disk space? Use t he -s ( size) swit ch t o det erm ine how m uch space an applicat ion uses. Send t he out put eit her t o a pager: % pkg_info -as | more

or t o a file t hat you can read at your leisure: % pkg_info -as > sizes

You'll t hen have an idea of which applicat ions are using t he m ost space so t hat you can decide which ones are wort h uninst alling. Rem em ber, you also have t he com m ent and dependencies swit ches t o help you decide.

- 439 -

Yet anot her way t o find out what soft ware you have inst alled is t o use pkg_version: % pkg_version | more

This will list each inst alled applicat ion, in alphabet ical order. You'll not e t hat each applicat ion is followed by one of t he t hree sym bols in Table 8- 1.

Ta ble 8 - 1 . pk g_ ve r sion sym bols Sym bol

M e a n in g

=

The applicat ion is up- t o- dat e.




Your index m ay be out - of- dat e.

So, t o det erm ine which applicat ions require upgrading: % pkg_version -l " ls.ps # file ls.ps ls.ps: PostScript document text conforming at level 3.0

Not e t hat t he default invocat ion of groff assum es t hat you wish t o convert a m anpage t o Post Script , so you need no addit ional swit ches.

9.3.5 Hacking the Hack I f you'd like t o publish your m anpages on a local web sit e, groff can also convert t o HTML—see man 1 groff for det ails. I f you prefer t o convert t o PDF, consider inst alling GNU Ghost Script from your port s or packages collect ion. Once inst alled, read man 1 gs for m ore det ails.

9.3.6 See Also man

manpath, man 7 groff ( t he groff form at t ing com m ands—look for t he Request Short Reference sect ion) , man 7 mdoc ( a m ini- t ut orial t hat includes a t em plat e for creat ing m anpages)

- 465 -

Hack 91 Get the Most Out of Manpages

N ow t h a t you k now how t o cr e a t e you r ow n m a npa ge s, you'll w a nt t o k now h ow t o ge t t he m ost ou t of you r m a n pa ge vie w in g. Since m ost docum ent at ion on Unix syst em s lives wit hin m anpages, it pays t o know how t o get t he m ost out of your m anpage- reading experience. How do you m ake sure you're aware of all of t he m anpages inst alled on a syst em ? How do you zero in on t he inform at ion you need, wit hout having t o read an ent ire m anpage? Yes, it 's a great experience t o read all of man tcsh at least once in your life, but you don't want t o do t hat when you're only int erest ed in a cert ain shell variable.

9.4.1 Finding Installed Manpages You m ay have not iced t hat , by default , what is [ H a ck # 1 3 ] doesn't find cust om m anpages or t hose inst alled by t hird- part y applicat ions. Not only is t his inconvenient , but it can also prevent your users from get t ing t he m ost out of t he applicat ions inst alled on a syst em . Rem em ber / et c/ m anpat h.config from [ H a ck # 9 0 ] ? % grep MAP /etc/manpath.config # MANPATH_MAP

path_element

manpath_element

MANPATH_MAP

/bin

/usr/share/man

MANPATH_MAP

/usr/bin

/usr/share/man

MANPATH_MAP

/usr/local/bin

/usr/local/man

MANPATH_MAP

/usr/X11R6/bin

/usr/X11R6/man

The makewhatis com m and act ually creat es t he whatis dat abase and, by default , makewhatis reads only / usr/ share/ m an. I t 'll skip any m anpages in / usr/ local/ m an and / usr/ X11R6/ m an, because it doesn't know t hey exist ! To gat her in t hose m issing m anpages, pass t hese ext ra direct ories t o makewhatis: # makewhatis /usr/local/man /usr/X11R6/man # The superuser can run t his com m and at any t im e, say, aft er inst alling new soft ware. I f you're a forget ful or appropriat ely lazy superuser, consider adding t his as a regular cron j ob.

Now users will be aware of all of t he m anpages on t he syst em .

- 466 -

9.4.2 Navigational Tricks There's not hing worse t han wading t hrough dozens of pages of inform at ion t hat are irrelevant t o your quest ion. Why wade when you can zero in on t he inform at ion you want ? When you read a m anpage, man sends t he t ext t o your default pager—a program designed for speedy navigat ion. FreeBSD 4.1 replaced t he more pager wit h less. less is chock- full of useful and configurable navigat ion t ricks, so t his is a case where less really is m ore. Even t hough your .cshrc file and man man show more as your default pager, rem em ber more is now really less.

less even com es wit h it s own help syst em cont aining an it em ized list of all of it s neat t ricks. Whenever you're in a m anpage—or, for t hat m at t er, in any file you've sent t o a pager—sim ply t ype h t o see t he help screen. I won't repeat t hat help here, but Table 9- 2 shows som e navigat ional keys t o get you m oving around.

Ta ble 9 - 2 . le ss n a viga t ion k e ys Ke y

Be h a vior

Enter

Scrolls down one line

y

Scrolls up one line ( t hink " yikes, I m issed it ! " )

Spacebar

Scrolls down one page

b

Scrolls up ( back) one page

g

Goes t o t he beginning of t he m anpage

q

Quit s t he pager ( so you don't have t o read t he whole m anpage)

9.4.3 Customizing less I t 's well wort h your t im e t o experim ent wit h how less form at s it s out put . For exam ple, when you open a m anpage, t he prom pt at t he bot t om of your screen indicat es how m any byt es of t hat m anpage you've read. I f you t ype -m, you'll change t o t he short prom pt , a single colon ( : ) . -M changes t o t he long prom pt , which displays t he line range you're current ly viewing. I f you really want t o know what line you're on, t ry -N. Read up on -P t o creat e your own cust om prom pt st ring. You can also configure how m any lines you scroll, also known as t he window size. Here I 'll change t he window size t o 10 lines: -z Scroll window size: 10

- 467 -

Scroll window size is 10 lines

(press RETURN)

Now when I press m y spacebar, I 'll scroll down 10 lines inst ead of t he ent ire screen. I f you experim ent wit h t he dozens of opt ions list ed in help, you'll find t hat t hey only last for t he cont ent s of t he current m anpage. I f you find opt ions you like, m ake t hem perm anent by adding t hem t o your ~ / .cshrc file. Here I 'll perm anent ly configure t he -M, or long, prom pt and a window size of 10: setenv LESS Mz10

Not e t hat I 've sim ply creat ed a st ring of desired opt ions, m inus t he swit ch indicat or ( -) . I 'll also have t o change t he line setenv PAGER more t o setenv PAGER less, so t hat applicat ions t hat honor m y pager choice will use less inst ead of more. To t est your changes, force t he shell t o reread it s configurat ion file, t hen open up a m anpage: % source ~/.cshrc % man man

That m anpage should now have a cust om ized prom pt and window.

9.4.4 Searching Text Now t hat you can m ove around, you'll want t o search for t he inform at ion you need. Aft er all, you're usually looking for som et hing specific when you read a m anpage. Fort unat ely, less provides an easy- t o- use search feat ure. Press /, t he forward slash. Your prom pt will change t o / while less wait s for you t o t ype in a search st ring of one or m ore words. Consider adding I t o t he less configurat ion in your .cshrc file t o enable case- insensit ive searching. Wit hout it, searching for /long format in man ls will skip t he desired sect ion, as it is ent it led The Long Format.

Press Ent er once you've t yped in a search st ring, and less will t ake you t o t he first occurrence of t hat st ring. Repeat edly pressing n will scroll you t hrough t he next occurrences. Press N t o scroll back t hrough your search result s. I f you change your m ind and want t o search for som et hing else, press /. Suppose you're reading or searching along and find an int erest ing bit you'll want t o refer t o again. Mark your current posit ion wit h: m mark: a

Here I 've m arked m y posit ion wit h t he let t er a. I 'll t hen carry on wit h reading t he result s of t he rest of m y search. To ret urn t o t hat posit ion, I sim ply t ype a single quot e and t he posit ion m arker ( 'a) . You can m ark as m any as 26 posit ions ( one for each lowercase let t er) .

- 468 -

You can also use t wo single quot es ( '') t o t oggle back and fort h bet ween t wo posit ions. For exam ple, I m ay be in man systat and can't believe t he display includes a pigs opt ion. So I do a search for /pigs and read up on t hat t ype of display. '' will bring m e back t o t he original line t hat piqued m y curiosit y. Anot her '' will put m e back at m y search result .

9.4.5 See Also • • • •

manpath man man man makewhatis man less

- 469 -

Hack 92 Apply, Understand, and Create Patches

Som e t im e s on ly t h e lit t le diffe r e n ce s m a t t e r . Despit e all your best effort s, event ually you'll end up wit h m ult iple versions of a file. Perhaps you forgot t o keep your .vim rc in sync bet ween t wo m achines [ H a ck # 1 0 ] . Alt ernat ively, you m ay want t o see t he changes bet ween an old configurat ion file and t he new version. You m ay even want t o dist ribut e a bugfix t o a m anpage or program . Sending t he ent ire changed file won't always work: it t akes up t oo m uch space and it 's hard t o find exact ly what changed. I t 's oft en easier and usually fast er t o see only t he changes ( see [ H a ck # 8 0 ] for a pract ical exam ple) . That 's where diff com es in: it shows t he differences bet ween t wo files. As you'd expect , applying changes m anually is t edious. Ent er patch, which applies t he changes from a diff file.

9.5.1 Finding Differences Suppose you've shared a useful script wit h a friend and bot h of you have added new feat ures. I nst ead of print ing out bot h copies and m arking differences by hand or, worse, t rying t o reconcile t hings by copying and past ing from one program t o anot her, use diff t o see only t he differences bet ween t he t wo program s. For exam ple, I 've cust om ized an earlier version of t he copydot files.pl script from [ H a ck # 9 ] t o run on Linux inst ead of FreeBSD. When it cam e t im e t o unify t he program s, I want ed t o see t he changes as a whole. diff requires t wo argum ent s, t he source file and t he dest inat ion. Here's t he crypt ic ( at first ) result : $ diff -u copydotfiles.pl copydotfiles_linux.pl --- copydotfiles.pl +++ copydotfiles_linux.pl

2004-02-23 16:09:49.000000000 -0800 2004-02-23 16:09:32.000000000 -0800

@@ -5,8 +5,8 @@ #

- change ownership of those files

# You may wish to change these two constants for your system:

-use constant HOMEDIR => '/usr/home'; -use constant SKELDIR => '/usr/share/skel'; +use constant HOMEDIR => '/home'; +use constant SKELDIR => '/etc/skel';

- 470 -

use strict;

@@ -19,8 +19,8 @@ { for my $dotfile (@ARGV) { -

my $source = catfile( SKELDIR( ),

-

my $dest

+

my $source = catfile( SKELDIR( ),

+

my $dest

= catfile( $user->{homedir},

'dot' . $dotfile ); $dotfile ); $dotfile );

= catfile( $user->{homedir}, $dotfile );

if (-e $dest) {

This out put reveals only t hree changes. Linux and FreeBSD keep user hom e direct ories and skelet on configurat ion files in different direct ories. Fort unat ely, t his only involved changing t wo const ant s at t he t op of t he file. The -u flag produces unified out put , m ingling t he source and dest inat ion lines. I t 's not t he default , but it 's t he easiest t o read and t o explain. Count yourself lucky if you never run across t he alt ernat ives.

As you m ay have guessed from t he nam e, only t he differences appear. Each of t he t wo files has a separat e m arker at t he left m ost colum n. Let 's look at t hat header again: --- copydotfiles.pl

2004-02-23 16:09:49.000000000 -0800

+++ copydotfiles_linux.pl

2004-02-23 16:09:32.000000000 -0800

The first line m arks t he source file, t he FreeBSD version. We're m arking changes against t hat file. diff will m ark lines t hat have changed from t hat file wit h a leading m inus ( -) charact er. The second line m arks t he dest inat ion file, t he Linux version. Lines t hat have changed in t his file appear wit h a leading plus ( +) charact er. diff produces out put t hat you can apply t o t he first file t o produce t he second file. That is, you should rem ove ( or subt ract ) all of t he lines wit h t he leading m inus charact er and add all of t he lines wit h t he leading plus charact er t o produce t he dest inat ion file. The rest of t he out put consist s of hunks. Each hunk also has a header: @@ -5,8 +5,8 @@

- 471 -

This indicat es t hat t he hunk st art s on line 5 of t he source file and affect s eight lines. I t also st art s on t he fift h line of t he dest inat ion file and affect s eight lines—a sim ple subst it ut ion. I n general, you can ignore t his unless you're working on som et hing really det ailed. The act ual lines of t he file are m ore im port ant . Pay close at t ent ion t o t he leading charact ers. #

- change ownership of those files

# You may wish to change these two constants for your system:

-use constant HOMEDIR => '/usr/home'; -use constant SKELDIR => '/usr/share/skel'; +use constant HOMEDIR => '/home'; +use constant SKELDIR => '/etc/skel';

use strict;

Again, t his is a sim ple subst it ut ion. Since diff only works on lines, it has no way of indicat ing t hat only t he value of t he const ant s has changed.

9.5.2 Applying Patches By redirect ing t his out put t o a file, I can produce a pat ch file. Though anyone who can read diff out put could apply t hose changes m anually, it 's m uch easier t o use t he patch program , especially if t he file I 'm pat ching has had ot her changes in t he m eant im e. As long as t hose changes do not overlap, patch will work m agically well. Suppose I 'd writ t en: $ diff -u copydotfiles.pl copydotfiles_linux.pl > dotfiles.patch

Now anyone who want s t o apply t he changes from t he lat t er file t o t he form er file can apply t he pat ch. Copy t he dot files.pat ch file int o t he sam e direct ory as copydot files.pl and use t he com m and: $ patch < dotfiles.patch patching file copydotfiles.pl

I f you're lucky, t he pat ch will apply wit h lit t le fanfare. I f you're unlucky, t hings m ay have m oved around in your file since t he creat ion of t he pat ch. I n t hat case, patch m ay warn about som e fuzz. I f I rearrange a couple of lines in t he first hunk t hat aren't act ually changed in t he pat ch, I m ight see a m essage such as: $ patch < dot.patch patching file copydotfiles.pl

- 472 -

Hunk #1 succeeded at 7 with fuzz 2 (offset 2 lines).

I f I were really unlucky, I 'd have had changes in t he lines t he pat ch also changed. patch t ries as hard as it can t o m assage pat ches, but som et im es it j ust can't resolve t hings. You'll see out put like t his in t hose cases: $ patch < dot.patch patching file copydotfiles.pl Hunk #1 succeeded at 7 with fuzz 2 (offset 2 lines). Hunk #2 FAILED at 21. 1 out of 2 hunks FAILED -- saving rejects to file copydotfiles.pl.rej

I n t his case, it 's up t o you, t he user, t o resolve any changes. patch has act ually creat ed t wo new files, copydot files.pl.orig and copydot files.pl.rej . The first cont ains t he file before any pat ching at t em pt ; t he second cont ains t he hunks patch could not apply. Fort unat ely, t he original file does cont ain t he hunks t hat could apply wit hout conflict s. I n t his case, it 's easier t o open t he copydot files.pl.rej file t o apply t he changes m anually. *************** *** 21,28 **** { for my $dotfile (@ARGV) { -

my $source = catfile( SKELDIR( ),

-

my $dest

= catfile( $user->{homedir},

'dot' . $dotfile ); $dotfile );

if (-e $dest) { --- 21,28 ---{ for my $dotfile (@ARGV) { +

my $source = catfile( SKELDIR( ),

+

my $dest

= catfile( $user->{homedir},

if (-e $dest)

- 473 -

$dotfile ); $dotfile );

{

This form at is a lit t le harder t o read t han t he unified form at , but it 's reasonably st raight forward. The t op half com es from t he source file in t he pat ch and represent s lines 21 t hrough 28 of t he original file. Again, t he leading m inus charact er represent s lines t o rem ove. The bot t om half com es from t he dest inat ion file in t he pat ch, also lines 21 t hrough 28. This cont ains t wo lines t o add. Looking in copydot files.pl around t hose lines, it t urns out t hat t he first line cont aining SKELDIR( ) has changed subt ly, t hus causing t he conflict : { for my $dotfile (@ARGV) { my $source = catfile( SKELDIR( ), my $dest

"dot$dotfile" );

= catfile( $user->{homedir},

$dotfile );

if (-e $dest) {

I have t wo opt ions: I could edit t he file direct ly, m aking t he m odificat ions as seen in eit her t he source file or t he dest inat ion file of t he pat ch, or I could ignore t his hunk if t he local m odificat ions are bet t er t han t hose of t he pat ch. I n t his case, t he pat ch is clearly an im provem ent . Since it 's only t wo lines, I 'll j ust m ake t he changes direct ly. Ot herwise, I could revert t he changes in m y local file and t ry t o reapply t he rej ect ed hunks.

9.5.3 Creating Patches I t 's oft en handy t o creat e pat ches from norm al files, as in t he previous exam ple, when sharing code or t ext wit h anot her user. I t 's also useful t o see t he differences bet ween configurat ion files when upgrading an applicat ion. Knowing how t o read a diff bet ween your version of ht t pd.conf and ht t pd.conf.default can save you hours of debugging t im e. What if you want t o find differences bet ween ent ire direct ories, t hough? Suppose you want t o see t he changes bet ween t wo versions of a program . I f you can't upgrade t o t he new version right away but want t o see if t here's a pat ch available t hat you can apply, use diff on t he direct ories t hem selves. Be sure t o pass t he recursive flag ( -r) if you want t o com pare files in subdirect ories: $ diff -ur sdl/trunk SDL_Perl-2.1.0 > sdl_trunk.patch

I f t hat 's not appropriat e and you want t o pat ch only a couple of files at a t im e, run diff m ult iple t im es. Append t he out put t o a com bined pat ch. patch is sm art enough t o recognize t he st art of file m arkers: $ diff -u sdl/trunk/CHANGELOG SDL_Perl-2.1.0/CHANGELOG >> \

- 474 -

sdl_textfiles.patch $ diff -u sdl/trunk/README SDL_Perl-2.1.0/README >> \ sdl_textfiles.patch $ diff -u sdl/trunk/INSTALL SDL_Perl-2.1.0/INSTALL >> \ sdl_textfiles.patch

Finally, if you need t o creat e a pat ch for a file t hat doesn't exist , use t he null file flag (-n) wit h / dev/ null as t he source: $ diff -un /dev/null SDL_Perl-2.1.0/LICENSE >> \ sdl_textfiles.patch

This will creat e t he file when som eone applies t he pat ch. You could also touch t he file in t he source direct ory.

9.5.4 Revision Control Life's m uch easier when you're working wit h revision cont rol. Som eday, you m ay find yourself pat ching source code or t ext files in core BSD. Modify t he code in your t ree, m ake sure it works, and t hen use cvs diff -u t o generat e pat ches t o m ail t o t he appropriat e developm ent list . Subversion, t he likely successor t o CVS, generat es t he right kind of pat ches wit hout t he -u flag—sim ply use svn diff. There is a FreeBSD port and a Net BSD package for Subversion. You can also download binary packages and source for m ost operat ing syst em s from ht t p: / / subversion.t igris.org/ . Once you're used t o using pat ches t o keep t rack of file differences, you m ay find yourself t em pt ed t o keep all im port ant files under version cont rol. Hey, why not ?

9.5.5 See Also • • •

man diff man patch " CVS hom edir," Joey Hess's Linux Journal art icle on keeping his hom e direct ory in CVS ( ht t p: / / www.linuxj ournal.com / art icle.php?sid= 5976)

- 475 -

Hack 93 Display Hardware Information

I f you 'r e ne w t o Fr e e BSD , you m a y be w on de r in g w h e r e t o find in for m a t ion a bou t your syst e m 's h a r dw a r e a nd t he r e sou r ce s it u se s. You've probably not iced t hat your FreeBSD syst em didn't ship wit h a Microsoft - st yle Device Manager. However, it does have plent y of useful ut ilit ies for gat hering hardware inform at ion.

9.6.1 Viewing Boot Messages When you boot your syst em , t he kernel probes your hardware devices and displays t he result s t o your screen. You can view t hese m essages, even before you log in, by pressing t he scroll lock key and using your up arrow t o scroll back t hrough t he m essage buffer. When you're finished, press scroll lock again t o ret urn t o t he login or com m and prom pt . You can t ype dmesg any t im e you need t o read t he system m essage buffer. However, if it 's been a while since boot up, it 's quit e possible t hat syst em m essages have overwrit t en t he boot m essages. I f so, look in t he file / var/ run/ dm esg.boot , which cont ains t he m essages from t he lat est boot . This is an ASCI I t ext file, so you can send it t o a pager such as more or less. You m ay find it m ore convenient t o search for som et hing part icular. For exam ple, suppose you've added sound support t o your kernel by adding device pcm t o your kernel configurat ion file. This com m and will show if t he PCM device was successfully loaded by t he new kernel: % grep pcm /var/run/dmesg.boot pcm0: port 0xa800-0xa83f irq 10 at device 7.0 on pci0 pcm0:

I n t his exam ple, t he kernel did indeed probe m y Creat ive sound card at boot up.

9.6.2 Viewing Resource Information Som et im es you j ust want t o know which devices are using which syst em resources. This com m and will display t he I RQs, DMAs, I / O port s, and I / O m em ory addresses in use: % devinfo -u Interrupt request lines: 0 (root0) 1 (atkbd0) 2 (root0) 3 (sio1) 4 (sio0)

- 476 -

5 (rl0) 6 (fdc0) 7 (ppc0) 8 (root0) 9 (acpi0) 10 (pcm0) 11 (rl1) 12 (psm0) 13 (root0) 14 (ata0) 15 (ata1) DMA request lines: 0-1 (root0) 2 (fdc0) 3 (ppc0) 4-7 (root0) I/O ports: 0x0-0xf (root0) 0x10-0x1f (acpi_sysresource0) 0x20-0x21 (root0)

I/O memory addresses: 0x0-0x9ffff (root0) 0xa0000-0xbffff (vga0) 0xc0000-0xcbfff (orm0) 0xcc000-0xfbffffff (root0) 0xfc000000-0xfdffffff (agp0) 0xfe000000-0xffffffff (root0)

Alt ernat ely, use devinfo -r if you prefer t o see your list ing by device. I f you're unsure what a device is, use t he whatis com m and. For exam ple, in m y list ing, ppc0 uses I RQ 7 and DMA 3. To find out what ppc0 is:

- 477 -

% whatis ppc ppc(4)

Parallel Port Chipset driver

Don't include t he t railing num ber when using t he whatis com m and.

9.6.3 Gathering Interface Statistics There are several ways t o gat her net work int erface inform at ion. One of t he handiest is t he i swit ch t o netstat: % netstat -i Name

Mtu Network

Address

rl0*

1500

00:05:5d:d2:19:b7

0

0

0

0

0

rl1*

1500

00:05:5d:d1:ff:9d

0

0

0

0

0

ed0

1500

00:50:ba:de:36:33

15247

0

11301

0

78

ed0

1500 192.168.2

genisis.

15091

-

11222

-

-

lp0*

1500

0

0

0

0

0

lo0

16384

179

0

179

0

0

lo0

16384 your-net

179

-

179

-

-

localhost

Ipkts Ierrs

Opkts Oerrs

Coll

This com m and shows all int erfaces, bot h physical and virt ual. This part icular syst em has t hree net work int erface cards: rl0, rl1, and ed0. The first t wo int erfaces are shut down, as indicat ed by t he * aft er t he device nam e. These t hree are Et hernet cards, as indicat ed by t heir MAC addresses. ( This is also an excellent way t o find all of t he MAC addresses on your syst em ) . The ed0 int erface and loopback int erface ( lo0) have each been configured wit h a host nam e and an I P address, as indicat ed by t he Network colum n. I f you're only int erest ed in seeing int erfaces configured wit h an I Pv4 address, add t he -f ( fam ily) swit ch: % netstat -i -f inet ed0 lo0

1500 192.168.2 16384 your-net

genisis.

15091

-

11222

-

-

localhost

179

-

179

-

-

9.6.4 Viewing Kernel Environment You can also find hardware inform at ion by using kenv t o view your kernel environm ent . kenv will dum p several screens wort h of inform at ion, so use grep when possible t o zero in on t he inform at ion you want . For exam ple, t o view I RQ inform at ion: % kenv | grep irq hint.ata.0.irq="14"

- 478 -

hint.ata.1.irq="15" hint.atkbd.0.irq="1" hint.ed.0.irq="10" hint.fdc.0.irq="6" hint.ie.0.irq="10" hint.le.0.irq="5" hint.lnc.0.irq="10" hint.pcic.1.irq="11" hint.ppc.0.irq="7" hint.psm.0.irq="12" hint.sio.0.irq="4" hint.sio.1.irq="3" hint.sio.2.irq="5" hint.sio.3.irq="9" hint.sn.0.irq="10"

I f you're unsure what is using a list ed I RQ, use whatis t o look up t he second word ( t he one aft er hint) . For exam ple, t his will show what is using m y I RQ 12: % whatis psm psm(4)

- PS/2 mouse style pointing device driver

I act ually prefer t he out put of kenv t o t hat of devinfo. Here, I 'll search for t he I / O addresses used by m y COM port s: % kenv | grep port | grep sio hint.sio.0.port="0x3F8" hint.sio.1.port="0x2F8" hint.sio.2.port="0x3E8" hint.sio.3.port="0x2E8"

To see which devices are disabled: % kenv | grep disabled hint.sio.2.disabled="1" hint.sio.3.disabled="1"

- 479 -

BSD gives t he first com port t he num ber zero, so it looks like I have COM3 and COM4 disabled on t his syst em .

9.6.5 See Also • • • •

man man man man

dmesg devinfo netstat kenv

- 480 -

Hack 94 Determine Who Is on the System

As a syst e m a dm in ist r a t or , it pa ys t o k n ow w h a t 's ha ppe n in g on you r syst e m s. Sure, you spend t im e reading your logs, but do you t ake advant age of t he ot her inform at ion- gat hering ut ilit ies available t o you? Silent ly, in t he background, your syst em t racks all kinds of neat inform at ion. I f you know enough t o peek under t he syst em hood, you can get a very good view of what is occurring on t he syst em at any given point in t im e. For t he experienced hacker, t he out put from t hese com m ands m ay suggest int erest ing script ing possibilit ies.

9.7.1 Who's on First? Have you ever needed t o know who logged int o a syst em and for how long? Use t he users com m and t o see who's logged in now: % users dru biko

Perhaps you prefer t o know who is on which t erm inal. Try who. Here, t he H includes colum n headers and t he u shows each user's idle t im e: % who -Hu NAME

LINE

TIME

IDLE

FROM

dru

ttyv1

Jan 25 08:59 01:00

biko

ttyv5

Jan 25 09:57

dru

ttyp0

Jan 25 09:58 00:02 (hostname)

.

Feel free t o experim ent wit h who's swit ches t o find an out put t hat suit s your needs. Here, dru and biko have logged in physically at t his syst em 's keyboard using virt ual t erm inals 1 and 5. dru has also logged in over t he first psuedot erm inal ( over t he net work) from t he specified host nam e. To find out what everyone is doing, use w: % w 10:07AM USER

up

1:20, 9 users, load averages: 0.02, 0.02, 0.09 TTY

FROM

LOGIN@

- 481 -

IDLE WHAT

dru

v1

-

8:59AM

1:08 pine

biko

v5

-

9:57AM

- w

dru

p0

hostname

9:58AM

4 -csh (csh)

I f you're j ust int erest ed in t hat first line of out put , use uptime.

Not ice t hat as a regular user, I was easily able t o find out who is logged in, where t hey are, and what t hey're current ly doing. I f you don't want regular users knowing what com m ands ot her users are current ly running, see [ H a ck # 5 7 ] .

9.7.2 When Did That Happen? You're not lim it ed t o finding out what 's happening at t his part icular m om ent . Use lastlogin t o see t he m ost recent t im e at which each of your users logged in: % lastlogin dru

ttyv1

Sun Jan 25 08:59:36 2004

biko

ttyv5

Sun Jan 25 09:57:18 2004

dlavigne

ttyv6

Sat Jan 24 09:48:32 2004

dru

ttyp0

hostname

Sun Jan 25 09:58:50 2004

rembackup

ttyp0

hostname

Fri Jan 23 01:00:00 2004

For a slight ly different out put , last can show who is st ill logged in: % last | grep still dru

ttyp0

dru biko

hostname

Sun Jan 25 09:58

still logged in

ttyv1

Sun Jan 25 08:59

still logged in

ttyv5

Sun Jan 25 09:57

still logged in

Do you need a record of syst em shut downs or reboot s? The / var/ log/ wt m p dat abase holds t his inform at ion. Use last t o view t he desired st at ist ics: % last reboot reboot

~

Tue Jan 20 15:37

reboot

~

Tue Nov 25 07:24

reboot

~

Sun Aug

wtmp begins Tue Jul

1 15:27:26 EDT 2003

- 482 -

3 09:05

% last shutdown shutdown

~

wtmp begins Tue Jul

Wed Dec 24 22:14 1 15:27:26 EDT 2003

9.7.3 Details, Details Anot her opt ion t o consider is enabling syst em account ing, which m aint ains a dat abase of ext rem ely det ailed st at ist ics of every process and subprocess t hat has been execut ed on a syst em . # touch /var/account/acct # accton /var/account/acct

Not e t hat t he accton com m and will fail if you don't specify t he nam e of t he account ing log or if t hat log doesn't already exist . Also, in a queer case of logic, t yping accton wit h no argum ent s really t urns account ing off. Once account ing is enabled, use lastcomm t o view t he cont ent s of / var/ account / acct : % lastcomm lastcomm

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

man

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

sh

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

sh

-F

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

less

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

col

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

groff

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

grotty

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

troff

-

dlavigne

ttyv6

0.08 secs Sun Jan 25 11:33

tbl

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

zcat

-

dlavigne

ttyv6

0.00 secs Sun Jan 25 11:33

cron

-F

root

__

0.00 secs Sun Jan 25 11:33

sh

-

operator

__

0.00 secs Sun Jan 25 11:33

sh

-

operator

__

0.00 secs Sun Jan 25 11:33

dd

-

operator

__

0.00 secs Sun Jan 25 11:33

mv

-

operator

__

0.00 secs Sun Jan 25 11:33

- 483 -

mv

-

operator

__

0.00 secs Sun Jan 25 11:33

mv

-

operator

__

0.00 secs Sun Jan 25 11:33

rm

-

operator

__

0.00 secs Sun Jan 25 11:33

jot

-

operator

__

0.00 secs Sun Jan 25 11:33

accton

-

root

ttyv0

0.00 secs Sun Jan 25 11:32

This com es from a quiet syst em one m inut e aft er enabling account ing. A cron j ob happened t o be running at t he t im e, hence t he operator lines. The user dlavigne6 also opened up a m anpage during t hat period. Not e all of t he processes involved before man act ually st art ed. This com m and can also show you which processes ended abnorm ally. Search for t he D flag, which indicat es t hat t he process dum ped core: % lastcomm | grep -w "D"

Depending upon your securit y requirem ent s, you m ay not want users t o have access t o such det ailed inform at ion. Aft er all, lastcomm will show every process run by every user. Tight ening perm issions will fix t hat : # chmod 600 /var/account/acct # su dlavigne % lastcomm lastcomm: /var/account/acct: Permission denied

Also, if you're planning on using lastcomm as an ext ra audit t rail, consider changing t his file's flags [ H a ck # 5 6 ] . You'll also want t o have plent y of disk space on t he filesyst em holding t he dat abase. Finally, t o enable syst em account ing when t he syst em boot s, add t his line t o / et c/ rc.conf: accounting_enable="YES"

9.7.4 See Also • • • • • •

man man man man man man

users who w lastlogin last lastcomm

- 484 -

Hack 95 Spelling Bee

For t h ose w ho e dit t h e ir t e x t a t t h e com m a n d line. Like m ost com put er users, you probably find yourself spending a fair bit of t im e t yping, whet her responding t o em ail, navigat ing t he web, or working on t hat résum é or t hesis. How oft en do you find yourself looking at a word, wondering if you've spelled it correct ly? How oft en do you rack your brain t rying t o find a m ore int erest ing or descript ive word? You've probably discovered t hat Unix doesn't com e wit h a built - in dict ionary or t hesaurus. Sure, you can inst all a feat ure- rich GUI office suit e, but what alt ernat ives are t here for users who prefer less bloat on t heir syst em s or are accessing syst em s from t he com m and line?

9.8.1 Quick Spellcheck I f you're in doubt about t he spelling of a word, t ry using look. Sim ply include as m uch of t he word as you're sure about . For exam ple, if you can't rem em ber how t o spell " bodacious" but you're pret t y sure it st art s wit h " boda" : % look boda bodach bodacious bodaciously I f you don't have access t o a GUI , see [ H a ck # 1 2 ] .

I find look especially helpful wit h suffixes. I t 's very handy if you can't rem em ber when t o use " ly" , " ally" , or " ily" . For exam ple: % look mandator mandator mandatorily mandatory

9.8.2 Creating a Dictionary or Thesaurus look is a useful spellchecker, but it won't show you t he m eanings or synonym s of a word. Accordingly, I found m yself spending a fair bit of t im e at ht t p: / / dict ionary.reference.com / . While t here, I not iced a pat t ern. What ever word I searched for was appended t o t he URL as

- 485 -

search?q= < m yword> . Whenever I used t he dict ionary, t he URL st art ed wit h dict ionary, which changed t o t he word t hesaurus whenever I did a t hesaurus lookup. That suggest ed t o m e t hat it would be very easy t o generat e m y own cust om lookup ut ilit y, so I st art ed out wit h t hese t wo script s: % more ~/bin/dict #!/bin/sh # script to look up the definition of word from dictionary.reference.com # replaces $1 with user's search string # or gives error message if user forgets to include search string if test $1 then w3m "http://dictionary.reference.com/search?q="$1"" else echo "Don't forget to include the word you would like to search for" exit 1 fi

% more ~/bin/thes

#!/bin/sh # script to find the synonym of word from thesaurus.reference.com # replaces $1 with user's search string # or gives error message if user forgets to include search string if test $1 then w3m "http://thesaurus.reference.com/search?q="$1"" else echo "Don't forget to include the word you would like to search for" exit 1 fi

Recognize t hose posit ional param et ers we saw before in [ H a ck # 1 3 ] ? When I use eit her script , I include t he word t hat I would like t o look up.

- 486 -

The ut ilit y I chose t o grab t he result s is t he com m and- line browser w3m, which can be built from / usr/ port s/ www/ w3m . I f you have already inst alled anot her com m and- line browser, such as lynx or links, specify your browser in your own script . Don't forget t o m ake your script execut able wit h chmod +x. Then, t o look up t he m eaning of a word: % dict palladium

Or, t o find it s synonym s and ant onym s: % thes brusque I f you're not st uck at t he com m and line, Mozilla- based browsers allow you t o creat e sim ilar short cut s. See Asa Dot zler's art icle on cust om keywords at ht t p: / / www.m ozilla.org/ docs/ end- user/ keywords.ht m l.

9.8.3 Improved Dictionary Well, t hat 's a fair st art —m y browser now aut om agically t akes m e t o t he correct sect ion of an online dict ionary or t hesaurus whenever I 'm curious about a part icular word. However, what if I want t o forgo using a browser alt oget her? FreeBSD com es wit h t he fetch ut ilit y specifically t o ret rieve web inform at ion. Why not use it t o ret rieve t he result s? Before edit ing m y script s, I t ried various invocat ions of fetch at t he com m and line unt il I had achieved m y desired result s. I st art ed out by replacing w3m wit h fetch ( not e t hat I had t o supply a word, in t his case test, as I was at t he com m and line, not wit hin a script ) : % fetch "http://dictionary.reference.com/search?q=test"

This worked, but it result ed in a file called search?q= < m yword> , where < m yword> was t he word I had supplied as t he param et er. Aft er a while, m y hom e direct ory would be full of hundreds of files st art ing wit h search?q. So, I specified t he nam e of a file t o which t o writ e t he result s: % fetch -o results "http://dictionary.reference.com/search?q=test"

Now, regardless of t he num ber of t im es I use m y script , I 'll only have one file called result s. There's a problem wit h t hat file, t hough. I t 's an HTML file, so unless I enj oy wading t hrough HTML t ags in order t o read m y result s, I have t o open up t hat file in a browser. That sort a defeat s m y goal of not using a browser. So, I went out on t he Web looking for an HTML- t o- ASCI I convert er. I t ried out several before set t ling on a Perl script called html2txt . I t hen t ried piping t he result s file t o t he convert er: % fetch -o results "http://dictionary.reference.com/search?q=test" \ | html2txt results Cannot open HTML source file : results, Error No such file or directory

- 487 -

Receiving results: 21791 bytes

That 's when I hit a t im ing issue. I t t akes a few seconds for fetch t o ret rieve t he file, so html2txt com plains when t he shell asks for it t o work on t hat ( as of yet ) nonexist ent file. To solve t hat , I asked t he shell t o wait unt il aft er fetch was finished by using && inst ead of |: % fetch -o results "http://dictionary.reference.com/search?q=test" \ && html2txt results

To finish off m y com m and, I ask for t he ASCI I - fied file t o be opened up in a pager so I can view t he result s: % fetch -o results "http://dictionary.reference.com/search?q=test" \ && html2txt results && more results.txt

Not e t hat t his part icular convert er creat es an ASCI I file wit h t he sam e nam e, but wit h a .t xt ext ension.

9.8.4 Become a Crossword Champion Did you know t hat your syst em has a built - in crossword- puzzle solver? You m ay never have t o leave a square em pt y again if you rem em ber t his lit t le t rick. Consider a word t hat resem bles: t _ _ _ k _ _ _r

This one- liner will show your possibilit ies, allowing you t o choose t he word t hat m at ches t he clue definit ion: % grep -wi 't...k...r' /usr/share/dict/words/ thickener trickster trinketer truckster

Here, grep searched t hrough t he dict ionary words inst alled on your syst em . ( This is t he sam e file t hat look searches.) Use single quot es for your search phrase, and replace each blank square in your crossword wit h a ..

9.8.5 See Also man

fetch, The Perl HTML- t o- t ext convert er at ht t p: / / www.ft ls.org/ en/ exam ples/ perl- t ools/ ht m l2t xt .sht m l, " Wanna Cheat at Crosswords?" ( ht t p: / / www.osxfaq.com / t ips/ unix- t ricks/ week23/ friday.ws)

- 488 -

Hack 96 Leave on Time

Use you r t e r m ina l's bu ilt - in t im e r s a nd sch e du le r s. You know how it is. You sit down in front of a keyboard and quickly becom e absorbed in your work. At som e point you rem em ber t o look up, only t o not ice t hat everyone else is gone for t he day. I f t hat doesn't describe you, I bet you can t hink of at least one person it does describe.

9.9.1 Don't Forget to Leave Fort unat ely t he leave com m and can save you from t he em barrassm ent of forget t ing im port ant appoint m ent s. Use it at any t im e by t yping: % leave When do you have to leave?

There are t hree ways t o respond t o t hat quest ion: • • •

Press Ent er t o abort . Type hhmm, where hh represent s t he hour and mm represent s t he m inut e. Type +number, where number represent s how m any hours or m inut es from now you'd like t o leave.

For exam ple, t o leave at 5 PM: % leave 500 Alarm set for Tue Dec 30 17:00:00 EST 2003. (pid 50097)

leave 1700 will achieve t he sam e result s. Or, t o leave in 45 m inut es: % leave +45 Alarm set for Tue Dec 30 9:52:00 EST 2003. (pid 50108)

Be sure t o include t he + if you're not specifying an act ual t im e. You can t hen carry on wit h your day. Five m inut es before it 's t im e t o leave, your t erm inal will beep and display t his m essage: You have to leave in 5 minutes.

You'll receive anot her warning one m inut e before t he set t im e, t hen every m inut e t hereaft er. leave definit ely works for t he procrast inat or and t hose who always need t o do

- 489 -

j ust one m ore t hing before leaving. The only way t o end t he incessant nagging is t o log out or killall leave ( but please don't t ake t hat last com m and lit erally! ) . Consider placing / usr/ bin/ leave in / usr/ share/ skel/ dot .cshrc [ H a ck # 9 ] .

9.9.2 Creating Terminal Sticky Notes leave is nice for scheduling your own depart ure, but what if you want t o schedule t he execut ion of com m ands? I bet you're t hinking " use at or cron." Have you ever t ried t he scheduler built int o tcsh? While sched can execut e any com m and at a given t im e, you can also use it as a rem inder syst em . I use it as a t erm inal st icky- not e syst em t hat won't clut t er up m y m onit or. For exam ple, it 's 9: 00, I 've j ust logged in, and I 'm m ulling over m y t o- do list for t he day. As I m ent ally review m y list , I t ype t he following: % sched 11:55 echo Lunch with Robyn today. % sched 2:30 echo Reminder: project due by 4:30. % sched 5:00 echo Go home!!!

Now at any point in t he day I can review m y t o- do list : % sched

1

11:55

echo Lunch with Robyn today.

2

2:30

echo Reminder: project due by 4:30.

3

5:00

echo Go home!!!

As each appoint ed t im e arrives, t he desired rem inder will appear on m y t erm inal. To rem ove an it em from your t o- do list , sim ply t ype sched -#, where # represent s t he num ber of t hat it em in t he schedule. Logging out of your shell will also rem ove all it em s from your list since sched is a shell com m and.

9.9.3 Saving Your Schedule What if you plan on logging out during t he day? You cert ainly don't want t o recreat e your schedule every t im e you log in. I t 's a sim ple m at t er t o save t he schedule. Place t his line in your ~ / .logout file: sched > schedule

This will send t he out put of sched t o a file in your hom e direct ory called schedule, saving any it em s in your t o- do list t o t he specified file when you log out . Unfort unat ely, t here's no sim ple way t o pipe t hat list back int o sched when you log back in. This has t o do wit h how t he C shell handles it s built - in com m ands. You would t hink t hat :

- 490 -

% sched < schedule

would reverse t he process, but it doesn't . I f you really m iss your shell sending you rem inders at t heir appoint ed t im es, consider locking your t erm inal [ H a ck # 7 ] inst ead of logging out during t he day.

9.9.4 See Also • •

man leave man tcsh

- 491 -

Hack 97 Run Native Java Applications

Un t il r e ce nt ly, r u n n ing Ja va a pplica t ion s on Fr e e BSD m e a n t u sin g t h e Linu x com pa t ibilit y m ode . Linux program s can som et im es be problem at ic on FreeBSD. Java© uses t hreading very heavily, and t hat 's probably t he poorest - em ulat ed part of Linux binary com pat ibilit y. Som e Java applicat ions or class libraries j ust don't work correct ly under Linux em ulat ion. Nat ive versions of t he Java dist ribut ion had rest rict ive licenses, and it required a great deal of work t o download and com pile t hem . Fort unat ely, t he FreeBSD Foundat ion has negot iat ed a FreeBSD Java license wit h Sun Microsyst em s. This hack dem onst rat es how t o configure t he FreeBSD version of Java. What about nat ive Java on Net BSD or OpenBSD? At t he t im e of writ ing, neit her syst em had a nat ive Java port . You can run Java on a Linux em ulat or or via Tom cat .

9.10.1 Choosing Which Java Port to Install The first requirem ent for running Java applicat ions is a Java Virt ual Machine ( JVM) and t he associat ed runt im e support libraries. There are several Java Runt im e Environm ent s ( JREs) or Java Developm ent Kit s ( JDKs) available in port s. A JRE cont ains everyt hing necessary for an end user t o run Java applicat ions. A JDK cont ains all t hat , plus various ext ra bit s required for developing, com piling, and debugging Java code.

The m ain crit eria for choosing a port are: • • •

Which version of Java do you need? Do you want t o run FreeBSD nat ive code or Linux code run under em ulat ion? Do you prefer t o run a precom piled binary or com pile it yourself from source code?

Unless you have a specific requirem ent for an earlier version, choose t he lat est st able release, which, as of t his writ ing, is Java 1.4.2. The nat ive version, found in / usr/ port s/ j ava/ j dk14, will give you t he best perform ance, but you will have t o com pile it yourself. That is m ore easily said t han done: com piling t he JDK requires a great deal of disk space and CPU power, as well as a working copy of t he 1.4.2 JDK. The first t im e you com pile, you will have t o inst all one of t he Linux JDKs, such as t he recom m ended / usr/ port s/ j ava/ linux- sun- j dk14, but once you have a working nat ive JDK, you can use it t o com pile any updat es and uninst all t he Linux version. You can inst all several Java versions sim ult aneously wit hout t hem int erfering wit h each ot her. Each will inst all int o it s own subdirect ory of / usr/ local.

- 492 -

I f you need a precom piled nat ive version, choose one of t he Diablo Java 1.3.1 port s. These use t he sam e code base as t he / usr/ port s/ j ava/ j dk13 port , and t hey're cert ified, licensed, and released t hrough t he sponsorship provided by t he FreeBSD Foundat ion ( ht t p: / / www.freebsdfoundat ion.org/ downloads/ j ava.sht m l) . Diablo JDK 1.4 and JRE 1.4 versions are under developm ent , but not yet available. The Diablo Java packages are st andard FreeBSD packages, so you can inst all t hem via pkd_add. However, you're bet t er off inst alling from t he Diablo port s, as t hat will provide you wit h t he correct dependencies. For exam ple, t o inst all t he Lat t e Diablo JRE 1.3.1 port , visit ht t p: / / www.freebsdfoundat ion.org/ cgi- bin/ download.cgi?package= diablo- j re- 1.3.10.t ar.bz2. Read and accept t he license t erm s, and save t he downloaded file as / usr/ port s/ dist files/ diablo- j re- 1.3.1- 0.t ar.bz2. Then: # cd /usr/ports/java/diablo-jre13 # make install

9.10.2 Running Java Applications St art ing up any Java applicat ion m eans running a Java Virt ual Machine, which in t urn loads a nam ed Java class. That class is t he ent ry point for t he program . The JVM always requires t he CLASSPATH environm ent variable t o cont ain a list of .j ar archives t hat st ore all of t he Java classes required by t he applicat ion. You can provide ext ra argum ent s t o t he JVM—t o lim it it s use of m em ory or ot her syst em resources, for exam ple—and t he applicat ion it self m ay t ake furt her com m and- line argum ent s.

9.10.3 Standalone Java Applications Many Java applicat ions provide a shell script t o set up t he environm ent and t o execut e t he JVM wit h t he appropriat e argum ent s. A t ypical exam ple is ant ( see / usr/ port s/ devel/ apache- ant ) , t he Java equivalent t o make. The inst allat ion process edit s t he script t hat will becom e / usr/ local/ bin/ ant t o use t he Java version used when building t he port . However, you can override t he default Java version wit hin t he script by set t ing t he JAVA_HOME environm ent variable: % setenv JAVA_HOME=/usr/local/jdk14

9.10.4 Javavmwrapper Given t he wide variet y of JVMs available under FreeBSD, adding code t o all Java applicat ion wrapper script s or ot herwise configuring st andalone Java applicat ions t o use t he correct JVM could becom e a m aint enance night m are. Fort unat ely, t he / usr/ port s/ j ava/ j avawm wrapper port provides t he / usr/ local/ bin/ j avavm script , which all applicat ions can run t o discover t he sit e's default JVM. javavm's configurat ion file, / usr/ local/ et c/ j avavm s, cont ains a list of inst alled JVMs in t he order of t heir preference. I nst alling or rem oving a JVM t hrough port s will m odify t his file. You can also edit it by hand.

- 493 -

9.10.5 Applets I n t he case of a Java applet , t he web browser st art s t he JVM and downloads and runs t he applet from t he Web. Applet s run in a special sandbox t hat denies t hem access t o m ost of t he local syst em , except for t he browser window. Java support in web browsers derived from Net scape ( including Mozilla, Firebird, and Galeon) uses a plug- in t hat com es st andard wit h t he JDK. For t he nat ive JDK 1.4.2, t he plug- in is / usr/ local/ j dk1.4.2/ j re/ plugin/ i386/ ns610/ libj avaplugin_oj i.so. To m ake t his plugin available t o web browsers, creat e a sym link t o t his file from / usr/ X11R6/ lib/ browser_plugins: # cd /usr/X11R6/lib/browser_plugins # ln -s /usr/local/jdk1.4.2/jre/plugin/i386/ns610/libjavaplugin_oji.so .

Launch a web browser and t ype about:plugins int o t he locat ion bar. You should see an ent ry for t he " Java( TM) Plug- in," which claim s t o handle about 30 MI ME t ypes, all variant s on application/x-java-som et hing. I f you're using a Linux web browser under em ulat ion, inst all t he plug- in from one of t he Linux Java versions.

9.10.6 Servlets A servlet is all or part of a web applicat ion writ t en in Java. I t runs t hrough a servlet cont ainer applicat ion, which abst ract s out all of t he com m on server- side funct ionalit y. Tom cat ( / usr/ port s/ www/ j akart a- t om cat 41) and Jet t y ( / usr/ port s/ www/ j et t y) are t wo exam ples of t hese applicat ions. The servlet cont ainer applicat ion runs in m uch t he sam e way as st andalone Java applicat ions.

9.10.7 Java WebStart WebSt art is a web- based m echanism for downloading and updat ing Java applicat ions. Use t he Preferences m enu it em in javaws t o cont rol t he JVM t hat will run t he WebSt art - ed applicat ions. Unlike applet s, t he downloaded applicat ions run independent ly of t he web browser. You don't need t o download t hem again each t im e t hey run. They also have full access t o t he underlying syst em . The javaws applicat ion is a st andard part of Java 1.4 or above. I t lives in ${ JAVA_HOME} / j re/ j avaws/ j avaws.

9.10.8 See Also •

FreeBSD Foundat ion's Java downloads ( ht t p: / / www.freebsdfoundat ion.org/ downloads/ j ava.sht m l)

- 494 -

Hack 98 Rotate Your Signature

En d you r e m a il com m u n ica t ion s w it h a sh or t w it t icism . We all seem t o know at least one geek friend or m ailing- list post er whose em ails always end wit h a different and hum ourous bit of random nonsense. You m ay be aware t hat t his is t he work of her ~ / .signat ure file, but have you ever wondered how she m anages t o rot at e t hose signat ures? While t here are several ut ilit ies in t he port s collect ion t hat will random ize your signat ure, it is easy enough t o roll your own signat ure rot at or using t he fortune program and a few lines of shell script ing.

9.11.1 If Your Mail Program Supports a Pipe Your approach will vary slight ly, depending on whet her your part icular m ail user agent ( MUA) support s pipes. I f it does, it 's capable of int erpret ing t he cont ent s of a file as com m and out put , j ust like when you use a pipe ( |) on t he com m and line. I use pine, which support s bot h st at ic signat ure files and signat ures t hat com e from t he piped out put of a signat ure rot at ion program . When configuring pine, choose Setup from t he m ain m enu, t hen C for t he configurat ion edit or. Find t he signature-file opt ion and give it t his value: .signature |

The pipe charact er t ells pine t o process t hat filenam e as a program inst ead of insert ing it s cont ent s lit erally. Also enable t he signature-at-bottom opt ion found in t he Reply Preferences t o ensure your signat ure is placed at t he bot t om of your em ails, even when replying t o an em ail. Next , creat e a file called ~ / .signat ure cont aining t hese lines: echo "Your random fortune:" /usr/games/fortune -s

This isn't quit e a shell script : I don't have t o include t he #!/bin/sh line or use chmod +x t o set t he file as execut able. However, pine will execut e t hose t wo lines whenever I com pose or reply t o an em ail, adding som et hing like t his t o t he bot t om of t he em ail: Your random fortune: "Right now I'm having amnesia and deja vu at the same time." -- Steven Wright

- 495 -

I also included t he short swit ch ( -s) t o fortune, as it 's bad Net iquet t e t o end an em ail wit h a long signat ure. I f you t ry a few t est m essages, you'll see t hat every em ail receives a different , random signat ure. Depending upon your audience, you m ay wish t o filt er furt her t he fort unes t o use as signat ures. You'll find t he available fort unes in / usr/ share/ gam es/ fort une. I f your friends are Trekkies, m odify t he fortune line in your ~ / .signat ure like so: /usr/games/fortune -s startrek

I f t hey t end t o be cynical, t ry murphy inst ead.

9.11.2 Pipeless Signature Rotation Som e MUAs, such as Mozilla's m ailer, don't support pipes. You'll know yours doesn't if your t est m essage produces no fort une. Fort unat ely, t here's anot her opt ion. Creat e a file as before, but t his t im e m ake it a Bourne script . I 'll save m ine in ~ / bin and m ake it execut able using chmod +x: #!/bin/sh echo "Your random fortune:" > $HOME/.signature /usr/games/fortune -s >> $HOME/.signature

This script does t wo t hings. I t echoes t he first line t o t he ~ / .signat ure file, t hen appends t he result s of t he fortune program t o t he sam e file. To configure Mozilla t o use t his signat ure file, open t he Mail & Newsgroups window, and choose Mail & Newsgroups Account Set t ings from t he Edit m enu. Select t he " At t ach t his signat ure" opt ion from t he m ain m enu, and use t he Choose but t on t o give t he locat ion of ~ / .signat ure. What do you t hink will happen when I com pose an em ail? Since Mozilla only underst ands lit eral signat ure files, it will fait hfully reproduce t he current cont ent s of ~ / .signat ure. I f I haven't run m y script yet , t hat file doesn't exist . I f I have run t he script , t he result ing file rem ains t he sam e unt il t he script runs again. This is different from pine, which has t he capabilit y of execut ing t he com m ands found in m y signat ure file. Since Mozilla can't , you'll have t o rem em ber t o run t he script m anually before you com pose an em ail or schedule it s periodic execut ion using cron. This m ay be a lit t le disappoint ing if you want every recipient t o receive a unique signat ure, or not a big deal if you send only one or t wo em ails a day and aren't a st ickler for random ness.

9.11.3 Hacking the Hack Hm m , what would happen if .signat ure were a nam ed pipe connect ed t o a program t hat provided a random signat ure on every read? There are m any possibilit ies here.

9.11.4 See Also man

fortune

- 496 -

Hack 99 Useful One-Liners

Unix is am azing. Only your im aginat ion lim it s t he usefulness of t he built - in com m ands. You can creat e your own com m ands and t hen pipe t hem t oget her, allowing one ut ilit y t o work on t he result s of anot her. I f you're like m e, you've run across dozens of useful com binat ions over t he years. Here are som e of m y favorit e one- liners, int ended t o dem onst rat e useful ideas as well as t o prim e your pum p for writ ing your own one- liner hacks.

9.12.1 Simultaneously Download and Untar Have you ever downloaded an ext rem ely large archive over a slow connect ion? I t seem s t o t ake forever t o receive t he archive and forever t o unt ar it . Being im pat ient , I hat e not knowing how m any of t he archived files are already here. I m iss t he abilit y t o work on t hose files while t he rest of t he archive finishes it s slow m igrat ion ont o m y syst em . This one- liner will decom press and unt ar t he files as t he archive downloads, wit hout int erfering wit h t he download. Here's an exam ple of downloading and unt arring t he port s collect ion: # tail -f -b=1m ports.tar.gz | tar -zxvf ports.tar.gz ports/ ports/Mk/

Here I 've asked tail t o st ream up t o one m egabyt e of t he specified file as it is received. I t will pipe t hose byt es t o t he tar ut ilit y, which I 've direct ed t o decom press ( -z) and t o ext ract ( x) t he specified file ( f) while displaying t he result s verbosely ( v) . To use t his com m and, download t he archive t o where you'd like t o unt ar it —in t his exam ple, / usr. Sim ply replace t he filenam e port s.t ar.gz wit h t he nam e of your archive.

9.12.2 When Did I Change That File? Do you ever need t o know t he last m odificat ion dat e of a file? Consider a long list ing: % ls -l filename -rw-r--r--

1 dru

wheel

12962 Dec 16 18:01 filename

I f you count t he fields, t he sixt h ( Dec) , sevent h ( 16) , and eight h ( 18:01) fields all cont ain part of t he m odificat ion dat e. However, t here's whit espace separat ing t hose fields, which m akes it difficult t o det erm ine t heir exact charact er posit ions. Fort unat ely, awk doesn't m ind variable whit espace, so t his one- liner will always work: % echo filename was last modified on `/bin/ls -l filename \

- 497 -

| awk '{print $6, $7, $8}'` filename was last modified on Dec 16 18:01

Here I 've asked echo t o repeat a st ring as well as t he result s of a com m and cont ained wit hin single quot es. The first half of t hat com m and is sim ply ls -l filename. I 've piped t he out put of t hat com m and t o awk, which will print t he sixt h ( $6) , sevent h ( $7) , and eight h ( $8) fields of t he long list ing. Not e t hat t he awk act ion is enclosed bet ween '{ }'. While t his is a useful one- liner, it is fairly awkward t o t ype as needed. However, if you replace filename wit h a posit ional param et er [ H a ck # 1 3 ] , you have a very handy script . I 'll call m ine when: % more ~/bin/when #!/bin/sh # script to list date of a file's last modification # replaces $1 with specified filename # or gives error message if user forgets to include filename if test $1 then echo $1 was last modified on `/bin/ls -l $1| awk '{print $6, $7, $8}'` else echo "Don't forget the name of the file you're interested in" exit 1 fi

Once you've m ade your script execut able, use when filename t o find t he dat e of a file's m ost recent m odificat ion.

9.12.3 Finding Symlinks I f you ever need t o find sym bolic links, you're in luck. find's -type l or link opt ion serves j ust t his purpose. St art wit h t his invocat ion: % find /etc -type l -ls 25298

0 lrwxrwxrwx

1 root

wheel

23 Apr

2003 /etc/termcap -> /usr/share/misc/termcap 25299 Apr

7

25301

0 lrwxrwxrwx

1 root

wheel

13

wheel

12

2003 /etc/rmt -> /usr/sbin/rmt 0 lrwxrwxrwx

1 root

- 498 -

7

Apr

7

25305

2003 /etc/aliases -> mail/aliases 0 lrwxr-xr-x

1 root

wheel

36

Oct 26 09:08 /etc/localtime -> /usr/share/zoneinfo/America/Montreal

Well, t hat worked, but t he out put is downright ugly. Let 's pipe t he result s t o our good friend awk t o display only t he last t hree fields. I f you count t hem , t hose are fields 11 t hrough 13: % find /etc -type l -ls | awk '{print $11, $12, $13}' /etc/termcap -> /usr/share/misc/termcap /etc/rmt -> /usr/sbin/rmt /etc/aliases -> mail/aliases /etc/localtime -> /usr/share/zoneinfo/America/Montreal

Aah, m uch bet t er. I f you ever plan on needing t o find sym links, it 's well wort h saving t his in a shell script sim ilar t o t he when script shown previously.

9.12.4 Making cron More User-Friendly Are you always forget t ing t he m eanings of t he various fields in a crontab? I t would probably be a lot easier if your crontab began like t his: # minute (0-59), # |

hour (0-23),

# |

|

day of the month (1-31),

# |

|

|

month of the year (1-12),

# |

|

|

|

day of the week (0-6 with 0=Sunday).

# |

|

|

|

|

3

2

*

*

0,6

commands /some/command/to/run

To achieve t hat , t ype t hose lines int o a t ext file, say ~ / cronheader. ( Be pat ient , we're get t ing t o t he one- liner.) Then, open up your crontab edit or: % crontab -e

Unless you've changed your default edit or, t his will open up your crontab using vi. Place your cursor at t he beginning of t he file, and t ype t he following: !!more /usr/home/dru/cronheader

- 499 -

The !! t ells vi t o insert t he out put of t he specified com m and. Be sure t o give t he full pat hnam e t o your file. vi will insert it s cont ent s for you once you press Ent er. When you're finished, t ype :wq as usual t o exit t he edit or.

9.12.5 See Also • • • •

man man man man

tail tar cut awk

- 500 -

Hack 100 Fun with X

Use t h e u t ilit ie s t ha t com e w it h t h e cor e X dist r ibu t ion. There are so m any GUI ut ilit ies, available eit her as part of your favorit e Window Manager or as a separat e inst allat ion, t hat you can forget t hat t he core X dist ribut ion also provides several useful and light weight program s. Do you need t o m onit or console m essages, m anage your clipboard, send pop- up m essages, or creat e and view screenshot s? Before you hit t he port s collect ion, give t he built - in ut ilit ies a t ry.

9.13.1 Seeing Console Messages I n [ H a ck # 4 2 ] , we saw how t o redirect console m essages. I f you're using an X session, t he xconsole ut ilit y fulfills t his purpose. To st art t his ut ilit y, sim ply t ype it s nam e int o an xterm or use t he Run com m and provided by your window m anager. By default , only t he superuser can st art xconsole. A regular user will inst ead receive a Couldn't open console m essage. This is a safet y precaut ion on m ult iuser syst em s, prevent ing regular users from viewing syst em m essages. I f you're t he only user who uses your syst em , rem ove t he com m ent ( #) from t his line in / et c/ fbt ab: #/dev/ttyv0

0600

/dev/console

I f you spend a lot of your t im e at an X session, consider adding xconsole t o your ~ / .xinit rc file so it will st art aut om at ically ( see [ H a ck # 9 ] ) .

9.13.2 Managing Your Clipboard I f you do a lot of copying and past ing, xclipboard is anot her excellent candidat e for aut om at ic st art up. This ut ilit y st ores each of your clipboard select ions as a separat e ent it y, allowing you t o scroll t hrough t hem one at a t im e in a sim ple GUI window. I n addit ion t o t he Next and Prev but t ons, a Delet e but t on let s you rem ove unwant ed it em s and a Save but t on allows you t o save all of your it em s as a file.

9.13.3 Sending Pop-up Messages Do you find yourself st art ing a com m and t hat t akes a while t o execut e, cont inuing your work in an X session, t hen ret urning periodically t o t he original t erm inal or xterm t o see how t hat com m and is perking along? Wouldn't it be easier t o send yourself a pop- up m essage once t he com m and com plet es? For exam ple, suppose I want t o know when t he script from [ H a ck # 8 0 ] finishes. I could execut e t hat script as follows: #~/bin/mycustomupgrade.sh && xmessage -nearmouse cvsup is complete.

When t he upgrade com plet es, a pop- up m essage wit h t he t ext cvsup is complete. will appear in m y X session near m y m ouse. That m essage will disappear once I click on t he Okay but t on.

- 501 -

I f you're in t he habit of using su -l t o provide a new login when you becom e t he superuser, you'll find t hat t he preceding com m and will fail t o send you a pop- up m enu. ( I 'm assum ing you're logged in as a regular user when you start your X session. You should be! ) I nst ead, you'll receive t his error m essage: Xlib: connection to ":0.0" refused by server Xlib: No protocol specified Error: Can't open display: :0.0

This has t o do wit h t he X aut horizat ion process. I f I st art m y X session as t he user dru and use su t o execut e a com m and, I 'm st ill logged in as dru, so I 'm allowed t o send a m essage t o m y display. However, if I use su -l t o execut e t he com m and, I 'm no longer logged in as dru but as root. The X server refuses t o let anot her user int erfere wit h m y display, which is a good t hing. A quick workaround is t o not use su -l when sending pop- up m essages t o your regular user account . An alt ernat ive is t o underst and t he X aut horizat ion process. You can t hen use t his knowledge t o enable t he superuser t o send a m essage t o any user on any display. 9 .1 3 .3 .1 Un de r st a ndin g X a ut h or iza t ion Your X server uses a t oken known as an MI T m agic cookie t o provide aut horizat ion. When you st art your X session, t he server creat es and st ores t his unique cookie in ~ / .Xaut horit y. You can view it at any t im e using t his com m and: % xauth list genisis/unix:0

MIT-MAGIC-COOKIE-1

7e7bc20f9413469a7376e2e5c91aa6f1

Take not e t hat you're t he only user wit h access t o t his file: % ls -l ~/.Xauthority -rw-------

1

dru

wheel

101

Feb 18 13:28 .Xauthority

Always keep in t he back of your m ind, t hough, t hat file ownership does not m at t er t o t he superuser. For exam ple, if I need t o send an im port ant m essage t o t he user dru, I can ssh int o t he syst em she's working on and becom e t he superuser. Then: # cp ~dru/.Xauthority .

I now have a copy of dru's m agic cookie. However, before I can use it , I 'll first have t o change m y display. Since I sshed int o a t erm inal, I current ly don't have one: # echo $DISPLAY DISPLAY: Undefined variable.

I don't want j ust any display, I want t he display dru is current ly using. I can find t he nam e of her display by reading her m agic cookie:

- 502 -

# xauth list genisis/unix:0

MIT-MAGIC-COOKIE-1

7e7bc20f9413469a7376e2e5c91aa6f1

The nam e of her display is genisis/unix:0, where genisis represent s t he host nam e of t he syst em . I 'll now at t ach t o t hat display and send m y m essage: # setenv DISPLAY genisis/unix:0 # xmessage -nearmouse Time to go home, Dru... (prompt hangs until dru responds by pressing the "Okay" button)

This cheat works on any syst em t o which you have superuser access. Technically, you can execut e any com m and X underst ands in a user's X session once you have his cookie and display. Do rem em ber t o use your superuser powers for good, t hough.

9.13.4 Taking Screenshots Have you ever needed t o send a user a screenshot ? There are port s available for t his purpose, but t he built - in X com m and xwd will suffice. Creat ing a screenshot is a sim ple m at t er of: % xwd -out screenshot.xwd

The com m and will appear t o hang as it wait s for you t o click your m ouse on t he port ion of t he screen you'd like t o capt ure. Use t he -root swit ch t o capt ure t he ent ire screen and save yourself a click. You can view and m anipulat e t he result ing file wit h m ost t hird- part y im age edit ors, including xv and gimp. For quick viewing, t hough, not hing beat s t he built - in xwud: % xwud -in screenshot.xwd

Your result s won't seem t hat im pressive if you use xwud im m ediat ely, as your screen st ill probably looks like your screenshot . When you're finished viewing t he screenshot , press Ct rl- c.

9.13.5 See Also • • • • •

man man man man man

xconsole xclipboard xauth xwd xwud

- 503 -