Learning Go: An Idiomatic Approach to Real-world Go Programming, 2nd Edition [2 ed.] 1098139291, 9781098139292

Citation preview

nd co ion Se dit E

Learning

Go

An Idiomatic Approach to Real-World Go Programming

Jon Bodner

Learning Go Go has rapidly become the preferred language for building web services. Plenty of tutorials are available to teach Go’s syntax to developers with experience in other programming languages, but tutorials aren’t enough. They don’t teach Go’s idioms, so developers end up recreating patterns that don’t make sense in a Go context. This practical guide provides the essential background you need to write clear and idiomatic Go. No matter your level of experience, you’ll learn how to think like a Go developer. Author Jon Bodner introduces the design patterns experienced Go developers have adopted and explores the rationale for using them. This updated edition also includes a new chapter on Go tooling. This book helps you:

• Write idiomatic code in Go and design a Go project • Understand the reasons behind Go’s design decisions • Set up a Go development environment for solo developers or teams

• Learn how and when to use reflection, unsafe, and cgo • Discover how Go’s features allow the language to run efficiently

• Know which features you should use sparingly or not at all • Improve performance, optimize memory usage, and reduce garbage collection

• Learn how to use Go’s advanced development tools

GO / PROGR AMMING L ANGUAGES

US $65.99 CAN $82.99 ISBN: 978-1-098-13929-2

“Go is unique and even experienced programmers have to unlearn a few things and think differently about software. Learning Go does a good job of working through the big features of the language while pointing out idiomatic code, pitfalls, and design patterns along the way.” —Aaron Schlesinger

Senior Engineer, Microsoft

Jon Bodner is a staff engineer at Datadog, where he leads the effort to simplify onboarding to the company’s APM products. He’s also the creator of the Proteus data access library. He’s been a software engineer, lead developer, and architect for over 25 years.

Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia

Praise for Learning Go, Second Edition

“The first edition of Learning Go was an excellent starting point for any developer interested in learning Go, and the second edition is even better. This book is thorough without being monotonous, which is perfect for introducing newcomers to the Go ecosystem.” —Jonathan Hall, Go Developer and Content Creator "Learning Go does more than teach Go, it teaches good idiomatic Go. It is the perfect book for programmers familiar with other languages who want to learn Go.” —Chris Hines, Senior Principal Software Engineer, Comcast

Praise for Learning Go, First Edition

“Go is unique and even experienced programmers have to unlearn a few things and think differently about software. Learning Go does a good job of working through the big features of the language while pointing out idiomatic code, pitfalls, and design patterns along the way.” —Aaron Schlesinger, Senior Engineer, Microsoft “Jon has been a critical voice in the Go community for many years and we have been strongly benefitted from his talks and articles. With Learning Go, Jon has written the programmers’ guide to learning Go. It strikes just the right balance of giving a good overview of what you need to know without rehashing well understood concepts from other languages.” —Steve Francia, Go language product lead, Google, and author of Hugo, Cobra, and Viper “Bodner gets Go. In clear, lively prose, he teaches the language from its basics to advanced topics like reflection and C interop. He demonstrates through numerous examples how to write idiomatic Go, with its emphasis on clarity and simplicity. He also takes the time to explain the underlying concepts that can affect your program’s behavior, like the effects of pointers on memory layout and garbage collection. Beginners who read this book will come up to speed quickly, and even experienced Go programmers are likely to learn something.” —Jonathan Amsterdam, Software Engineer on the Go team at Google

“Learning Go is the essential introduction to what makes the Go programming language unique as well as the design patterns and idioms that make it so powerful. Jon Bodner manages to connect the fundamentals of the language to Go’s philosophy, guiding readers to write Go the way it was meant to be written.” —Robert Liebowitz, Software Engineer at Morning Consult “Jon wrote a book that does more than just reference Go; it provides an idiomatic and practical understanding of the language. Jon’s industry experience is what drives this book, and it will help those looking to be immediately productive in the language.” —William Kennedy, Managing Partner at Ardan Labs

SECOND EDITION

Learning Go

An Idiomatic Approach to Real-World Go Programming

Jon Bodner

Beijing

Boston Farnham Sebastopol

Tokyo

Learning Go by Jon Bodner Copyright © 2024 Jon Bodner. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected].

Acquisitions Editor: Suzanne McQuade Developmental Editor: Rita Fernando Production Editor: Beth Kelly Copyeditor: Kim Cofer Proofreader: Sharon Wilkey March 2021: January 2024:

Indexer: Sue Klefstad Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea

First Edition Second Edition

Revision History for the Second Edition 2024-01-11:

First Release

See http://oreilly.com/catalog/errata.csp?isbn=0636920787686 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning Go, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-13929-2 [LSI]

Table of Contents

Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii 1. Setting Up Your Go Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Installing the Go Tools Troubleshooting Your Go Installation Go Tooling Your First Go Program Making a Go Module go build go fmt go vet Choose Your Tools Visual Studio Code GoLand The Go Playground Makefiles The Go Compatibility Promise Staying Up-to-Date Exercises Wrapping Up

1 2 2 3 3 4 5 7 8 8 9 10 12 13 14 14 15

2. Predeclared Types and Declarations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 The Predeclared Types The Zero Value Literals Booleans Numeric Types A Taste of Strings and Runes

17 17 18 19 20 25 vii

Explicit Type Conversion Literals Are Untyped var Versus := Using const Typed and Untyped Constants Unused Variables Naming Variables and Constants Exercises Wrapping Up

26 27 28 30 32 32 33 35 36

3. Composite Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Arrays—Too Rigid to Use Directly Slices len append Capacity make Emptying a Slice Declaring Your Slice Slicing Slices copy Converting Arrays to Slices Converting Slices to Arrays Strings and Runes and Bytes Maps Reading and Writing a Map The comma ok Idiom Deleting from Maps Emptying a Map Comparing Maps Using Maps as Sets Structs Anonymous Structs Comparing and Converting Structs Exercises Wrapping Up

37 39 41 41 42 43 44 45 46 49 50 51 52 56 57 58 59 59 59 60 61 63 64 65 66

4. Blocks, Shadows, and Control Structures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Blocks Shadowing Variables if for, Four Ways

viii

|

Table of Contents

67 68 71 72

The Complete for Statement The Condition-Only for Statement The Infinite for Statement break and continue The for-range Statement Labeling Your for Statements Choosing the Right for Statement switch Blank Switches Choosing Between if and switch goto—Yes, goto Exercises Wrapping Up

72 73 74 75 76 82 83 84 87 89 89 92 92

5. Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Declaring and Calling Functions Simulating Named and Optional Parameters Variadic Input Parameters and Slices Multiple Return Values Multiple Return Values Are Multiple Values Ignoring Returned Values Named Return Values Blank Returns—Never Use These! Functions Are Values Function Type Declarations Anonymous Functions Closures Passing Functions as Parameters Returning Functions from Functions defer Go Is Call by Value Exercises Wrapping Up

93 94 95 96 97 97 98 99 100 103 103 105 107 108 109 114 116 117

6. Pointers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 A Quick Pointer Primer Don’t Fear the Pointers Pointers Indicate Mutable Parameters Pointers Are a Last Resort Pointer Passing Performance The Zero Value Versus No Value The Difference Between Maps and Slices

119 123 125 129 130 131 131

Table of Contents

|

ix

Slices as Buffers Reducing the Garbage Collector’s Workload Tuning the Garbage Collector Exercises Wrapping Up

135 136 139 141 141

7. Types, Methods, and Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Types in Go Methods Pointer Receivers and Value Receivers Code Your Methods for nil Instances Methods Are Functions Too Functions Versus Methods Type Declarations Aren’t Inheritance Types Are Executable Documentation iota Is for Enumerations—Sometimes Use Embedding for Composition Embedding Is Not Inheritance A Quick Lesson on Interfaces Interfaces Are Type-Safe Duck Typing Embedding and Interfaces Accept Interfaces, Return Structs Interfaces and nil Interfaces Are Comparable The Empty Interface Says Nothing Type Assertions and Type Switches Use Type Assertions and Type Switches Sparingly Function Types Are a Bridge to Interfaces Implicit Interfaces Make Dependency Injection Easier Wire Go Isn’t Particularly Object-Oriented (and That’s Great) Exercises Wrapping Up

143 144 145 148 149 150 150 151 152 154 156 157 158 162 162 164 165 166 167 170 173 174 178 178 178 179

8. Generics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Generics Reduce Repetitive Code and Increase Type Safety Introducing Generics in Go Generic Functions Abstract Algorithms Generics and Interfaces Use Type Terms to Specify Operators Type Inference and Generics Type Elements Limit Constants

x

|

Table of Contents

181 184 187 188 190 193 193

Combining Generic Functions with Generic Data Structures More on comparable Things That Are Left Out Idiomatic Go and Generics Adding Generics to the Standard Library Future Features Unlocked Exercises Wrapping Up

194 196 198 199 201 201 201 202

9. Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 How to Handle Errors: The Basics Use Strings for Simple Errors Sentinel Errors Errors Are Values Wrapping Errors Wrapping Multiple Errors Is and As Wrapping Errors with defer panic and recover Getting a Stack Trace from an Error Exercises Wrapping Up

203 205 205 208 210 212 214 217 218 220 221 221

10. Modules, Packages, and Imports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Repositories, Modules, and Packages Using go.mod Use the go Directive to Manage Go Build Versions The require Directive Building Packages Importing and Exporting Creating and Accessing a Package Naming Packages Overriding a Package’s Name Documenting Your Code with Go Doc Comments Using the internal Package Avoiding Circular Dependencies Organizing Your Module Gracefully Renaming and Reorganizing Your API Avoiding the init Function if Possible Working with Modules Importing Third-Party Code Working with Versions

223 224 225 226 227 227 227 230 230 231 234 235 236 238 239 240 240 245

Table of Contents

|

xi

Minimal Version Selection Updating to Compatible Versions Updating to Incompatible Versions Vendoring Using pkg.go.dev Publishing Your Module Versioning Your Module Overriding Dependencies Retracting a Version of Your Module Using Workspaces to Modify Modules Simultaneously Module Proxy Servers Specifying a Proxy Server Using Private Repositories Additional Details Exercises Wrapping Up

247 248 248 250 251 251 252 254 255 255 259 259 260 260 261 261

11. Go Tooling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Using go run to Try Out Small Programs Adding Third-Party Tools with go install Improving Import Formatting with goimports Using Code-Quality Scanners staticcheck revive golangci-lint Using govulncheck to Scan for Vulnerable Dependencies Embedding Content into Your Program Embedding Hidden Files Using go generate Working with go generate and Makefiles Reading the Build Info Inside a Go Binary Building Go Binaries for Other Platforms Using Build Tags Testing Versions of Go Using go help to Learn More About Go Tooling Exercises Wrapping Up

263 264 266 267 268 269 270 272 274 277 278 281 281 283 284 285 286 286 286

12. Concurrency in Go. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 When to Use Concurrency Goroutines Channels

xii

|

Table of Contents

287 289 291

Reading, Writing, and Buffering Using for-range and Channels Closing a Channel Understanding How Channels Behave select Concurrency Practices and Patterns Keep Your APIs Concurrency-Free Goroutines, for Loops, and Varying Variables Always Clean Up Your Goroutines Use the Context to Terminate Goroutines Know When to Use Buffered and Unbuffered Channels Implement Backpressure Turn Off a case in a select Time Out Code Use WaitGroups Run Code Exactly Once Put Your Concurrent Tools Together When to Use Mutexes Instead of Channels Atomics—You Probably Don’t Need These Where to Learn More About Concurrency Exercises Wrapping Up

291 292 292 293 294 297 297 298 299 300 301 302 304 304 306 308 309 313 317 317 317 318

13. The Standard Library. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 io and Friends time Monotonic Time Timers and Timeouts encoding/json Using Struct Tags to Add Metadata Unmarshaling and Marshaling JSON, Readers, and Writers Encoding and Decoding JSON Streams Custom JSON Parsing net/http The Client The Server ResponseController Structured Logging Exercises Wrapping Up

319 324 326 327 327 327 329 330 331 332 336 336 337 342 344 347 347

Table of Contents

|

xiii

14. The Context. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 What Is the Context? Values Cancellation Contexts with Deadlines Context Cancellation in Your Own Code Exercises Wrapping Up

349 352 358 363 367 368 369

15. Writing Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 Understanding the Basics of Testing Reporting Test Failures Setting Up and Tearing Down Testing with Environment Variables Storing Sample Test Data Caching Test Results Testing Your Public API Using go-cmp to Compare Test Results Running Table Tests Running Tests Concurrently Checking Your Code Coverage Fuzzing Using Benchmarks Using Stubs in Go Using httptest Using Integration Tests and Build Tags Finding Concurrency Problems with the Data Race Detector Exercises Wrapping Up

371 373 373 376 376 377 377 378 380 382 384 386 393 397 402 405 406 408 408

16. Here Be Dragons: Reflect, Unsafe, and Cgo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 Reflection Lets You Work with Types at Runtime Types, Kinds, and Values Make New Values Use Reflection to Check If an Interface’s Value Is nil Use Reflection to Write a Data Marshaler Build Functions with Reflection to Automate Repetitive Tasks You Can Build Structs with Reflection, but Don’t Reflection Can’t Make Methods Use Reflection Only if It’s Worthwhile unsafe Is Unsafe Using Sizeof and Offsetof

xiv

|

Table of Contents

410 411 415 417 417 422 423 424 424 425 426

Using unsafe to Convert External Binary Data Accessing Unexported Fields Using unsafe Tools Cgo Is for Integration, Not Performance Exercises Wrapping Up

428 432 433 433 438 438

Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441

Table of Contents

|

xv

Preface

In the preface of the first edition, I wrote: My first choice for a book title was Boring Go because, properly written, Go is boring…. Boring does not mean trivial. Using Go correctly requires an understanding of how its features are intended to fit together. While you can write Go code that looks like Java or Python, you’re going to be unhappy with the result and wonder what all the fuss is about. That’s where this book comes in. It walks through the features of Go, explaining how to best use them to write idiomatic code that can grow.

Go remains a small language with a small feature set. It still lacks inheritance, aspect-oriented programming, function overloading, operator overloading, pattern matching, named parameters, exceptions, and many additional features that compli‐ cate other languages. So why does a book on a boring language need an update? There are a few reasons for this edition. First, just as boring doesn’t mean trivial, it also does not mean unchanging. Over the past three years, new features, tools, and libraries have arrived. Improvements like structured logging, fuzzing, workspaces, and vulnerability checking help Go developers create reliable, lasting, maintainable code. Now that Go developers have several years of experience with generics, the standard library has started to include type constraints and generic functions to reduce repetitive code. Even the unsafe package has been updated to make it a little safer. Go developers need a resource that explains how to best use these new features. Secondly, some aspects of Go weren’t done justice by the first edition. The introduc‐ tory chapter didn’t flow as well as I’d like. The rich Go tool ecosystem wasn’t explored. And first-edition readers asked for exercises and additional sample code. This edition attempts to address those limitations. Finally, the Go team has introduced something new, and, dare I say, exciting. There’s now a strategy that allows Go to keep the backward compatibility required for long-term software engineering projects while providing the ability to introduce

xvii

backward-breaking changes to address long-standing design flaws. The new for loop variable scoping rules introduced in Go 1.22 are the first feature to take advantage of this approach. Go is still boring, it’s still fantastic, and it’s better than ever. I hope you enjoy this second edition.

Who Should Read This Book This book is targeted at developers who are looking to pick up a second (or fifth) language. The focus is on people who are new to Go. This ranges from those who don’t know anything about Go other than it has a cute mascot, to those who have already worked through a Go tutorial or even written some Go code. The focus for Learning Go isn’t just how to write programs in Go; it’s how to write Go idiomatically. More experienced Go developers can find advice on how to best use the newer features of the language. The most important thing is that the reader wants to learn how to write Go code that looks like Go. Experience is assumed with the tools of the developer trade, such as version control (preferably Git) and IDEs. Readers should be familiar with basic computer science concepts like concurrency and abstraction, as the book explains how they work in Go. Some of the code examples are downloadable from GitHub, and dozens more can be tried out online on The Go Playground. While an internet connection isn’t required, it is helpful when reviewing executable examples. Since Go is often used to build and call HTTP servers, some examples assume familiarity with basic HTTP concepts. While most of Go’s features are found in other languages, Go makes different trade‐ offs, so programs written in it have a different structure. Learning Go starts by look‐ ing at how to set up a Go development environment, and then covers variables, types, control structures, and functions. If you are tempted to skip over this material, resist the urge and take a look. It is often the details that make your Go code idiomatic. Some of what seems obvious at first glance might actually be subtly surprising when you think about it in depth.

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions.

xviii

|

Preface

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold

Shows commands or other text that should be typed literally by the user. Constant width italic

Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion.

This element signifies a general note.

This element indicates a warning or caution.

Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/learning-go-book-2e. If you have a technical question or a problem using the code examples, please send email to [email protected]. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant

Preface

|

xix

amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning Go by Jon Bodner (O’Reilly). Copyright 2024 Jon Bodner, 978-1-098-13929-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected].

O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) [email protected] https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/learning-go-2ed. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media Follow us on Twitter: https://twitter.com/oreillymedia Watch us on YouTube: https://youtube.com/oreillymedia xx

|

Preface

Acknowledgments for the 2nd Edition I’ve been humbled and flattered by the response to the first edition of Learning Go. It was started in late 2019, completed in late 2020, and released in early 2021. It was, unintentionally, my own pandemic project. Some people baked bread; I explained pointers. After I finished, people asked me what my next book would be. I told them that I was going to take a well-deserved break and then get to work on a romance novel. But before I could start writing about pirates and princesses, something amazing happened: Learning Go was a huge success. It spent the better part of a year as one of the five most popular books on O’Reilly Learning. As sales continued, I saw things in the first edition that I wanted to fix and received letters from readers pointing out errors and omissions. Some were fixed as errata in later printings, but I contacted O’Reilly to see if they would like a new edition, and they were interested. Returning to Learning Go and O’Reilly for the second edition was a pleasant experi‐ ence. Rita Fernando provided guidance, feedback, and editing. This edition has been made immeasurably better by feedback from Jonathan Amsterdam, Leam Hall, Katie Hockman, Thomas Hunter, Max Horstmann, and Natalie Pistunovich. Chris Hines did an incredibly thorough job finding my mistakes and suggesting better examples. The comments and feedback from Abby Deng and the members of the Go Book Club at Datadog allowed me to makes changes based on feedback from developers new to Go. The (hopefully few) remaining errors are all mine. My wife and children were kind enough to not mind me skipping out on family movie nights while I figured out the best way to describe new Go features. I also want to thank the readers of the first edition who reached out to me with their kind words. I thank you all for your support and encouragement.

Acknowledgments for the 1st Edition Writing a book seems like a solitary task, but it doesn’t happen without the help of a great number of people. I mentioned to Carmen Andoh that I wanted to write a book on Go and at GopherCon 2019, and she introduced me to Zan McQuade at O’Reilly. Zan guided me through the acquisition process and continued to provide me advice while I was writing Learning Go. Michele Cronin edited the text, gave feedback, and listened during the inevitable rough patches. Tonya Trybula’s copyediting and Beth Kelly’s production editing made my draft production-quality. While writing, I received critical feedback (and encouragement) from many people including Jonathan Altman, Jonathan Amsterdam, Johnny Ray Austin, Chris Fauer‐ bach, Chris Hines, Bill Kennedy, Tony Nelson, Phil Pearl, Liz Rice, Aaron Schlesinger, Chris Stout, Kapil Thangavelu, Claire Trivisonno, Volker Uhrig, Jeff Wendling, and Preface

|

xxi

Kris Zaragoza. I’d especially like to recognize Rob Liebowitz, whose detailed notes and rapid responses made this book far better than it would have been without his efforts. My family put up with me spending nights and weekends at the computer instead of with them. In particular, my wife, Laura, graciously pretended that I didn’t wake her up when I’d come to bed at 1 A.M. or later. Finally, I want to remember the two people who started me on this path four decades ago. The first is Paul Goldstein, the father of a childhood friend. In 1982, Paul showed us a Commodore PET, typed PRINT 2 + 2, and hit the Enter key. I was amazed when the screen said 4 and was instantly hooked. He later taught me how to program and even let me borrow the PET for a few weeks. Second, I’d like to thank my mother for encouraging my interest in programming and computers, despite having no idea what any of it was for. She bought me the BASIC programming cartridge for the Atari 2600, a VIC-20, and then a Commodore 64, along with the programming books that inspired me to want to write my own someday. Thank you all for helping make this dream of mine come true.

xxii

|

Preface

CHAPTER 1

Setting Up Your Go Environment

Every programming language needs a development environment, and Go is no exception. If you’ve already built a Go program or two, then you have a working environment, but you might have missed out on some of the newer techniques and tools. If this is your first time setting up Go on your computer, don’t worry; it’s easy to install Go and its supporting tools. After setting up the environment and verifying it, you will build a simple program, learn about the different ways to build and run Go code, and then explore some tools and techniques that make Go development easier.

Installing the Go Tools To build Go code, you need to download and install the Go development tools. You can find the latest version of the tools at the downloads page on the Go website. Choose the download for your platform and install it. The .pkg installer for Mac and the .msi installer for Windows automatically install Go in the correct location, remove any old installations, and put the Go binary in the default executable path. If you are a Mac developer, you can install Go using Homebrew with the command brew install go. Windows developers who use Chocolatey can install Go with the command choco install golang.

The various Linux and BSD installers are gzipped TAR files and expand to a directory named go. Copy this directory to /usr/local and add /usr/local/go/bin to your $PATH so that the go command is accessible:

1

$ tar -C /usr/local -xzf go1.20.5.linux-amd64.tar.gz $ echo 'export PATH=$PATH:/usr/local/go/bin' >> $HOME/.bash_profile $ source $HOME/.bash_profile

You might need root permissions to write to /usr/local. If the tar command fails, rerun it with sudo tar -C /usr/local -xzf go1.20.5.linux-amd64.tar.gz. Go programs compile to a single native binary and do not require any additional software to be installed in order to run them. This is in contrast to languages like Java, Python, and JavaScript, which require you to install a virtual machine to run your program. Using a single native binary makes it a lot easier to distribute programs written in Go. This book doesn’t cover containers, but developers who use Docker or Kubernetes can often package a Go app inside a scratch or distroless image. You can find details in Geert Baeke’s blog post “Distroless or Scratch for Go Apps”.

You can validate that your environment is set up correctly by opening up a terminal or command prompt and typing: $ go version

If everything is set up correctly, you should see something like this printed: go version go1.20.5 darwin/arm64

This tells you that this is Go version 1.20.5 on macOS. (Darwin is the operating system at the heart of macOS, and arm64 is the name for the 64-bit chips based on ARM’s designs.) On x64 Linux, you would see: go version go1.20.5 linux/amd64

Troubleshooting Your Go Installation If you get an error instead of the version message, it’s likely that you don’t have go in your executable path, or you have another program named go in your path. On macOS and other Unix-like systems, use which go to see the go command being executed, if any. If nothing is returned, you need to fix your executable path. If you’re on Linux or BSD, it’s possible you installed the 64-bit Go development tools on a 32-bit system or the development tools for the wrong chip architecture.

Go Tooling All of the Go development tools are accessed via the go command. In addition to go version, there’s a compiler (go build), code formatter (go fmt), dependency manager (go mod), test runner (go test), a tool that scans for common coding mistakes (go vet), and more. They are covered in detail in Chapters 10, 11, and 2

|

Chapter 1: Setting Up Your Go Environment

15. For now, let’s take a quick look at the most commonly used tools by writing everyone’s favorite first application: “Hello, World!"” Since the introduction of Go in 2009, several changes have occur‐ red in the way Go developers organize their code and their depen‐ dencies. Because of this churn, there’s lots of conflicting advice, and most of it is obsolete (for example, you can safely ignore discussions about GOROOT and GOPATH). For modern Go development, the rule is simple: you are free to organize your projects as you see fit and store them anywhere you want.

Your First Go Program Let’s go over the basics of writing a Go program. Along the way, you will see the parts that make up a simple Go program. You might not understand everything just yet, and that’s OK; that’s what the rest of the book is for!

Making a Go Module The first thing you need to do is create a directory to hold your program. Call it ch1. On the command line, enter the new directory. If your computer’s terminal uses bash or zsh, this looks like the following: $ mkdir ch1 $ cd ch1

Inside the directory, run the go mod init command to mark this directory as a Go module: $ go mod init hello_world go: creating new go.mod: module hello_world

You will learn more about what a module is in Chapter 10, but for now, all you need to know is that a Go project is called a module. A module is not just source code. It is also an exact specification of the dependencies of the code within the module. Every module has a go.mod file in its root directory. Running go mod init creates this file for you. The contents of a basic go.mod file look like this: module hello_world go 1.20

The go.mod file declares the name of the module, the minimum supported version of Go for the module, and any other modules that your module depends on. You can think of it as being similar to the requirements.txt file used by Python or the Gemfile used by Ruby. Your First Go Program

|

3

You shouldn’t edit the go.mod file directly. Instead, use the go get and go mod tidy commands to manage changes to the file. Again, all things related to modules are covered in Chapter 10.

go build Now let’s write some code! Open up a text editor, enter the following text, and save it inside ch1 with the filename hello.go: package main import "fmt" func main() { fmt.Println("Hello, world!") }

(Yes, the indentation in this code example looks sloppy: I meant to do that! You will see why in just a bit.) Let’s quickly go over the parts of the Go file you created. The first line is a package declaration. Within a Go module, code is organized into one or more packages. The main package in a Go module contains the code that starts a Go program. Next there is an import declaration. The import statement lists the packages refer‐ enced in this file. You’re using a function in the fmt (usually pronounced “fumpt”) package from the standard library, so you list the package here. Unlike other lan‐ guages, Go imports only whole packages. You can’t limit the import to specific types, functions, constants, or variables within a package. All Go programs start from the main function in the main package. You declare this function with func main() and a left brace. Like Java, JavaScript, and C, Go uses braces to mark the start and end of code blocks. The body of the function consists of a single line. It says that you are calling the Println function in the fmt package with the argument "Hello, world!". As an experienced developer, you can likely guess what this function call does. After the file is saved, go back to your terminal or command prompt and type: $ go build

This creates an executable called hello_world (or hello_world.exe on Windows) in the current directory. Run it and you will unsurprisingly see Hello, world! printed on the screen: $ ./hello_world Hello, world!

4

|

Chapter 1: Setting Up Your Go Environment

The name of the binary matches the name in the module declaration. If you want a different name for your application, or if you want to store it in a different location, use the -o flag. For example, if you wanted to compile the code to a binary called “hello,” you would use the following: $ go build -o hello

In “Using go run to Try Out Small Programs” on page 263, I will cover another way to execute a Go program.

go fmt One of the chief design goals for Go was to create a language that allowed you to write code efficiently. This meant having simple syntax and a fast compiler. It also led Go’s authors to reconsider code formatting. Most languages allow a great deal of flexibility in the way code is formatted. Go does not. Enforcing a standard format makes it a great deal easier to write tools that manipulate source code. This simplifies the compiler and allows the creation of some clever tools for generating code. There is a secondary benefit as well. Developers have historically wasted extraordi‐ nary amounts of time on format wars. Since Go defines a standard way of formatting code, Go developers avoid arguments over brace style and tabs versus spaces. For example, Go programs use tabs to indent, and it is a syntax error if the opening brace is not on the same line as the declaration or command that begins the block. Many Go developers think the Go team defined a standard format as a way to avoid developer arguments and discovered the tooling advantages later. However, Russ Cox, the development lead for Go, has publicly stated that better tooling was his original motivation.

The Go development tools include a command, go fmt, which automatically fixes the whitespace in your code to match the standard format. However, it can’t fix braces on the wrong line. Run it with the following: $ go fmt ./... hello.go

Using ./... tells a Go tool to apply the command to all the files in the current directory and all subdirectories. You will see it again as you learn about more Go tools. If you open up hello.go, you’ll see that the line with fmt.Println is now properly indented with a single tab.

Your First Go Program

|

5

Remember to run go fmt before you compile your code, and, at the very least, before you commit source code changes to your repository! If you forget, make a separate commit that does only go fmt ./... so you don’t hide logic changes in an avalanche of formatting changes.

The Semicolon Insertion Rule The go fmt command won’t fix braces on the wrong line because of the semicolon insertion rule. Like C or Java, Go requires a semicolon at the end of every statement. However, Go developers should never put the semicolons in themselves. The Go compiler adds them automatically, following a simple rule described in Effective Go. If the last token before a newline is any of the following, the lexer inserts a semicolon after the token: • An identifier (which includes words like int and float64) • A basic literal such as a number or string constant • One of the tokens: break, continue, fallthrough, return, ++, --, ), or } With this simple rule in place, you can see why putting a brace in the wrong place breaks. If you write your code like this: func main() { fmt.Println("Hello, world!") }

the semicolon insertion rule sees the ) at the end of the func main() line and turns that into: func main(); { fmt.Println("Hello, world!"); };

And that’s not valid Go. The semicolon insertion rule and the resulting restriction on brace placement is one of the things that makes the Go compiler simpler and faster, while at the same time enforcing a coding style. That’s clever.

6

|

Chapter 1: Setting Up Your Go Environment

go vet In one class of bugs, the code is syntactically valid but quite likely incorrect. The go tool includes a command called go vet to detect these kinds of errors. Add one to the program and watch it get detected. Change the fmt.Println line in hello.go to the following: fmt.Printf("Hello, %s!\n")

fmt.Printf is similar to printf in C, Java, Ruby, and many other languages. If you haven’t seen fmt.Printf before, it is a function

with a template for its first parameter, and values for the placehold‐ ers in the template in the remaining parameters.

In this example, you have a template ("Hello, %s!\n") with a %s placeholder, but no value was specified for the placeholder. This code will compile and run, but it’s not correct. One of the things that go vet detects is whether a value exists for every placeholder in a formatting template. Run go vet on the modified code, and it finds an error: $ go vet ./... # hello_world ./hello.go:6:2: fmt.Printf format %s reads arg #1, but call has 0 args

Now that go vet found the bug, you can easily fix it. Change line 6 in hello.go to: fmt.Printf("Hello, %s!\n", "world")

While go vet catches several common programming errors, there are things that it cannot detect. Luckily, third-party Go code-quality tools can close the gap. Some of the most popular code-quality tools are covered in “Using Code-Quality Scanners” on page 267. Just as you should run go fmt to make sure your code is formatted properly, run go vet to scan for possible bugs in valid code. These commands are just the first step in ensuring that your code is of high quality. In addition to the advice in this book, all Go developers should read through Effective Go and the Code Review Comments page on Go’s wiki to understand what idiomatic Go code looks like.

Your First Go Program

|

7

Choose Your Tools While you wrote a small Go program using nothing more than a text editor and the go command, you’ll probably want more advanced tools when working on larger projects. Go IDEs provide many advantages over text editors, including automatic formatting on save, code completion, type checking, error reporting, and integrated debugging. Excellent Go development tools are available for most text editors and IDEs. If you don’t already have a favorite tool, two of the most popular Go development environments are Visual Studio Code and GoLand.

Visual Studio Code If you are looking for a free development environment, Visual Studio Code from Microsoft is your best option. Since it was released in 2015, VS Code has become the most popular source code editor for developers. It does not ship with Go support, but you can make it a Go development environment by downloading the Go extension from the extensions gallery. VS Code’s Go support relies on third-party extensions that are accessed via its built-in Marketplace. This includes the Go Development tools, the Delve debugger, and gopls, a Go language server developed by the Go team. While you need to install the Go compiler yourself, the Go extension will install Delve and gopls for you. What is a language server? It’s a standard specification for an API that enables editors to implement intelligent editing behavior, like code completion, quality checks, or finding all the places a variable or function is used in your code. You can learn more about lan‐ guage servers and their capabilities by checking out the Language Server Protocol website.

Once your tools are set up, you can open your project and work with it. Figure 1-1 shows you what your project window should look like. “Getting Started with VS Code Go” is a walkthrough that demonstrates the VS Code Go extension.

8

| Chapter 1: Setting Up Your Go Environment

Figure 1-1. Visual Studio Code

GoLand GoLand is the Go-specific IDE from JetBrains. While JetBrains is best known for Java-centric tools, GoLand is an excellent Go development environment. As you can see in Figure 1-2, GoLand’s user interface looks similar to IntelliJ, PyCharm, RubyMine, WebStorm, Android Studio, or any of the other JetBrains IDEs. Its Go support includes refactoring, syntax highlighting, code completion and navigation, documentation pop-ups, a debugger, code coverage, and more. In addition to Go support, GoLand includes JavaScript/HTML/CSS and SQL database tools. Unlike VS Code, GoLand doesn’t require you to install a plug-in to get it to work.

Choose Your Tools

|

9

Figure 1-2. GoLand If you have already subscribed to IntelliJ Ultimate, you can add Go support via a plug-in. While GoLand is commercial software, JetBrains has a Free License Program for students and core open source contributors. If you don’t qualify for a free license, a 30-day free trial is available. After that, you have to pay for GoLand.

The Go Playground There’s one more important tool for Go development, but this is one that you don’t install. Visit The Go Playground and you’ll see a window that resembles Figure 1-3. If you have used a command-line environment like irb, node, or python, you’ll find The Go Playground has a similar feel. It gives you a place to try out and share small programs. Enter your program into the window and click the Run button to execute the code. The Format button runs go fmt on your program and updates your imports. The Share button creates a unique URL that you can send to someone else to take a look at your program or to come back to your code at a future date (the URLs have proven to be persistent for a long time, but I wouldn’t rely on the playground as your source code repository).

10

| Chapter 1: Setting Up Your Go Environment

Figure 1-3. The Go Playground As you can see in Figure 1-4, you can simulate multiple files by separating each file with a line that looks like -- filename.go --. You can even create simulated subdirectories by including a / in the filename, such as -- subdir/my_code.go -- . Be aware that The Go Playground is on someone else’s computer (in particular, Google’s computer), so you don’t have completely free rein. It gives you a choice of a few versions of Go (usually the current release, the previous release, and the latest development version). You can make network connections only to localhost, and processes that run for too long or use too much memory are stopped. If your program depends on time, you need to take into account that the clock is set to November 10, 2009, 23:00:00 UTC (the date of the initial announcement of Go). Even with these limitations, The Go Playground is a useful way to try out new ideas without creating a new project locally. Throughout this book, you’ll find links to The Go Playground so you can run code examples without copying them onto your computer. Do not put sensitive information (such as personally identifiable information, passwords, or private keys) into your playground! If you click the Share button, the information is saved on Google’s servers and is accessible to anyone who has the associ‐ ated Share URL. If you do this by accident, contact Google at [email protected] with the URL and the reason the content needs to be removed.

Choose Your Tools

|

11

Figure 1-4. The Go Playground supports multiple files

Makefiles An IDE is nice to use, but it’s hard to automate. Modern software development relies on repeatable, automatable builds that can be run by anyone, anywhere, at any time. Requiring this kind of tooling is good software engineering practice. It avoids the age-old situation where a developer absolves themselves of any build problems with a shrug and the statement, “It works on my machine!” The way to do this is to use some kind of script to specify your build steps. Go developers have adopted make as their solution. It lets developers specify a set of operations that are necessary to build a program and the order in which the steps must be performed. You may not be familiar with make, but it’s been used to build programs on Unix systems since 1976. Create a file called Makefile in the ch1 directory with the following contents: .DEFAULT_GOAL := build .PHONY:fmt vet build fmt: go fmt ./... vet: fmt go vet ./... build: vet go build

12

|

Chapter 1: Setting Up Your Go Environment

Even if you haven’t seen a Makefile before, figuring out what’s going on is not too difficult. Each possible operation is called a target. The .DEFAULT_GOAL defines which target is run when no target is specified. In this case, the default is the build target. Next you have the target definitions. The word before the colon (:) is the name of the target. Any words after the target (like vet in the line build: vet) are the other targets that must be run before the specified target runs. The tasks that are performed by the target are on the indented lines after the target. The .PHONY line keeps make from getting confused if a directory or file in your project has the same name as one of the listed targets. Run make and you should see the following output: $ make go fmt ./... go vet ./... go build

Entering a single command formats the code correctly, checks it for nonobvious errors, and compiles it. You can also vet the code with make vet, or just run the formatter with make fmt. This might not seem like a big improvement, but ensuring that formatting and vetting always happen before a developer (or a script running on a continuous integration build server) triggers a build means you won’t miss any steps. One drawback to Makefiles is that they are exceedingly picky. You must indent the steps in a target with a tab. They are also not supported out-of-the-box on Windows. If you are doing your Go development on a Windows computer, you need to install make first. The easiest way to do so is to first install a package manager like Chocola‐ tey and then use it to install make (for Chocolatey, the command is choco install make). If you want to learn more about writing Makefiles, there’s a good tutorial by Chase Lambert, but it does use a tiny bit of C to explain the concepts. You can find the code from this chapter in the Chapter 1 repository for this book.

The Go Compatibility Promise As with all programming languages, the Go development tools are periodically upda‐ ted. Since Go 1.2, a new release has occurred roughly every six months. Patch releases with bug and security fixes are also released as needed. Given the rapid development cycles and the Go team’s commitment to backward compatibility, Go releases tend to be incremental rather than expansive. The Go Compatibility Promise is a detailed description of how the Go team plans to avoid breaking Go code. It says that there won’t be backward-breaking changes to the language or the standard library for any Go version that starts with 1, unless the change is required for a bug or security fix. In The Go Compatibility Promise

|

13

his GopherCon 2022 keynote talk “Compatibility: How Go Programs Keep Working”, Russ Cox discusses all the ways that the Go Team works to keep Go code from breaking. He says, “I believe that prioritizing compatibility was the most important design decision that we made in Go 1.” This guarantee doesn’t apply to the go commands. There have been backwardincompatible changes to the flags and functionality of the go commands, and it’s entirely possible that it will happen again.

Staying Up-to-Date Go programs compile to a standalone native binary, so you don’t need to worry that updating your development environment could cause your currently deployed programs to fail. You can have programs compiled with different versions of Go running simultaneously on the same computer or virtual machine. When you are ready to update the Go development tools installed on your computer, Mac and Windows users have the easiest path. Those who installed with brew or chocolatey can use those tools to update. Those who used the installers on https:// golang.org/dl can download the latest installer, which removes the old version when it installs the new one. Linux and BSD users need to download the latest version, move the old version to a backup directory, unpack the new version, and then delete the old version: $ mv /usr/local/go /usr/local/old-go $ tar -C /usr/local -xzf go1.20.6.linux-amd64.tar.gz $ rm -rf /usr/local/old-go

Technically, you don’t need to move the existing installation to a new location; you could just delete it and install the new version. However, this falls in the “better safe than sorry” category. If some‐ thing goes wrong while installing the new version, it’s good to have the previous one around.

Exercises Each chapter has exercises at the end to let you try out the ideas that I cover. You can find answers to these exercises in the Chapter 1 repository. 1. Take the “Hello, world!” program and run it on The Go Playground. Share a link to the code in the playground with a coworker who would love to learn about Go. 2. Add a target to the Makefile called clean that removes the hello_world binary and any other temporary files created by go build. Take a look at the Go command documentation to find a go command to help implement this. 14

| Chapter 1: Setting Up Your Go Environment

3. Experiment with modifying the formatting in the “Hello, world!” program. Add blank lines, spaces, change indentation, insert newlines. After making a modifi‐ cation, run go fmt to see if the formatting change is undone. Also, run go build to see if the code still compiles. You can also add additional fmt.Println calls so you can see what happens if you put blank lines in the middle of a function.

Wrapping Up In this chapter, you learned how to install and configure your Go development environment. You also learned about tools for building Go programs and ensuring code quality. Now that your environment is ready, you’re on to the next chapter, where you’ll explore the built-in types in Go and how to declare variables.

Wrapping Up

|

15

CHAPTER 2

Predeclared Types and Declarations

Now that you have your development environment set up, it’s time to start looking at Go’s language features and how to best use them. When trying to figure out what “best” means, there is one overriding principle: write your programs in a way that makes your intentions clear. As I go through features and discuss the options, I’ll explain why I find that a particular approach produces clearer code. I’ll start by looking at the types that are built into Go and how to declare variables of those types. While every programmer has experience with these concepts, Go does some things differently, and subtle differences exist between Go and other languages.

The Predeclared Types Go has many types built into the language. These are called predeclared types. They are similar to types that are found in other languages: booleans, integers, floats, and strings. Using these types idiomatically is sometimes a challenge for developers who are transitioning from another language. You’ll look at these types and see how they work best in Go. Before I review the types, let’s cover some of the concepts that apply to all types.

The Zero Value Go, like most modern languages, assigns a default zero value to any variable that is declared but not assigned a value. Having an explicit zero value makes code clearer and removes a source of bugs found in C and C++ programs. As I talk about each type, I will also cover the zero value for the type. You can find details on the zero value in The Go Programming Language Specification.

17

Literals A Go literal is an explicitly specified number, character, or string. Go programs have four common kinds of literals. (I’ll cover a rare fifth kind of literal when discussing complex numbers.) An integer literal is a sequence of numbers. Integer literals are base 10 by default, but different prefixes are used to indicate other bases: 0b for binary (base 2), 0o for octal (base 8), or 0x for hexadecimal (base 16). You can use either upper- or lowercase letters for the prefix. A leading 0 with no letter after it is another way to represent an octal literal. Do not use it, as it is very confusing. To make it easier to read longer integer literals, Go allows you to put underscores in the middle of your literal. This allows you to, for example, group by thousands in base 10 (1_234). These underscores have no effect on the value of the number. The only limitations on underscores are that they can’t be at the beginning or end of numbers, and you can’t have them next to each other. You could put an underscore between every digit in your literal (1_2_3_4), but don’t. Use them to improve read‐ ability by breaking up base 10 numbers at the thousands place or to break up binary, octal, or hexadecimal numbers at 1-, 2-, or 4-byte boundaries. A floating-point literal has a decimal point to indicate the fractional portion of the value. They can also have an exponent specified with the letter e and a positive or negative number (such as 6.03e23). You also have the option to write them in hexadecimal by using the 0x prefix and the letter p for indicating any exponent (0x12.34p5, which is equal to 582.5 in base 10). As integer literals, you can use underscores to format your floating-point literals. A rune literal represents a character and is surrounded by single quotes. Unlike many other languages, in Go single quotes and double quotes are not interchange‐ able. Rune literals can be written as single Unicode characters ('a'), 8-bit octal numbers ('\141'), 8-bit hexadecimal numbers ('\x61'), 16-bit hexadecimal num‐ bers ('\u0061'), or 32-bit Unicode numbers ('\U00000061'). There are also several backslash-escaped rune literals, with the most useful ones being newline ('\n'), tab ('\t'), single quote ('\''), and backslash ('\\'). Practically speaking, use base 10 to represent your integer and floating-point literals. Octal representations are rare, mostly used to represent POSIX permission flag values (such as 0o777 for rwxrwxrwx). Hexadecimal and binary are sometimes used for bit filters or networking and infrastructure applications. Avoid using any of the numeric escapes for rune literals, unless the context makes your code clearer.

18

|

Chapter 2: Predeclared Types and Declarations

There are two ways to indicate string literals. Most of the time, you should use double quotes to create an interpreted string literal (e.g., type "Greetings and Salutations"). These contain zero or more rune literals. They are called “inter‐ preted” because they interpret rune literals (both numeric and backslash escaped) into single characters. One rune literal backslash escape is not legal in a string literal: the single quote escape. It is replaced by a backslash escape for double quotes.

The only characters that cannot appear in an interpreted string literal are unesca‐ ped backslashes, unescaped newlines, and unescaped double quotes. If you use an interpreted string literal and want your greetings on a different line from your salutations and want “Salutations” to appear in quotes, you need to type "Greetings and\n\"Salutations\"". If you need to include backslashes, double quotes, or newlines in your string, using a raw string literal is easier. These are delimited with backquotes (`) and can contain any character except a backquote. There’s no escape character in a raw string literal; all characters are included as is. When using a raw string literal, you write a multiline greeting like so: `Greetings and "Salutations"`

Literals are considered untyped. I’ll explore this concept more in “Literals Are Unty‐ ped” on page 27. As you will see in “var Versus :=” on page 28, there are situations in Go where the type isn’t explicitly declared. In those cases, Go uses the default type for a literal; if there’s nothing in the expression that makes clear what the type of the literal is, the literal defaults to a type. I will mention the default type for literals when discussing the different predeclared types.

Booleans The bool type represents Boolean variables. Variables of bool type can have one of two values: true or false. The zero value for a bool is false: var flag bool // no value assigned, set to false var isAwesome = true

It’s hard to talk about variable types without showing a variable declaration, and vice versa. I’ll use variable declarations first and describe them in “var Versus :=” on page 28.

The Predeclared Types

|

19

Numeric Types Go has a large number of numeric types: 12 types (and a few special names) that are grouped into three categories. If you are coming from a language like JavaScript that gets along with only a single numeric type, this might seem like a lot. And in fact, some types are used frequently while others are more esoteric. I’ll start by looking at integer types before moving on to floating-point types and the very unusual complex type.

Integer types Go provides both signed and unsigned integers in a variety of sizes, from one to eight bytes. They are shown in Table 2-1. Table 2-1. The integer types in Go Type name Value range int8

–128 to 127

int16

–32768 to 32767

int32

–2147483648 to 2147483647

int64

–9223372036854775808 to 9223372036854775807

uint8

0 to 255

uint16

0 to 65535

uint32

0 to 4294967295

uint64

0 to 18446744073709551615

It might be obvious from the name, but the zero value for all of the integer types is 0.

The special integer types Go does have some special names for integer types. A byte is an alias for uint8; it is legal to assign, compare, or perform mathematical operations between a byte and a uint8. However, you rarely see uint8 used in Go code; just call it a byte. The second special name is int. On a 32-bit CPU, int is a 32-bit signed integer like an int32. On most 64-bit CPUs, int is a 64-bit signed integer, just like an int64. Because int isn’t consistent from platform to platform, it is a compile-time error to assign, compare, or perform mathematical operations between an int and an int32 or int64 without a type conversion (see “Explicit Type Conversion” on page 26 for more details). Integer literals default to being of int type.

20

|

Chapter 2: Predeclared Types and Declarations

Some uncommon 64-bit CPU architectures use a 32-bit signed integer for the int type. Go supports three of them: amd64p32, mips64p32, and mips64p32le.

The third special name is uint. It follows the same rules as int, only it is unsigned (the values are always 0 or positive). There are two other special names for integer types, rune and uintptr. You looked at rune literals earlier and I’ll discuss the rune type in “A Taste of Strings and Runes” on page 25 and uintptr in Chapter 16.

Choosing which integer to use Go provides more integer types than some other languages. Given all these choices, you might wonder when you should use each of them. You should follow three simple rules: • If you are working with a binary file format or network protocol that has an integer of a specific size or sign, use the corresponding integer type. • If you are writing a library function that should work with any integer type, take advantage of Go’s generics support and use a generic type parameter to represent any integer type (I talk more about functions and their parameters in Chapter 5 and more about generics in Chapter 8.) • In all other cases, just use int. You’ll likely find legacy code where there’s a pair of functions that do the same thing, but one has int64 for the parameters and variables and the other has uint64. The reason is that the API was created before generics were added to Go. Without generics, you needed to write functions with slightly different names to imple‐ ment the same algorithm with different types. Using int64 and uint64 meant that you could write the code once and let callers use type conversions to pass values in and convert data that’s returned. You can see this pattern in the Go standard library within the functions FormatInt and FormatUint in the strconv package.

Integer operators Go integers support the usual arithmetic operators: +, -, *, /, with % for modulus. The result of an integer division is an integer; if you want to get a floating-point result, you need to use a type conversion to make your integers into floating-point numbers.

The Predeclared Types

|

21

Also, be careful not to divide an integer by 0; this causes a panic (I talk more about panics in “panic and recover” on page 218). Integer division in Go follows truncation toward zero; see the Go spec’s section on arithmetic operators for the full details.

You can combine any of the arithmetic operators with = to modify a variable: +=, -=, *=, /=, and %=. For example, the following code results in x having the value 20: var x int = 10 x *= 2

You compare integers with ==, !=, >, >=, , >=, 5 { x, y := 5, 20 fmt.Println(x, y) } fmt.Println(x) }

Running this code gives you the following: 5 20 10

Although there was an existing definition of x in an outer block, x was still shadowed within the if statement. That’s because := reuses only variables that are declared in the current block. When using :=, make sure that you don’t have any variables from an outer scope on the lefthand side unless you intend to shadow them. You also need to be careful to ensure that you don’t shadow a package import. I’ll talk more about importing packages in Chapter 10, but you’ve been importing the fmt package to print out results of our programs. Let’s see what happens when you declare a variable called fmt within your main function, as shown in Example 4-3. You can try to run it on The Go Playground or in the sample_code/shadow_package_names directory in the Chapter 4 repository.

Shadowing Variables

|

69

Example 4-3. Shadowing package names func main() { x := 10 fmt.Println(x) fmt := "oops" fmt.Println(fmt) }

When you try to run this code, you get an error: fmt.Println undefined (type string has no field or method Println)

Notice that the problem isn’t that you named your variable fmt; it’s that you tried to access something that the local variable fmt didn’t have. Once the local variable fmt is declared, it shadows the package named fmt in the file block, making it impossible to use the fmt package for the rest of the main function.

The Universe Block One more block is a little weird: the universe block. Remember, Go is a small language with only 25 keywords. What’s interesting is that the built-in types (like int and string), constants (like true and false), and functions (like make or close) aren’t included in that list. Neither is nil. So, where are they? Rather than make them keywords, Go considers these predeclared identifiers and defines them in the universe block, which is the block that contains all other blocks. Because these names are declared in the universe block, they can be shadowed in other scopes. You can see this happen by running the code in Example 4-4 on The Go Playground or in the sample_code/shadow_true directory in the Chapter 4 repository.

Example 4-4. Shadowing true fmt.Println(true) true := 10 fmt.Println(true)

When you run it, you’ll see: true 10

You must be very careful to never redefine any identifiers in the universe block. If you accidentally do so, you will get some strange behavior. If you are lucky, you’ll get compilation failures. If you are not, you’ll have a harder time tracking down the source of your problems.

70

| Chapter 4: Blocks, Shadows, and Control Structures

Since shadowing is useful in a few instances (later chapters will point them out), go vet does not report it as a likely error. In “Using Code-Quality Scanners” on page 267, you’ll learn about third-party tools that can detect accidental shadowing in your code.

if The if statement in Go is much like the if statement in most programming lan‐ guages. Because it is such a familiar construct, I’ve used it in previous sample code without worrying that it’d be confusing. Example 4-5 shows a more complete sample. Example 4-5. if and else n := rand.Intn(10) if n == 0 { fmt.Println("That's too low") } else if n > 5 { fmt.Println("That's too big:", n) } else { fmt.Println("That's a good number:", n) }

The most visible difference between if statements in Go and other languages is that you don’t put parentheses around the condition. But Go adds another feature to if statements that helps you better manage your variables. As I discussed in “Shadowing Variables” on page 68, any variable declared within the braces of an if or else statement exists only within that block. This isn’t that uncommon; it is true in most languages. What Go adds is the ability to declare variables that are scoped to the condition and to both the if and else blocks. Take a look at Example 4-6, which rewrites our previous example to use this scope. Example 4-6. Scoping a variable to an if statement if n := rand.Intn(10); n == 0 { fmt.Println("That's too low") } else if n > 5 { fmt.Println("That's too big:", n) } else { fmt.Println("That's a good number:", n) }

Having this special scope is handy. It lets you create variables that are available only where they are needed. Once the series of if/else statements ends, n is undefined. You can test this by trying to run the code in Example 4-7 on The Go Playground or in the sample_code/if_bad_scope directory in the Chapter 4 repository. if

|

71

Example 4-7. Out of scope… if n := rand.Intn(10); n == 0 { fmt.Println("That's too low") } else if n > 5 { fmt.Println("That's too big:", n) } else { fmt.Println("That's a good number:", n) } fmt.Println(n)

Attempting to run this code produces a compilation error: undefined: n

Technically, you can put any simple statement before the compar‐ ison in an if statement—including a function call that doesn’t return a value or an assignment of a new value to an existing variable. But don’t do this. Use this feature only to define new variables that are scoped to the if/else statements; anything else would be confusing. Also be aware that just like any other block, a variable declared as part of an if statement will shadow variables with the same name that are declared in containing blocks.

for, Four Ways As in other languages in the C family, Go uses a for statement to loop. What makes Go different from other languages is that for is the only looping keyword in the language. Go accomplishes this by using the for keyword in four formats: • A complete, C-style for • A condition-only for • An infinite for • for-range

The Complete for Statement The first for loop style is the complete for declaration you might be familiar with from C, Java, or JavaScript, as shown in Example 4-8.

72

|

Chapter 4: Blocks, Shadows, and Control Structures

Example 4-8. A complete for statement for i := 0; i < 10; i++ { fmt.Println(i) }

You might be unsurprised to find that this program prints out the numbers from 0 to 9, inclusive. Just like the if statement, the for statement does not use parentheses around its parts. Otherwise, it should look familiar. The if statement has three parts, separated by semicolons. The first part is an initialization that sets one or more variables before the loop begins. You should remember two important details about the initialization section. First, you must use := to initialize the variables; var is not legal here. Second, just as variable declarations in if statements, you can shadow a variable here. The second part is the comparison. This must be an expression that evaluates to a bool. It is checked immediately before each iteration of the loop. If the expression evaluates to true, the loop body is executed. The last part of a standard for statement is the increment. You usually see something like i{plus}{plus} here, but any assignment is valid. It runs immediately after each iteration of the loop, before the condition is evaluated. Go allows you to leave out one or more of the three parts of the for statement. Most commonly, you’ll either leave off the initialization if it is based on a value calculated before the loop: i := 0 for ; i < 10; i++ { fmt.Println(i) }

or you’ll leave off the increment because you have a more complicated increment rule inside the loop: for i := 0; i < 10; { fmt.Println(i) if i % 2 == 0 { i++ } else { i+=2 } }

The Condition-Only for Statement When you leave off both the initialization and the increment in a for statement, do not include the semicolons. (If you do, go fmt will remove them.) That leaves a for, Four Ways

|

73

for statement that functions like the while statement found in C, Java, JavaScript, Python, Ruby, and many other languages. It looks like Example 4-9.

Example 4-9. A condition-only for statement i := 1 for i < 100 { fmt.Println(i) i = i * 2 }

The Infinite for Statement The third for statement format does away with the condition too. Go has a version of a for loop that loops forever. If you learned to program in the 1980s, your first program was probably an infinite loop in BASIC that printed HELLO to the screen forever: 10 PRINT "HELLO" 20 GOTO 10

Example 4-10 shows the Go version of this program. You can run it locally or try it out on The Go Playground or in the sample_code/infinite_for directory in the Chapter 4 repository. Example 4-10. Infinite looping nostalgia package main import "fmt" func main() { for { fmt.Println("Hello") } }

Running this program gives you the same output that filled the screens of millions of Commodore 64s and Apple ][s: Hello Hello Hello Hello Hello Hello Hello ...

74

|

Chapter 4: Blocks, Shadows, and Control Structures

Press Ctrl-C when you are tired of walking down memory lane. If you run Example 4-10 on The Go Playground, you’ll find that it will stop execution after a few seconds. As a shared resource, the playground doesn’t allow any one program to run for too long.

break and continue How do you get out of an infinite for loop without using the keyboard or turning off your computer? That’s the job of the break statement. It exits the loop immediately, just like the break statement in other languages. Of course, you can use break with any for statement, not just the infinite for statement. Go has no equivalent of the do keyword in Java, C, and JavaScript. If you want to iterate at least once, the cleanest way is to use an infinite for loop that ends with an if statement. If you have some Java code, for example, that uses a do/while loop: do { // things to do in the loop } while (CONDITION);

the Go version looks like this: for { // things to do in the loop if !CONDITION { break } }

Note that the condition has a leading ! to negate the condition from the Java code. The Go code is specifying how to exit the loop, while the Java code specifies how to stay in it.

Go also includes the continue keyword, which skips over the rest of the for loop’s body and proceeds directly to the next iteration. Technically, you don’t need a continue statement. You could write code like Example 4-11. Example 4-11. Confusing code for i := 1; i >> result, remainder = div_and_remainder(5,2) >>> result 2.5 >>> remainder 1

In Python, you can assign all the values returned to a single variable or to multiple variables. That’s not how Go works. You must assign each value returned from a function. If you try to assign multiple return values to one variable, you get a compile-time error.

Ignoring Returned Values What if you call a function and don’t want to use all the returned values? As covered in “Unused Variables” on page 32, Go does not allow unused variables. If a function returns multiple values, but you don’t need to read one or more of the values, assign the unused values to the name _. For example, if you weren’t

Declaring and Calling Functions

|

97

going to read remainder, you would write the assignment as result, _, err := divAndRemainder(5, 2). Surprisingly, Go does let you implicitly ignore all of the return values for a function. You can write divAndRemainder(5,2), and the returned values are dropped. You have actually been doing this since the earliest examples: fmt.Println returns two values, but it is idiomatic to ignore them. In almost all other cases, you should make it explicit that you are ignoring return values by using underscores. Use _ whenever you don’t need to read a value that’s returned by a function.

Named Return Values In addition to letting you return more than one value from a function, Go allows you to specify names for your return values. Rewrite the divAndRemainder function one more time, this time using named return values: func divAndRemainder(num, denom int) (result int, remainder int, err error) { if denom == 0 { err = errors.New("cannot divide by zero") return result, remainder, err } result, remainder = num/denom, num%denom return result, remainder, err }

When you supply names to your return values, what you are doing is predeclaring variables that you use within the function to hold the return values. They are written as a comma-separated list within parentheses. You must surround named return values with parentheses, even if there is only a single return value. Named return values are initialized to their zero values when created. This means that you can return them before any explicit use or assignment. If you want to name only some of the return values, you can do so by using _ as the name for any return values you want to remain nameless.

One important thing to note: the name that’s used for a named returned value is local to the function; it doesn’t enforce any name outside of the function. It is perfectly legal to assign the return values to variables of different names:

98

|

Chapter 5: Functions

func main() { x, y, z := divAndRemainder(5, 2) fmt.Println(x, y, z) }

While named return values can sometimes help clarify your code, they do have some potential corner cases. First is the problem of shadowing. Just as with any other variable, you can shadow a named return value. Be sure that you are assigning to the return value and not to a shadow of it. The other problem with named return values is that you don’t have to return them. Let’s take a look at another variation on divAndRemainder. You can run it on The Go Playground or in the divAndRemainderConfusing function in main.go in the sample_code/named_div directory in the Chapter 5 repository: func divAndRemainder(num, denom int) (result int, remainder int, err error) { // assign some values result, remainder = 20, 30 if denom == 0 { return 0, 0, errors.New("cannot divide by zero") } return num / denom, num % denom, nil }

Notice that you assigned values to result and remainder and then returned different values directly. Before running this code, try to guess what happens when 5 and 2 are passed to this function. The result might surprise you: 2 1

The values from the return statement were returned even though they were never assigned to the named return parameters. That’s because the Go compiler inserts code that assigns whatever is returned to the return parameters. The named return parameters give a way to declare an intent to use variables to hold the return values, but don’t require you to use them. Some developers like to use named return parameters because they provide addi‐ tional documentation. However, I find them of limited value. Shadowing makes them confusing, as does simply ignoring them. Named return parameters are essential in one situation. I will talk about that when I cover defer later in the chapter.

Blank Returns—Never Use These! If you use named return values, you need to be aware of one severe misfeature in Go: blank (sometimes called naked) returns. If you have named return values, you can just write return without specifying the values that are returned. This returns the last values assigned to the named return values. Rewrite the divAndRemainder function one last time, this time using blank returns:

Declaring and Calling Functions

|

99

func divAndRemainder(num, denom int) (result int, remainder int, err error) { if denom == 0 { err = errors.New("cannot divide by zero") return } result, remainder = num/denom, num%denom return }

Using blank returns makes a few additional changes to the function. When there’s invalid input, the function returns immediately. Since no values were assigned to result and remainder, their zero values are returned. If you are returning the zero values for your named return values, be sure they make sense. Also note that you still have to put a return at the end of the function. Even though the function is using blank returns, this function returns values. Leaving out return is a compile-time error. At first, you might find blank returns handy since they allow you to avoid some typing. However, most experienced Go developers consider blank returns a bad idea because they make it harder to understand data flow. Good software is clear and readable; it’s obvious what is happening. When you use a blank return, the reader of your code needs to scan back through the program to find the last value assigned to the return parameters to see what is actually being returned. If your function returns values, never use a blank return. It can make it very confusing to figure out what value is actually returned.

Functions Are Values Just as in many other languages, functions in Go are values. The type of a function is built out of the keyword func and the types of the parameters and return values. This combination is called the signature of the function. Any function that has the exact same number and types of parameters and return values meets the type signature. Since functions are values, you can declare a function variable: var myFuncVariable func(string) int

100

|

Chapter 5: Functions

myFuncVariable can be assigned any function that has a single parameter of type string and returns a single value of type int. Here’s a longer example: func f1(a string) int { return len(a) } func f2(a string) int { total := 0 for _, v := range a { total += int(v) } return total } func main() { var myFuncVariable func(string) int myFuncVariable = f1 result := myFuncVariable("Hello") fmt.Println(result) myFuncVariable = f2 result = myFuncVariable("Hello") fmt.Println(result) }

You can run this program on The Go Playground or in the sample_code/func_value directory in the Chapter 5 repository. The output is: 5 500

The default zero value for a function variable is nil. Attempting to run a function variable with a nil value results in a panic (which is covered in “panic and recover” on page 218). Having functions as values allows you to do some clever things, such as build a simple calculator using functions as values in a map. Let’s see how this works. The following code is available on The Go Playground or in the sample_code/calculator directory in the Chapter 5 repository. First, create a set of functions that all have the same signature: func add(i int, j int) int { return i + j } func sub(i int, j int) int { return i - j } func mul(i int, j int) int { return i * j } func div(i int, j int) int { return i / j }

Functions Are Values

|

101

Next, create a map to associate a math operator with each function: var opMap = map[string]func(int, int) int{ "+": add, "-": sub, "*": mul, "/": div, }

Finally, try out the calculator with a few expressions: func main() { expressions := [][]string{ {"2", "+", "3"}, {"2", "-", "3"}, {"2", "*", "3"}, {"2", "/", "3"}, {"2", "%", "3"}, {"two", "+", "three"}, {"5"}, } for _, expression := range expressions { if len(expression) != 3 { fmt.Println("invalid expression:", expression) continue } p1, err := strconv.Atoi(expression[0]) if err != nil { fmt.Println(err) continue } op := expression[1] opFunc, ok := opMap[op] if !ok { fmt.Println("unsupported operator:", op) continue } p2, err := strconv.Atoi(expression[2]) if err != nil { fmt.Println(err) continue } result := opFunc(p1, p2) fmt.Println(result) } }

You’re using the strconv.Atoi function in the standard library to convert a string to an int. The second value returned by this function is an error. Just as before, you check for errors that are returned by functions and handle error conditions properly. You use op as the key to the opMap map, and assign the value associated with the key to the variable opFunc. The type of opFunc is func(int, int) int. If there wasn’t a 102

| Chapter 5: Functions

function in the map associated with the provided key, you print an error message and skip the rest of the loop. You then call the function assigned to the opFunc variable with the p1 and p2 variables that you decoded earlier. Calling a function in a variable looks just like calling a function directly. When you run this program, you can see the simple calculator at work: 5 -1 6 0 unsupported operator: % strconv.Atoi: parsing "two": invalid syntax invalid expression: [5]

Don’t write fragile programs. The core logic for this example is relatively short. Of the 22 lines inside the for loop, 6 of them implement the actual algorithm, and the other 16 are for error checking and data validation. You might be tempted to not validate incoming data or check errors, but doing so produces unstable, unmaintainable code. Error handling is what separates the profes‐ sionals from the amateurs.

Function Type Declarations Just as you can use the type keyword to define a struct, you can use it to define a function type too (I’ll go into more details on type declarations in Chapter 7): type opFuncType func(int,int) int

You can then rewrite the opMap declaration to look like this: var opMap = map[string]opFuncType { // same as before }

You don’t have to modify the functions at all. Any function that has two input parameters of type int and a single return value of type int automatically meets the type and can be assigned as a value in the map. What’s the advantage of declaring a function type? One use is documentation. It’s useful to give something a name if you are going to refer to it multiple times. You will see another use in “Function Types Are a Bridge to Interfaces” on page 173.

Anonymous Functions You can not only assign functions to variables, but also define new functions within a function and assign them to variables. Here’s a program to demonstrate this, which

Functions Are Values

|

103

you can run on The Go Playground. The code is also available in the sample_code/ anon_func directory in the Chapter 5 repository: func main() { f := func(j int) { fmt.Println("printing", j, "from inside of an anonymous function") } for i := 0; i < 5; i++ { f(i) } }

Inner functions are anonymous; they don’t have a name. You declare an anonymous function with the keyword func immediately followed by the input parameters, the return values, and the opening brace. It is a compile-time error to try to put a function name between func and the input parameters. Just like any other function, an anonymous function is called by using parentheses. In this example, you are passing the i variable from the for loop in here. It is assigned to the j input parameter of the anonymous function. The program gives the following output: printing printing printing printing printing

0 1 2 3 4

from from from from from

inside inside inside inside inside

of of of of of

an an an an an

anonymous anonymous anonymous anonymous anonymous

function function function function function

You don’t have to assign an anonymous function to a variable. You can write them inline and call them immediately. The previous program can be rewritten into this: func main() { for i := 0; i < 5; i++ { func(j int) { fmt.Println("printing", j, "from inside of an anonymous function") }(i) } }

You can run this example on The Go Playground or in the sample_code/anon_func directory in the Chapter 5 repository. Now, this is not something that you would normally do. If you are declaring and executing an anonymous function immediately, you might as well get rid of the anonymous function and just call the code. However, declaring anonymous functions without assigning them to variables is useful in two situations: defer statements and launching goroutines. I’ll talk about defer statements in a bit. Goroutines are covered in Chapter 12.

104

|

Chapter 5: Functions

Since you can declare variables at the package scope, you can also declare package scope variables that are assigned anonymous functions: var ( add sub mul div )

= = = =

func(i, func(i, func(i, func(i,

j j j j

int) int) int) int)

int int int int

{ { { {

return return return return

i i i i

+ * /

j j j j

} } } }

func main() { x := add(2, 3) fmt.Println(x) }

Unlike a normal function definition, you can assign a new value to a package-level anonymous function: func main() { x := add(2, 3) fmt.Println(x) changeAdd() y := add(2, 3) fmt.Println(y) } func changeAdd() { add = func(i, j int) int { return i + j + j } }

Running this code gives the following output: 5 8

You can try this sample on The Go Playground. The code is also in the sam‐ ple_code/package_level_anon directory in the Chapter 5 repository. Before using a package-level anonymous function, be very sure you need this capabil‐ ity. Package-level state should be immutable to make data flow easier to understand. If a function’s meaning changes while a program is running, it becomes difficult to understand not just how data flows, but how it is processed.

Closures Functions declared inside functions are special; they are closures. This is a computer science word that means that functions declared inside functions are able to access and modify variables declared in the outer function. Let’s look at a quick example to see how this works. You can find the code on The Go Playground. The code is also in the sample_code/closure directory in the Chapter 5 repository:

Closures

|

105

func main() { a := 20 f := func() { fmt.Println(a) a = 30 } f() fmt.Println(a) }

Running this program gives the following output: 20 30

The anonymous function assigned to f can read and write a, even though a is not passed in to the function. Just as with any inner scope, you can shadow a variable inside a closure. If you change the code to: func main() { a := 20 f := func() { fmt.Println(a) a := 30 fmt.Println(a) } f() fmt.Println(a) }

this will print out the following: 20 30 20

Using := instead of = inside the closure creates a new a that ceases to exist when the closure exits. When working with inner functions, be careful to use the correct assignment operator, especially when multiple variables are on the lefthand side. You can try this code on The Go Playground. The code is also in the sample_code/clo‐ sure_shadow directory in the Chapter 5 repository. This inner function and closure stuff might not seem all that useful at first. What benefit do you get from making mini-functions within a larger function? Why does Go have this feature? One thing that closures allow you to do is to limit a function’s scope. If a function is going to be called from only one other function, but it’s called multiple times, you can use an inner function to “hide” the called function. This reduces the number of declarations at the package level, which can make it easier to find an unused name. 106

|

Chapter 5: Functions

If you have a piece of logic that is repeated multiple times within a function, a closure can be used to remove that repetition. I wrote a simple Lisp interpreter with a Scan function that processes an input string to find the parts of a Lisp program. It relies on two closures, buildCurToken and update, to make the code shorter and easier to understand. You can see it on GitHub. Closures really become interesting when they are passed to other functions or returned from a function. They allow you to take the variables within your function and use those values outside of your function.

Passing Functions as Parameters Since functions are values and you can specify the type of a function using its parameter and return types, you can pass functions as parameters into functions. If you aren’t used to treating functions like data, you might need a moment to think about the implications of creating a closure that references local variables and then passing that closure to another function. It’s a very useful pattern that appears several times in the standard library. One example is sorting slices. The sort package in the standard library has a function called sort.Slice. It takes in any slice and a function that is used to sort the slice that’s passed in. Let’s see how it works by sorting a slice of a struct using two different fields. The sort.Slice function predates the addition of generics to Go, so it does some internal magic to make it work with any kind of slice. I’ll talk about this magic more in Chapter 16.

Let’s see how to use closures to sort the same data different ways. You can run this code on The Go Playground. The code is also in the sample_code/sort_sample directory in the Chapter 5 repository. First, define a simple type, a slice of values of that type, and print out the slice: type Person struct { FirstName string LastName string Age int } people := []Person{ {"Pat", "Patterson", 37}, {"Tracy", "Bobdaughter", 23}, {"Fred", "Fredson", 18}, } fmt.Println(people)

Closures

|

107

Next, sort the slice by last name and print out the results: // sort by last name sort.Slice(people, func(i, j int) bool { return people[i].LastName < people[j].LastName }) fmt.Println(people)

The closure that’s passed to sort.Slice has two parameters, i and j, but within the closure, people is used, so you can sort it by the LastName field. In computer science terms, people is captured by the closure. Next you do the same, sorting by the Age field: // sort by age sort.Slice(people, func(i, j int) bool { return people[i].Age < people[j].Age }) fmt.Println(people)

Running this code gives the following output: [{Pat Patterson 37} {Tracy Bobdaughter 23} {Fred Fredson 18}] [{Tracy Bobdaughter 23} {Fred Fredson 18} {Pat Patterson 37}] [{Fred Fredson 18} {Tracy Bobdaughter 23} {Pat Patterson 37}]

The people slice is changed by the call to sort.Slice. I talk about this briefly in “Go Is Call by Value” on page 114 and in more detail in the next chapter. Passing functions as parameters to other functions is often useful for performing different operations on the same kind of data.

Returning Functions from Functions In addition to using a closure to pass some function state to another function, you can also return a closure from a function. Let’s demonstrate this by writing a function that returns a multiplier function. You can run this program on The Go Playground. The code is also in the sample_code/makeMult directory in the Chapter 5 repository. Here is a function that returns a closure:

108

|

Chapter 5: Functions

func makeMult(base int) func(int) int { return func(factor int) int { return base * factor } }

And here is how the function is used: func main() { twoBase := makeMult(2) threeBase := makeMult(3) for i := 0; i < 3; i++ { fmt.Println(twoBase(i), threeBase(i)) } }

Running this program gives the following output: 0 0 2 3 4 6

Now that you’ve seen closures in action, you might wonder how often they are used by Go developers. It turns out that they are surprisingly useful. You saw how they are used to sort slices. A closure is also used to efficiently search a sorted slice with sort.Search. As for returning closures, you will see this pattern used when you build middleware for a web server in “Middleware” on page 340. Go also uses closures to implement resource cleanup, via the defer keyword. If you spend any time with programmers who use functional programming languages like Haskell, you might hear the term higher-order functions. That’s a fancy way to say that a function has a function for an input parameter or a return value. As a Go developer, you are as cool as they are!

defer Programs often create temporary resources, like files or network connections, that need to be cleaned up. This cleanup has to happen, no matter how many exit points a function has, or whether a function completed successfully or not. In Go, the cleanup code is attached to the function with the defer keyword. Let’s take a look at how to use defer to release resources. You’ll do this by writing a simple version of cat, the Unix utility for printing the contents of a file. You can’t pass in arguments on The Go Playground, but you can find the code for this example in the sample_code/simple_cat directory in the Chapter 5 repository: func main() { if len(os.Args) < 2 {

defer

|

109

log.Fatal("no file specified") } f, err := os.Open(os.Args[1]) if err != nil { log.Fatal(err) } defer f.Close() data := make([]byte, 2048) for { count, err := f.Read(data) os.Stdout.Write(data[:count]) if err != nil { if err != io.EOF { log.Fatal(err) } break } } }

This example introduces a few new features that I cover in more detail in later chapters. Feel free to read ahead to learn more. First, you make sure that a filename was specified on the command line by checking the length of os.Args, a slice in the os package. The first value in os.Args is the name of the program. The remaining values are the arguments passed to the program. You check that the length of os.Args is at least 2 to determine whether the argument to the program was provided. If it wasn’t, use the Fatal function in the log package to print a message and exit the program. Next, you acquire a read-only file handle with the Open function in the os package. The second value that’s returned by Open is an error. If there’s a problem opening the file, you print the error message and exit the program. As mentioned earlier, I’ll talk about errors in Chapter 9. Once you know there is a valid file handle, you need to close it after you use it, no matter how you exit the function. To ensure that the cleanup code runs, you use the defer keyword, followed by a function or method call. In this case, you use the Close method on the file variable. (I cover at methods in Go in Chapter 7.) Normally, a function call runs immediately, but defer delays the invocation until the surrounding function exits. You read from a file handle by passing a slice of bytes into the Read method on a file variable. I’ll cover how to use this method in detail in “io and Friends” on page 319, but Read returns the number of bytes that were read into the slice and an error. If an error occurs, you check whether it’s an end-of-file marker. If you are at the end of the file, you use break to exit the for loop. For all other errors, you report it and exit immediately, using log.Fatal. I’ll talk a little more about slices and function

110

| Chapter 5: Functions

parameters in “Go Is Call by Value” on page 114 and go into details on this pattern when I discuss pointers in the next chapter. Building and running the program from within the simple_cat directory produces the following result: $ go build $ ./simple_cat simple_cat.go package main import ( "io" "log" "os" ) ...

You should know a few more things about defer. First, you can use a function, method, or closure with defer. (When I mention functions with defer, mentally expand that to “functions, methods, or closures.”) You can defer multiple functions in a Go function. They run in last-in, first-out (LIFO) order; the last defer registered runs first. The code within defer functions runs after the return statement. As I mentioned, you can supply a function with input parameters to a defer. The input parameters are evaluated immediately and their values are stored until the function runs. Here’s a quick example that demonstrates both of these features of defer: func deferExample() int { a := 10 defer func(val int) { fmt.Println("first:", val) }(a) a = 20 defer func(val int) { fmt.Println("second:", val) }(a) a = 30 fmt.Println("exiting:", a) return a }

Running this code gives the following output: exiting: 30 second: 20 first: 10

You can run this on The Go Playground. It is also available in the sample_code/ defer_example directory in the Chapter 5 repository. defer

|

111

You can supply a function that returns values to a defer, but there’s no way to read those values: func example() { defer func() int { return 2 // there's no way to read this value }() }

You might be wondering whether there’s a way for a deferred function to examine or modify the return values of its surrounding function. There is, and it’s the best reason to use named return values. It allows your code to take actions based on an error. When I talk about errors in Chapter 9, I will discuss a pattern that uses defer to add contextual information to an error returned from a function. Let’s look at a way to handle database transaction cleanup using named return values and defer: func DoSomeInserts(ctx context.Context, db *sql.DB, value1, value2 string) (err error) { tx, err := db.BeginTx(ctx, nil) if err != nil { return err } defer func() { if err == nil { err = tx.Commit() } if err != nil { tx.Rollback() } }() _, err = tx.ExecContext(ctx, "INSERT INTO FOO (val) values $1", value1) if err != nil { return err } // use tx to do more database inserts here return nil }

I’m not going to cover Go’s database support in this book, but the standard library includes extensive support for databases in the database/sql package. In the exam‐ ple function, you create a transaction to do a series of database inserts. If any of them fails, you want to roll back (not modify the database). If all of them succeed, you want to commit (store the database changes). You use a closure with defer to check whether err has been assigned a value. If it hasn’t, you run tx.Commit, which could also return an error. If it does, the value err is modified. If any database interaction returned an error, you call tx.Rollback.

112

|

Chapter 5: Functions

New Go developers tend to forget the set of parentheses after the braces when specifying a function for defer. It is a compile-time error to leave them out, and eventually the habit sets in. It helps to remember that supplying parentheses allows you to specify values that will be passed into the function when it runs.

A common pattern in Go is for a function that allocates a resource to also return a closure that cleans up the resource. In the sample_code/simple_cat_cancel directory in the Chapter 5 repository, there is a rewrite of the simple cat program that does this. First you write a helper function that opens a file and returns a closure: func getFile(name string) (*os.File, func(), error) { file, err := os.Open(name) if err != nil { return nil, nil, err } return file, func() { file.Close() }, nil }

The helper function returns a file, a function, and an error. That * means that a file reference in Go is a pointer. I’ll talk more about that in the next chapter. Now in main, you use the getFile function: f, closer, err := getFile(os.Args[1]) if err != nil { log.Fatal(err) } defer closer()

Because Go doesn’t allow unused variables, returning the closer from the function means that the program will not compile if the function is not called. That reminds the user to use defer. As covered earlier, you put parentheses after closer when you defer it.

defer

|

113

Using defer can feel strange if you are used to a language that uses a block within a function to control when a resource is cleaned up, like the try/catch/finally blocks in Java, JavaScript, and Python or the begin/rescue/ensure blocks in Ruby. The downside to these resource-cleanup blocks is that they create another level of indentation in your function, and that makes the code harder to read. It’s not just my opinion that nested code is harder to follow. In research described in a 2017 paper in Empirical Software Engineering, Vard Antinyan et al. discovered that “Of… eleven proposed code characteristics, only two markedly influence complexity growth: the nesting depth and the lack of structure.” Research on what makes a program easier to read and understand isn’t new. You can find papers that are many decades old, including a paper from 1983 by Richard Miara et al. that tries to figure out the right amount of indentation to use (according to their results, two to four spaces).

Go Is Call by Value You might hear people say that Go is a call-by-value language and wonder what that means. It means that when you supply a variable for a parameter to a function, Go always makes a copy of the value of the variable. Let’s take a look. You can run this code on The Go Playground. It is also in the sample_code/pass_value_type directory in the Chapter 5 repository. First, you define a simple struct: type person struct { age int name string }

Next, you write a function that takes in an int, a string, and a person, and modifies their values: func modifyFails(i int, s string, p person) { i = i * 2 s = "Goodbye" p.name = "Bob" }

You then call this function from main and see whether the modifications stick: func main() { p := person{} i := 2 s := "Hello" modifyFails(i, s, p) fmt.Println(i, s, p) }

114

|

Chapter 5: Functions

As the name of the function indicates, running this code shows that a function won’t change the values of the parameters passed into it: 2 Hello {0 }

I included the person struct to show that this isn’t true just for primitive types. If you have programming experience in Java, JavaScript, Python, or Ruby, you might find the struct behavior strange. After all, those languages let you modify the fields in an object when you pass an object as a parameter to a function. The reason for the difference is something I will cover when I talk about pointers. The behavior is a little different for maps and slices. Let’s see what happens when you try to modify them within a function. You can run this code on The Go Playground. It is also in the sample_code/pass_map_slice directory in the Chapter 5 repository. You’re going to write a function to modify a map parameter and a function to modify a slice parameter: func modMap(m map[int]string) { m[2] = "hello" m[3] = "goodbye" delete(m, 1) } func modSlice(s []int) { for k, v := range s { s[k] = v * 2 } s = append(s, 10) }

You then call these functions from main: func main() { m := map[int]string{ 1: "first", 2: "second", } modMap(m) fmt.Println(m) s := []int{1, 2, 3} modSlice(s) fmt.Println(s) }

When you run this code, you’ll see something interesting: map[2:hello 3:goodbye] [2 4 6]

For the map, it’s easy to explain what happens: any changes made to a map param‐ eter are reflected in the variable passed into the function. For a slice, it’s more Go Is Call by Value

|

115

complicated. You can modify any element in the slice, but you can’t lengthen the slice. This is true for maps and slices that are passed directly into functions as well as map and slice fields in structs. This program leads to the question: why do maps and slices behave differently than other types? It’s because maps and slices are both implemented with pointers. I’ll go into more detail in the next chapter. Every type in Go is a value type. It’s just that sometimes the value is a pointer.

Call by value is one reason that Go’s limited support for constants is only a minor handicap. Since variables are passed by value, you can be sure that calling a function doesn’t modify the variable whose value was passed in (unless the variable is a slice or map). In general, this is a good thing. It makes it easier to understand the flow of data through your program when functions don’t modify their input parameters and instead return newly computed values. While this approach is easy to understand, at times you need to pass something mutable to a function. What do you do then? That’s when you need a pointer.

Exercises These exercises test your knowledge of functions in Go and how to use them. Solutions are available in the exercise_solutions directory in the Chapter 5 repository. 1. The simple calculator program doesn’t handle one error case: division by zero. Change the function signature for the math operations to return both an int and an error. In the div function, if the divisor is 0, return errors.New("division by zero") for the error. In all other cases, return nil. Adjust the main function to check for this error. 2. Write a function called fileLen that has an input parameter of type string and returns an int and an error. The function takes in a filename and returns the number of bytes in the file. If there is an error reading the file, return the error. Use defer to make sure the file is closed properly. 3. Write a function called prefixer that has an input parameter of type string and returns a function that has an input parameter of type string and returns a string. The returned function should prefix its input with the string passed into prefixer. Use the following main function to test prefixer:

116

|

Chapter 5: Functions

func main() { helloPrefix := prefixer("Hello") fmt.Println(helloPrefix("Bob")) // should print Hello Bob fmt.Println(helloPrefix("Maria")) // should print Hello Maria }

Wrapping Up In this chapter, you’ve looked at functions in Go, how they are similar to functions in other languages, and their unique features. In the next chapter, you’re going to look at pointers, find out that they aren’t nearly as scary as many new Go developers expect them to be, and learn how to take advantage of them to write efficient programs.

Wrapping Up

|

117

CHAPTER 6

Pointers

Now that you’ve seen variables and functions, it’s time to learn about pointer syntax. Then I’ll clarify the behavior of pointers in Go by comparing them to the behavior of classes in other languages. You’ll also learn how and when to use pointers, how memory is allocated in Go, and how using pointers and values properly makes Go programs faster and more efficient.

A Quick Pointer Primer A pointer is a variable that holds the location in memory where a value is stored. If you’ve taken computer science courses, you might have seen a graphic to represent the way variables are stored in memory. The representation of the following two variables would look something like Figure 6-1: var x int32 = 10 var y bool = true

Figure 6-1. Storing two variables in memory Every variable is stored in one or more contiguous memory locations, called addresses. Different types of variables can take up different amounts of memory. In this example, you have two variables, x, which is a 32-bit int, and y, which is a boolean. Storing a 32-bit int requires four bytes, so the value for x is stored in four bytes, starting at address 1 and ending at address 4. A boolean requires only a single byte (you need only a bit to represent true or false, but the smallest amount of

119

memory that can be independently addressed is a byte), so the value for y is stored in one byte at address 5, with true represented by the value 1. A pointer is a variable that contains the address where another variable is stored. Figure 6-2 demonstrates how the following pointers are stored in memory: var x int32 = 10 var y bool = true pointerX := &x pointerY := &y var pointerZ *string

Figure 6-2. Storing pointers in memory While different types of variables can take up different numbers of memory locations, every pointer, no matter what type it is pointing to, is always the same number of memory locations. The examples in this chapter use 4-byte pointers, but many modern computers use 8 bytes for a pointer. A pointer holds a number that indicates the location in memory where the data being pointed to is stored. That number is called the address. Our pointer to x, pointerX, is stored at location 6 and has the value 1, the address of x. Similarly, our pointer to y, pointerY, is stored at location 10 and has the value 5, the address of y. The last pointer, pointerZ, is stored at location 14 and has the value 0, because it doesn’t point to anything. The zero value for a pointer is nil. You’ve seen nil a few times before, as the zero value for slices, maps, and functions. All these types are implemented with pointers. (Two more types, channels and interfaces, are also implemented with pointers. You’ll look at them in detail in “A Quick Lesson on Interfaces” on page 157 and “Channels” on page 291.) As I covered in Chapter 3, nil is an untyped identifier that represents the lack of a value for certain types. Unlike NULL in C, nil is not another name for 0; you can’t convert it back and forth with a number. As alluded to in Chapter 4, nil is defined in the universe block. Because nil is a value defined in the universe block, it can be shadowed. Never name a variable or function nil, unless you are trying to trick your coworker and are unconcerned about your annual review.

120

|

Chapter 6: Pointers

Go’s pointer syntax is partially borrowed from C and C++. Since Go has a garbage collector, most memory management pain is removed. Furthermore, some tricks that you can do with pointers in C and C++, including pointer arithmetic, are not allowed in Go. The Go standard library does have an unsafe package that lets you do some low-level operations on data structures. While pointer manipulation is used in C for common operations, it is exceedingly rare for Go developers to use unsafe. You’ll take a quick look at it in Chapter 16.

The & is the address operator. It precedes a value type and returns the address where the value is stored: x := "hello" pointerToX := &x

The * is the indirection operator. It precedes a variable of pointer type and returns the pointed-to value. This is called dereferencing: x := 10 pointerToX := &x fmt.Println(pointerToX) // prints a memory address fmt.Println(*pointerToX) // prints 10 z := 5 + *pointerToX fmt.Println(z) // prints 15

Before dereferencing a pointer, you must make sure that the pointer is non-nil. Your program will panic if you attempt to dereference a nil pointer: var x *int fmt.Println(x == nil) // prints true fmt.Println(*x) // panics

A pointer type is a type that represents a pointer. It is written with a * before a type name. A pointer type can be based on any type: x := 10 var pointerToX *int pointerToX = &x

The built-in function new creates a pointer variable. It returns a pointer to a zerovalue instance of the provided type: var x = new(int) fmt.Println(x == nil) // prints false fmt.Println(*x) // prints 0

A Quick Pointer Primer

|

121

The new function is rarely used. For structs, use an & before a struct literal to create a pointer instance. You can’t use an & before a primitive literal (numbers, booleans, and strings) or a constant because they don’t have memory addresses; they exist only at compile time. When you need a pointer to a primitive type, declare a variable and point to it: x := &Foo{} var y string z := &y

Not being able to take the address of a constant is sometimes inconvenient. If you have a struct with a field of a pointer to a primitive type, you can’t assign a literal directly to the field: type person struct { FirstName string MiddleName *string LastName string } p := person{ FirstName: "Pat", MiddleName: "Perry", // This line won't compile LastName: "Peterson", }

Compiling this code returns the error: cannot use "Perry" (type string) as type *string in field value

If you try to put an & before "Perry", you’ll get the error message: cannot take the address of "Perry"

There are two ways around this problem. The first is to do what was shown previ‐ ously, which is to introduce a variable to hold the constant value. The second way is to write a generic helper function that takes in a parameter of any type and returns a pointer to that type: func makePointer[T any](t T) *T { return &t }

With that function, you can now write: p := person{ FirstName: "Pat", MiddleName: makePointer("Perry"), // This works LastName: "Peterson", }

122

|

Chapter 6: Pointers

Why does this work? When you pass a constant to a function, the constant is copied to a parameter, which is a variable. Since it’s a variable, it has an address in memory. The function then returns the variable’s memory address. Writing generic functions is covered in Chapter 8. Use a helper function to turn a constant value into a pointer.

Don’t Fear the Pointers The first rule of pointers is to not be afraid of them. If you are used to Java, JavaScript, Python, or Ruby, you might find pointers intimidating. However, pointers are actually the familiar behavior for classes. It’s the nonpointer structs in Go that are unusual. In Java and JavaScript, there is a difference in the behavior between primitive types and classes (Python and Ruby don’t have primitive values but use immutable instan‐ ces to simulate them). When a primitive value is assigned to another variable or passed to a function or method, any changes made to the other variable aren’t reflected in the original, as shown in Example 6-1. Example 6-1. Assigning primitive variables doesn’t share memory in Java int x = 10; int y = x; y = 20; System.out.println(x); // prints 10

However, let’s take a look at what happens when an instance of a class is assigned to another variable or passed to a function or method (the code in Example 6-2 is written in Python, but you can find similar code for Java, JavaScript, and Ruby in the sample_code/language_pointer_examples directory in the Chapter 6 repository). Example 6-2. Passing a class instance into a function class Foo: def __init__(self, x): self.x = x

def outer(): f = Foo(10) inner1(f) print(f.x)

Don’t Fear the Pointers

|

123

inner2(f) print(f.x) g = None inner2(g) print(g is None)

def inner1(f): f.x = 20

def inner2(f): f = Foo(30)

outer()

Running this code prints the following output: 20 20 True

That’s because the following scenarios are true in Java, Python, JavaScript, and Ruby: • If you pass an instance of a class to a function and you change the value of a field, the change is reflected in the variable that was passed in. • If you reassign the parameter, the change is not reflected in the variable that was passed in. • If you pass nil/null/None for a parameter value, setting the parameter itself to a new value doesn’t modify the variable in the calling function. Some people explain this behavior by saying that class instances are passed by ref‐ erence in these languages. This is untrue. If they were being passed by reference, scenarios two and three would change the variable in the calling function. These languages are always pass-by-value, just as in Go. What you are seeing is that every instance of a class in these languages is imple‐ mented as a pointer. When a class instance is passed to a function or method, the value being copied is the pointer to the instance. Since outer and inner1 are referring to the same memory, changes made to fields in f in inner1 are reflected in the variable in outer. When inner2 reassigns f to a new class instance, this creates a separate instance and does not affect the variable in outer. When you use a pointer variable or parameter in Go, you see the exact same behav‐ iors. The difference between Go and these languages is that Go gives you the choice to use pointers or values for both primitives and structs. Most of the time, you should use a value. Values make it easier to understand how and when your data 124

|

Chapter 6: Pointers

is modified. A secondary benefit is that using values reduces the amount of work that the garbage collector has to do. I’ll talk about that in “Reducing the Garbage Collector’s Workload” on page 136.

Pointers Indicate Mutable Parameters As you’ve already seen, Go constants provide names for literal expressions that can be calculated at compile time. Go has no mechanism to declare that other kinds of values are immutable. Modern software engineering embraces immutability. MIT’s course on Software Construction sums up the reasons: “[I]mmutable types are safer from bugs, easier to understand, and more ready for change. Mutability makes it harder to understand what your program is doing, and much harder to enforce contracts.” The lack of immutable declarations in Go might seem problematic, but the ability to choose between value and pointer parameter types addresses the issue. As the Software Construction course materials go on to explain: “[U]sing mutable objects is just fine if you are using them entirely locally within a method, and with only one reference to the object.” Rather than declare that some variables and parameters are immutable, Go developers use pointers to indicate that a parameter is mutable. Since Go is a call-by-value language, the values passed to functions are copies. For nonpointer types like primitives, structs, and arrays, this means that the called func‐ tion cannot modify the original. Since the called function has a copy of the original data, the original data’s immutability is guaranteed. I’ll talk about passing maps and slices to functions in “The Differ‐ ence Between Maps and Slices” on page 131.

However, if a pointer is passed to a function, the function gets a copy of the pointer. This still points to the original data, which means that the original data can be modified by the called function. This has a couple of related implications. The first implication is that when you pass a nil pointer to a function, you cannot make the value non-nil. You can reassign the value only if there was a value already assigned to the pointer. While confusing at first, it makes sense. Since the memory location was passed to the function via call-by-value, you can’t change the memory address, any more than you could change the value of an int parameter. You can demonstrate this with the following program: func failedUpdate(g *int) { x := 10

Pointers Indicate Mutable Parameters

|

125

g = &x } func main() { var f *int // f is nil failedUpdate(f) fmt.Println(f) // prints nil }

The flow through this code is shown in Figure 6-3.

Figure 6-3. Failing to update a nil pointer

126

|

Chapter 6: Pointers

You start with a nil variable f in main. When you call failedUpdate, you copy the value of f, which is nil, into the parameter named g. This means that g is also set to nil. You then declare a new variable x within failedUpdate with the value 10. Next, you change g in failedUpdate to point to x. This does not change the f in main, and when you exit failedUpdate and return to main, f is still nil. The second implication of copying a pointer is that if you want the value assigned to a pointer parameter to still be there when you exit the function, you must dereference the pointer and set the value. If you change the pointer, you have changed the copy, not the original. Dereferencing puts the new value in the memory location pointed to by both the original and the copy. Here’s a short program that shows how this works: func failedUpdate(px *int) { x2 := 20 px = &x2 } func update(px *int) { *px = 20 } func main() { x := 10 failedUpdate(&x) fmt.Println(x) // prints 10 update(&x) fmt.Println(x) // prints 20 }

The flow through this code is shown in Figure 6-4. In this example, you start with x in main set to 10. When you call failedUpdate, you copy the address of x into the parameter px. Next, you declare x2 in failedUpdate, set to 20. You then point px in failedUpdate to the address of x2. When you return to main, the value of x is unchanged. When you call update, you copy the address of x into px again. However, this time you change the value of what px in update points to, the variable x in main. When you return to main, x has been changed.

Pointers Indicate Mutable Parameters

|

127

Figure 6-4. The wrong way and the right way to update a pointer

128

|

Chapter 6: Pointers

Pointers Are a Last Resort That said, you should be careful when using pointers in Go. As discussed earlier, they make it harder to understand data flow and can create extra work for the garbage collector. Rather than populating a struct by passing a pointer to it into a function, have the function instantiate and return the struct (see Examples 6-3 and 6-4). Example 6-3. Don’t do this func MakeFoo(f *Foo) error { f.Field1 = "val" f.Field2 = 20 return nil }

Example 6-4. Do this func MakeFoo() (Foo, error) { f := Foo{ Field1: "val", Field2: 20, } return f, nil }

The only time you should use pointer parameters to modify a variable is when the function expects an interface. You see this pattern when working with JSON (I’ll talk more about the JSON support in Go’s standard library in “encoding/json” on page 327): f := struct { Name string `json:"name"` Age int `json:"age"` }{} err := json.Unmarshal([]byte(`{"name": "Bob", "age": 30}`), &f)

The Unmarshal function populates a variable from a slice of bytes containing JSON. It is declared to take a slice of bytes and an any parameter. The value passed in for the any parameter must be a pointer. If it is not, an error is returned. Why do you pass a pointer into Unmarshal instead of having it return a value? There are two reasons. First, this function predates the addition of generics to Go, and without generics (which I’ll talk about in detail in Chapter 8), there is no way to know what type of value to create and return. The second reason is that passing in a pointer gives you control over memory alloca‐ tion. Iterating over data and converting it from JSON to a Go struct is a common design pattern, so Unmarshal is optimized for this case. If the Unmarshal function Pointers Are a Last Resort

|

129

returned a value and Unmarshal was called in a loop, one struct instance would be created on each loop iteration. This creates a lot more work for the garbage collector, which slows down your program. You’ll see another use of this pattern when you look at “Slices as Buffers” on page 135, and I’ll talk more about efficient memory usage in “Reducing the Garbage Collector’s Workload” on page 136. Because JSON integration is so common, this API is sometimes treated as a common case by new Go developers, instead of the exception that it should be. When returning values from a function, you should favor value types. Use a pointer type as a return type only if there is state within the data type that needs to be modified. When you look at I/O in “io and Friends” on page 319, you’ll see that with buffers for reading or writing data. In addition, some data types used with concurrency must always be passed as pointers. You’ll see those in Chapter 12.

Pointer Passing Performance If a struct is large enough, using a pointer to the struct as either an input parameter or a return value improves performance. The time to pass a pointer into a function is constant for all data sizes, roughly one nanosecond. This makes sense, as the size of a pointer is the same for all data types. Passing a value into a function takes longer as the data gets larger. It takes about 0.7 milliseconds once the value gets to be around 10 megabytes of data. The behavior for returning a pointer versus returning a value is more interesting. For data structures that are smaller than 10 megabytes, it is actually slower to return a pointer type than a value type. For example, a 100-byte data structure takes around 10 nanoseconds to be returned, but a pointer to that data structure takes about 30 nanoseconds. As your data structures get larger, the performance advantage flips. It takes nearly 1.5 milliseconds to return 10 megabytes of data, but a little less than half a millisecond to return a pointer to it. You should be aware that these are very short times. For the vast majority of cases, the difference between using a pointer and a value won’t affect your program’s performance. But if you are passing megabytes of data between functions, consider using a pointer even if the data is meant to be immutable. All these numbers are from an i7-8700 computer with 32 GB of RAM. Different CPUs can produce different crossover points. For example, on an Apple M1 CPU with 16 GB of RAM, it is faster to return a pointer (5 microseconds) than a value (8 microseconds) at sizes of around 100 kilobytes. You can run your own perfor‐ mance tests with the code in the sample_code/pointer_perf directory in the Chapter 6 repository. Run the command go test ./… -bench=. to find your own results. (Benchmarks are covered in “Using Benchmarks” on page 393.)

130

| Chapter 6: Pointers

The Zero Value Versus No Value Go pointers are also commonly used to indicate the difference between a variable or field that’s been assigned the zero value and a variable or field that hasn’t been assigned a value at all. If this distinction matters in your program, use a nil pointer to represent an unassigned variable or struct field. Because pointers also indicate mutability, be careful when using this pattern. Rather than return a pointer set to nil from a function, use the comma ok idiom that you saw for maps, and return a value type and a boolean. Remember, if a nil pointer is passed into a function via a parameter or a field on a parameter, you cannot set the value within the function, as there’s nowhere to store the value. If a non-nil value is passed in for the pointer, do not modify it unless you document the behavior. Again, JSON conversions are the exception that proves the rule. When converting data back and forth from JSON (yes, I’ll talk more about the JSON support in Go’s standard library in “encoding/json” on page 327), you often need a way to differenti‐ ate between the zero value and not having a value assigned at all. Use a pointer value for fields in the struct that are nullable. When not working with JSON (or other external protocols), resist the temptation to use a pointer field to indicate no value. While a pointer does provide a handy way to indicate no value, if you are not going to modify the value, you should use a value type instead, paired with a boolean.

The Difference Between Maps and Slices As you saw in the previous chapter, any modifications made to a map that’s passed to a function are reflected in the original variable that was passed in. Now that you know about pointers, you can understand why: within the Go runtime, a map is implemented as a pointer to a struct. Passing a map to a function means that you are copying a pointer. Because of this, you should consider carefully before using maps for input parameters or return values, especially on public APIs. On an API-design level, maps are a bad choice because they say nothing about the values contained within; nothing explicitly defines any keys in the map, so the only way to know what they are is to trace through the code. From the standpoint of immutability, maps are bad because the only way to know what ended up in the map is to trace through all the functions that interact with it. This prevents your API from being self-documenting. If you are used to dynamic languages, don’t use a map as a replacement for another language’s lack of structure. Go is a strongly typed language; rather than passing a map around, use a struct.

The Difference Between Maps and Slices

|

131

(You’ll learn another reason to prefer structs when I talk about memory layout in “Reducing the Garbage Collector’s Workload” on page 136.) A map input parameter or return value is the correct choice in cer‐ tain situations. A struct requires you to name its fields at compile time. If the keys for your data aren’t known at compile time, a map is ideal.

Meanwhile, passing a slice to a function has more complicated behavior: any modifi‐ cation to the slice’s contents is reflected in the original variable, but using append to change the length isn’t reflected in the original variable, even if the slice has a capacity greater than its length. That’s because a slice is implemented as a struct with three fields: an int field for length, an int field for capacity, and a pointer to a block of memory. Figure 6-5 demonstrates the relationship.

Figure 6-5. The memory layout of a slice When a slice is copied to a different variable or passed to a function, a copy is made of the length, capacity, and the pointer. Figure 6-6 shows how both slice variables point to the same memory.

Figure 6-6. The memory layout of a slice and its copy 132

|

Chapter 6: Pointers

Changing the values in the slice changes the memory that the pointer points to, so the changes are seen in both the copy and the original. You see in Figure 6-7 how this looks in memory.

Figure 6-7. Modifying the contents of a slice If the slice copy is appended to and there is enough capacity in the slice for the new values, the length changes in the copy, and the new values are stored in the block of memory that’s shared by the copy and the original. However, the length in the original slice remains unchanged. The Go runtime prevents the original slice from seeing those values since they are beyond the length of the original slice. Figure 6-8 highlights the values that are visible in one slice variable but not in the other.

Figure 6-8. Changing the length is invisible in the original

The Difference Between Maps and Slices

|

133

If the slice copy is appended to and there isn’t enough capacity in the slice for the new values, a new, bigger block of memory is allocated, values are copied over, and the pointer, length, and capacity fields in the copy are updated. Changes to the pointer, length, and capacity are not reflected in the original, because they are only in the copy. Figure 6-9 shows how each slice variable now points to a different memory block.

Figure 6-9. Changing the capacity changes the storage The result is that a slice that’s passed to a function can have its contents modified, but the slice can’t be resized. As the only usable linear data structure, slices are frequently passed around in Go programs. By default, you should assume that a slice is not modified by a function. Your function’s documentation should specify whether it modifies the slice’s contents. The reason you can pass a slice of any size to a function is that the data type that’s passed to the function is the same for any size slice: a struct of two int values and a pointer. The reason you can’t write a function that takes an array of any size is that the entire array is passed to the function, not just a pointer to the data.

There is one situation where the ability to modify the contents (but not the size) of a slice input parameter is very useful. This makes them ideal for reusable buffers.

134

|

Chapter 6: Pointers

Slices as Buffers When reading data from an external resource (like a file or a network connection), many languages use code like this: r = open_resource() while r.has_data() { data_chunk = r.next_chunk() process(data_chunk) } close(r)

The problem with this pattern is that every time you iterate through that while loop, you allocate another data_chunk even though each one is used only once. This creates lots of unnecessary memory allocations, just as I discussed in “Pointers Are a Last Resort” on page 129 when looking at the Unmarshal function. Garbage-collected languages handle those allocations for you automatically, but the work still needs to be done to clean them up when you are done processing. Even though Go is a garbage-collected language, writing idiomatic Go means avoid‐ ing unneeded allocations. Rather than returning a new allocation each time you read from a data source, you create a slice of bytes once and use it as a buffer to read data from the data source: file, err := os.Open(fileName) if err != nil { return err } defer file.Close() data := make([]byte, 100) for { count, err := file.Read(data) process(data[:count]) if err != nil { if errors.Is(err, io.EOF) { return nil } return err } }

Remember that you can’t change the length or capacity of a slice when it is passed into a function, but you can change the contents up to the current length. In this code, you create a buffer of 100 bytes, and each time through the loop, you copy the next block of bytes (up to 100) into the slice. You then pass the populated portion of the buffer to process. If an error happens (other than io.EOF, a special-case error that indicates there is no more data to read), it is returned. When io.EOF is returned as the error, there is no more data and the function returns nil. You’ll look at more

Slices as Buffers

|

135

details about I/O in “io and Friends” on page 319 and error handling is covered in Chapter 9.

Reducing the Garbage Collector’s Workload Using buffers is just one example of how you reduce the work done by the garbage collector. When programmers talk about “garbage,” what they mean is “data that has no more pointers pointing to it.” Once there are no more pointers pointing to some data, the memory that this data takes up can be reused. If the memory isn’t recovered, the program’s memory usage would continue to grow until the computer ran out of RAM. The job of a garbage collector is to automatically detect unused memory and recover it so it can be reused. It is fantastic that Go has a garbage collector, because decades of experience have shown that it is very difficult for people to properly manage memory manually. But just because you have a garbage collector doesn’t mean you should create lots of garbage. If you’ve spent time learning how programming languages are implemented, you’ve probably learned about the heap and the stack. If you’re unfamiliar, here’s how a stack works. A stack is a consecutive block of memory. Every function call in a thread of execution shares the same stack. Allocating memory on the stack is fast and simple. A stack pointer tracks the last location where memory was allocated. Allocating additional memory is done by changing the value of the stack pointer. When a function is invoked, a new stack frame is created for the function’s data. Local variables are stored on the stack, along with parameters passed into a function. Each new variable moves the stack pointer by the size of the value. When a function exits, its return values are copied back to the calling function via the stack, and the stack pointer is moved back to the beginning of the stack frame for the exited function, deallocating all the stack memory that was used by that function’s local variables and parameters. Since version 1.17, Go uses a combination of registers (a small set of very high-speed memory that’s directly on the CPU) and the stack to pass values into and out of functions. It’s faster and more complicated, but the general concepts of stack-only function calls still apply.

To store something on the stack, you have to know exactly how big it is at compile time. When you look at the value types in Go (primitive values, arrays, and structs), they all have one thing in common: you know exactly how much memory they take at compile time. This is why the size is considered part of the type for an array. Because their sizes are known, they can be allocated on the stack instead of the heap. The size of a pointer type is also known, and it is also stored on the stack.

136

|

Chapter 6: Pointers

Go is unusual in that it can increase the size of a stack while the program is running. This is possible because each goroutine has its own stack and goroutines are managed by the Go runtime, not by the underlying operating system (I discuss goroutines when I talk about concurrency in Chapter 12). This has advantages (Go stacks start small and use less memory) and disadvantages (when the stack needs to grow, all data on the stack needs to be copied, which is slow). It’s also possible to write worst-case-scenario code that causes the stack to grow and shrink over and over.

The rules are more complicated when it comes to the data that the pointer points to. In order for Go to allocate the data the pointer points to on the stack, several conditions must be true. The data must be a local variable whose data size is known at compile time. The pointer cannot be returned from the function. If the pointer is passed into a function, the compiler must be able to ensure that these conditions still hold. If the size isn’t known, you can’t make space for it by moving the stack pointer. If the pointer variable is returned, the memory that the pointer points to will no longer be valid when the function exits. When the compiler determines that the data can’t be stored on the stack, we say that the data the pointer points to escapes the stack, and the compiler stores the data on the heap. The heap is the memory that’s managed by the garbage collector (or by hand in languages like C and C++). I won’t discuss the details of garbage collector algorithm implementation, but they are much more complicated than moving a stack pointer. Any data that’s stored on the heap is valid as long as it can be tracked back to a pointer type variable on a stack. Once there are no more variables on the stack pointing to that data, either directly or via a chain of pointers, the data becomes garbage, and it’s the job of the garbage collector to clear it out. This program on The Go Playground demonstrates when data on the heap becomes garbage. A common source of bugs in C programs is returning a pointer to a local variable. In C, this results in a pointer pointing to invalid memory. The Go compiler is smarter. When it sees that a pointer to a local variable is returned, the local variable’s value is stored on the heap.

The escape analysis done by the Go compiler isn’t perfect. In some cases, data that could be stored on the stack escapes to the heap. However, the compiler has to be conservative; it can’t take the chance of leaving a value on the stack when it might need to be on the heap because leaving a reference to invalid data causes memory corruption. Newer Go releases improve escape analysis. You might be wondering: what’s so bad about storing things on the heap? Two problems arise related to performance. First, the garbage collector takes time to do Reducing the Garbage Collector’s Workload

|

137

its work. It isn’t trivial to keep track of all available chunks of free memory on the heap or to track which used blocks of memory still have valid pointers. This is time that’s taken away from doing the processing that your program is written to do. Many garbage-collection algorithms have been written, and they can be placed into two rough categories: those that are designed for higher throughput (find the most garbage possible in a single scan) or lower latency (finish the garbage scan as quickly as possible). Jeffrey Dean, the genius behind many of Google’s engineering successes, cowrote a paper in 2013 called “The Tail at Scale”. It argues that systems should be optimized for latency, to keep response times low. The garbage collector used by the Go runtime favors low latency. Each garbage-collection cycle is designed to “stop the world” (i.e., pause your program) for fewer than 500 microseconds. However, if your Go program creates lots of garbage, the garbage collector won’t be able to find all the garbage during a cycle, slowing the collector and increasing memory usage. If you are interested in the implementation details, you may want to listen to the talk Rick Hudson gave at the International Sympo‐ sium on Memory Management in 2018, describing the history and implementation of the Go garbage collector.

The second problem deals with the nature of computer hardware. RAM might mean “random access memory,” but the fastest way to read from memory is to read it sequentially. A slice of structs in Go has all the data laid out sequentially in memory. This makes it fast to load and fast to process. A slice of pointers to structs (or structs whose fields are pointers) has its data scattered across RAM, making it far slower to read and process. Forrest Smith wrote an in-depth blog post that explores how much this can affect performance. His numbers indicate that it’s roughly two orders of magnitude slower to access data via pointers stored randomly in RAM. This approach of writing software that’s aware of the hardware it’s running on is called mechanical sympathy. The term comes from the world of car racing, where the idea is that a driver who understands what the car is doing can best squeeze the last bits of performance out of it. In 2011, Martin Thompson began applying the term to software development. Following best practices in Go gives it to you automatically. Compare Go’s approach to Java’s. In Java, local variables and parameters are stored in the stack, just as in Go. However, as discussed earlier, objects in Java are implemented as pointers. For every object variable instance, only the pointer to it is allocated on the stack; the data within the object is allocated on the heap. Only primitive values (numbers, booleans, and chars) are stored entirely on the stack. This means that the garbage collector in Java has to do a great deal of work. It also means that implementations of the List interface in Java are built using a pointer to an array of pointers. Even though they look like a linear data structure, reading from them actually involves bouncing through memory, which is highly inefficient. There are 138

|

Chapter 6: Pointers

similar behaviors for the sequential data types in Python, Ruby, and JavaScript. To work around all this inefficiency, the Java Virtual Machine includes some very clever garbage collectors that do lots of work, some optimized for throughput, some for latency, and all with configuration settings to tune them for the best performance. The virtual machines for Python, Ruby, and JavaScript are less optimized, and their performance suffers accordingly. Now you can see why Go encourages you to use pointers sparingly. You reduce the garbage collector’s workload by making sure that as much as possible is stored on the stack. Slices of structs or primitive types have their data lined up sequentially in memory for rapid access. And when the garbage collector does do work, it is opti‐ mized to return quickly rather than gather the most garbage. The key to making this approach work is to simply create less garbage in the first place. While focusing on optimizing memory allocations can feel like premature optimization, the idiomatic approach in Go is also the most efficient. If you want to learn more about heap versus stack allocation and escape analysis in Go, excellent blog posts cover the topic, including ones by Bill Kennedy of Ardan Labs and Achille Roussel and Rick Branson of Segment.

Tuning the Garbage Collector A garbage collector doesn’t immediately reclaim memory as soon as it is no longer referenced. Doing so would seriously impact performance. Instead, it lets the garbage pile up for a bit. The heap almost always contains both live data and memory that’s no longer needed. The Go runtime provides users a couple of settings to control the heap’s size. The first is the GOGC environment variable. The garbage collector looks at the heap size at the end of a garbage-collection cycle and uses the formula CURRENT_HEAP_SIZE + CURRENT_HEAP_SIZE*GOGC/100 to calculate the heap size that needs to be reached to trigger the next garbage-collection cycle. The GOGC heap size calculation is a little more complicated than just described. It takes into account not just the heap size, but the sizes of all the stacks of all the goroutines and the memory set aside to hold package-level variables. Most of the time, the heap size is far bigger than the size of these other memory areas, but in some situations they do have an effect.

By default, GOGC is set to 100, which means that the heap size that triggers the next collection is roughly double the heap size at the end of the current collection. Setting GOGC to a smaller value will decrease the target heap size, and setting it to a larger value will increase it. As a rough estimate, doubling the value of GOGC will halve the amount of CPU time spent on GC.

Tuning the Garbage Collector

|

139

Setting GOGC to off disables garbage collection. This will make your programs run faster. However, turning off garbage collection on a long-running process will poten‐ tially use all available memory on your computer. This is not usually considered optimal behavior. The second garbage-collection setting specifies a limit on the total amount of mem‐ ory your Go program is allowed to use. Java developers are likely familiar with the -Xmx JVM argument, and GOMEMLIMIT is similar. By default, it is disabled (technically, it is set to math.MaxInt64, but it’s unlikely that your computer has that much mem‐ ory). The value for GOMEMLIMIT is specified in bytes, but you can optionally use the suffixes B, KiB, MiB, GiB, and TiB. For example, GOMEMLIMIT=3GiB sets the memory limit to 3 gibibytes (which is equal to 3,221,225,472 bytes). If you haven’t seen these suffixes before, they are the official powerof-two analogues of the more commonly used power-of-ten KB, MB, GB, and TB. KiB is equal to 210, MiB is equal to 220, and so on. It is technically correct to use KiB, MiB, and friends when dealing with computers.

It might seem counterintuitive that limiting the maximum amount of memory could improve a program’s performance, but there are good reasons this flag was added. The primary reason is that computers (or virtual machines or containers) don’t have infinite RAM. If a sudden, temporary spike occurs in memory usage, relying on GOGC alone might result in the maximum heap size exceeding the amount of available memory. This can cause memory to swap to disk, which is very slow. Depending on your operating system and its settings, it might crash your program. Specifying a maximum memory limit prevents the heap from growing beyond the computer’s resources. GOMEMLIMIT is a soft limit that can be exceeded in certain circumstances. A common problem occurs in garbage-collected systems when the collector is unable to free up enough memory to get within a memory limit or the garbage-collection cycles are rapidly being triggered because a program is repeatedly hitting the limit. Called thrashing, this results in a program that does nothing other than run the garbage collector. If the Go runtime detects that thrashing is starting to happen, it chooses to end the current garbage-collection cycle and exceed the limit. This does mean that you should set GOMEMLIMIT below the absolute maximum amount of available memory so you have spare capacity.

If you specify a value for GOMEMLIMIT, you could set GOGC to off and not run out of memory, but this may not produce the desired performance effect. You are likely to find yourself trading frequent, very short pauses for infrequent, longer pauses. If you

140

|

Chapter 6: Pointers

are running a web service, this produces inconsistent response times, which was one of the behaviors that Go’s garbage collection was designed to avoid. The best option is to use these two environment variables together to ensure both a reasonable pace for garbage collection and a maximum that should be respected. You can learn more about how to use GOGC and GOMEMLIMIT by reading “A Guide to the Go Garbage Collector” from Go’s development team.

Exercises Now that you have learned about pointers and memory in Go, work through these exercises to reinforce using pointers effectively. You can find answers to these exerci‐ ses in the Chapter 6 repository. 1. Create a struct named Person with three fields: FirstName and LastName of type string and Age of type int. Write a function called MakePerson that takes in firstName, lastName, and age and returns a Person. Write a second function MakePersonPointer that takes in firstName, lastName, and age and returns a *Person. Call both from main. Compile your program with go build -gcflags="-m". This both compiles your code and prints out which values escape to the heap. Are you surprised about what escapes? 2. Write two functions. The UpdateSlice function takes in a []string and a string. It sets the last position in the passed-in slice to the passed-in string. At the end of UpdateSlice, print the slice after making the change. The GrowSlice function also takes in a []string and a string. It appends the string onto the slice. At the end of GrowSlice, print the slice after making the change. Call these functions from main. Print out the slice before each function is called and after each function is called. Do you understand why some changes are visible in main and why some changes are not? 3. Write a program that builds a []Person with 10,000,000 entries (they could all be the same names and ages). See how long it takes to run. Change the value of GOGC and see how that affects the time it takes for the program to complete. Set the environment variable GODEBUG=gctrace=1 to see when garbage collections happen and see how changing GOGC changes the number of garbage collections. What happens if you create the slice with a capacity of 10,000,000?

Wrapping Up This chapter peeked under the covers a bit to help you understand pointers, what they are, how to use them, and, most importantly, when to use them. In the next chapter, you’ll take a look at Go’s implementation of methods, interfaces, and types, how they differ from other languages, and the power they possess. Wrapping Up

|

141

CHAPTER 7

Types, Methods, and Interfaces

As you saw in earlier chapters, Go is a statically typed language with both built-in types and user-defined types. Like most modern languages, Go allows you to attach methods to types. It also has type abstraction, allowing you to write code that invokes methods without explicitly specifying the implementation. However, Go’s approach to methods, interfaces, and types is very different from that of most other languages in common use today. Go is designed to encourage the best practices advocated by software engineers, avoiding inheritance while encouraging composition. In this chapter, you’ll take a look at types, methods, and interfaces, and see how to use them to build testable and maintainable programs.

Types in Go Back in “Structs” on page 61, you saw how to define a struct type: type Person struct { FirstName string LastName string Age int }

This should be read as declaring a user-defined type with the name Person to have the underlying type of the struct literal that follows. In addition to struct literals, you can use any primitive type or compound type literal to define a concrete type. Here are a few examples: type Score int type Converter func(string)Score type TeamScores map[string]Score

143

Go allows you to declare a type at any block level, from the package block down. However, you can access the type only from within its scope. The only exceptions are types exported from other packages. I’ll talk more about those in Chapter 10. To make it easier to talk about types, I’ll define a couple of terms. An abstract type is one that specifies what a type should do but not how it is done. A concrete type specifies what and how. This means that tthe type has a specified way to store its data and provides an implementation of any methods declared on the type. While all types in Go are either abstract or concrete, some languages allow hybrid types, such as abstract classes or interfaces with default methods in Java.

Methods Like most modern languages, Go supports methods on user-defined types. The methods for a type are defined at the package block level: type Person struct { FirstName string LastName string Age int } func (p Person) String() string { return fmt.Sprintf("%s %s, age %d", p.FirstName, p.LastName, p.Age) }

Method declarations look like function declarations, with one addition: the receiver specification. The receiver appears between the keyword func and the name of the method. Like all other variable declarations, the receiver name appears before the type. By convention, the receiver name is a short abbreviation of the type’s name, usually its first letter. It is nonidiomatic to use this or self. There is one key difference between declaring methods and functions: methods can be defined only at the package block level, while functions can be defined inside any block. Just like functions, method names cannot be overloaded. You can use the same method names for different types, but you can’t use the same method name for two different methods on the same type. While this philosophy feels limiting when coming from languages that have method overloading, not reusing names is part of Go’s philosophy of making clear what your code is doing.

144

| Chapter 7: Types, Methods, and Interfaces

I’ll talk more about packages in Chapter 10, but be aware that methods must be declared in the same package as their associated type; Go doesn’t allow you to add methods to types you don’t control. While you can define a method in a different file within the same package as the type declaration, it is best to keep your type definition and its associated methods together so that it’s easy to follow the implementation. Method invocations should look familiar to those who have used methods in other languages: p := Person{ FirstName: "Fred", LastName: "Fredson", Age: 52, } output := p.String()

Pointer Receivers and Value Receivers As I covered in Chapter 6, Go uses parameters of pointer type to indicate that a parameter might be modified by the function. The same rules apply for method receivers too. They can be pointer receivers (the type is a pointer) or value receivers (the type is a value type). The following rules help you determine when to use each kind of receiver: • If your method modifies the receiver, you must use a pointer receiver. • If your method needs to handle nil instances (see “Code Your Methods for nil Instances” on page 148), then it must use a pointer receiver. • If your method doesn’t modify the receiver, you can use a value receiver. Whether you use a value receiver for a method that doesn’t modify the receiver depends on the other methods declared on the type. When a type has any pointer receiver methods, a common practice is to be consistent and use pointer receivers for all methods, even the ones that don’t modify the receiver. Here’s some simple code to demonstrate pointer and value receivers. It starts with a type that has two methods on it, one using a value receiver, the other with a pointer receiver: type Counter struct { total int lastUpdated time.Time } func (c *Counter) Increment() { c.total++ c.lastUpdated = time.Now() }

Methods

|

145

func (c Counter) String() string { return fmt.Sprintf("total: %d, last updated: %v", c.total, c.lastUpdated) }

You can then try out these methods with the following code. You can run it yourself on The Go Playground or use the code in the sample_code/pointer_value directory in the Chapter 7 repository: var c Counter fmt.Println(c.String()) c.Increment() fmt.Println(c.String())

You should see the following output: total: 0, last updated: 0001-01-01 00:00:00 +0000 UTC total: 1, last updated: 2009-11-10 23:00:00 +0000 UTC m=+0.000000001

One thing you might notice is that you were able to call the pointer receiver method even though c is a value type. When you use a pointer receiver with a local variable that’s a value type, Go automatically takes the address of the local variable when calling the method. In this case, c.Increment() is converted to (&c).Increment(). If you call a value receiver on a pointer variable, Go automatically dereferences the pointer when calling the method. In the code: c := &Counter{} fmt.Println(c.String()) c.Increment() fmt.Println(c.String())

the call c.String() is silently converted to (*c).String(). If you call a value receiver method with pointer instance whose value is nil, your code will compile but will panic at runtime (I discuss panics in “panic and recover” on page 218).

Be aware that the rules for passing values to functions still apply. If you pass a value type to a function and call a pointer receiver method on the passed value, you are invoking the method on a copy. You can try out the following code on The Go Playground or use the code in the sample_code/update_wrong directory in the Chapter 7 repository: func doUpdateWrong(c Counter) { c.Increment() fmt.Println("in doUpdateWrong:", c.String()) } func doUpdateRight(c *Counter) {

146

|

Chapter 7: Types, Methods, and Interfaces

c.Increment() fmt.Println("in doUpdateRight:", c.String()) } func main() { var c Counter doUpdateWrong(c) fmt.Println("in main:", c.String()) doUpdateRight(&c) fmt.Println("in main:", c.String()) }

When you run this code, you’ll get the following output: in doUpdateWrong: total: 1, last m=+0.000000001 in main: total: 0, last updated: in doUpdateRight: total: 1, last m=+0.000000001 in main: total: 1, last updated:

updated: 2009-11-10 23:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC updated: 2009-11-10 23:00:00 +0000 UTC 2009-11-10 23:00:00 +0000 UTC m=+0.000000001

The parameter in doUpdateRight is of type *Counter, which is a pointer instance. As you can see, you can call both Increment and String on it. Go considers both pointer and value receiver methods to be in the method set for a pointer instance. For a value instance, only the value receiver methods are in the method set. This seems like a pedantic detail right now, but I’ll come back to it when talking about interfaces in just a bit. This can be confusing to new Go programmers (and, to be honest, to not-so-new Go programmers), but Go’s automatic conversion from pointer types to value types, and vice versa, is purely syntactic sugar. It is independent of the method set concept. Alexey Gron‐ skiy has written a blog post that explores in detail why the method set of pointer instances have both pointer and value receiver meth‐ ods, but the method set of value instances has only value receiver methods.

One final note: do not write getter and setter methods for Go structs unless you need them to meet an interface (I’ll start covering interfaces in “A Quick Lesson on Interfaces” on page 157). Go encourages you to directly access a field. Reserve methods for business logic. The exceptions are when you need to update multiple fields as a single operation or when the update isn’t a straightforward assignment of a new value. The Increment method defined earlier demonstrates both of these properties.

Methods

|

147

Code Your Methods for nil Instances The previous section covered pointer receivers, which might make you wonder what happens when you call a method on a nil instance. In most languages, this produces some sort of error. (Objective-C allows you to call a method on a nil instance, but it always does nothing.) Go does something a little different. It actually tries to invoke the method. As men‐ tioned earlier, if it’s a method with a value receiver, you’ll get a panic, since there is no value being pointed to by the pointer. If it’s a method with a pointer receiver, it can work if the method is written to handle the possibility of a nil instance. In some cases, expecting a nil receiver makes the code simpler. Here’s an implemen‐ tation of a binary tree that takes advantage of nil values for the receiver: type IntTree struct { val int left, right *IntTree } func (it *IntTree) Insert(val int) *IntTree { if it == nil { return &IntTree{val: val} } if val < it.val { it.left = it.left.Insert(val) } else if val > it.val { it.right = it.right.Insert(val) } return it } func (it *IntTree) Contains(val int) bool { switch { case it == nil: return false case val < it.val: return it.left.Contains(val) case val > it.val: return it.right.Contains(val) default: return true } }

The Contains method doesn’t modify the *IntTree, but it is declared with a pointer receiver. This demonstrates the rule men‐ tioned previously about supporting a nil receiver. A method with a value receiver can’t check for nil and, as mentioned earlier, panics if invoked with a nil receiver. 148

| Chapter 7: Types, Methods, and Interfaces

The following code uses the tree. You can try it out on The Go Playground or use the code in the sample_code/tree directory in the Chapter 7 repository: func main() { var it *IntTree it = it.Insert(5) it = it.Insert(3) it = it.Insert(10) it = it.Insert(2) fmt.Println(it.Contains(2)) // true fmt.Println(it.Contains(12)) // false }

It’s very clever that Go allows you to call a method on a nil receiver, and there are situations where it is useful, like the previous tree node example. However, most of the time it’s not very useful. Pointer receivers work like pointer function parameters; it’s a copy of the pointer that’s passed into the method. Just like nil parameters passed to functions, if you change the copy of the pointer, you haven’t changed the original. This means you can’t write a pointer receiver method that handles nil and makes the original pointer non-nil. If your method has a pointer receiver and won’t work for a nil receiver, you have to decide how your method should handle a nil receiver. One choice is to treat it as a fatal flaw, like trying to access a position in a slice beyond its length. In that case, don’t do anything and let the code panic. (Also make sure you write good tests, as discussed in Chapter 15.) If a nil receiver is something that is recoverable, check for nil and return an error (I discuss errors in Chapter 9).

Methods Are Functions Too Methods in Go are so much like functions that you can use a method as a replace‐ ment for a function anytime there’s a variable or parameter of a function type. Let’s start with this simple type: type Adder struct { start int } func (a Adder) AddTo(val int) int { return a.start + val }

You create an instance of the type in the usual way and invoke its method: myAdder := Adder{start: 10} fmt.Println(myAdder.AddTo(5)) // prints 15

Methods

|

149

You can also assign the method to a variable or pass it to a parameter of type func(int)int. This is called a method value: f1 := myAdder.AddTo fmt.Println(f1(10))

// prints 20

A method value is a bit like a closure, since it can access the values in the fields of the instance from which it was created. You can also create a function from the type itself. This is called a method expression: f2 := Adder.AddTo fmt.Println(f2(myAdder, 15))

// prints 25

In a method expression, the first parameter is the receiver for the method; the function signature is func(Adder, int) int. Method values and method expressions aren’t clever corner cases. You’ll see one way to use them when you look at dependency injection in “Implicit Interfaces Make Dependency Injection Easier” on page 174.

Functions Versus Methods Since you can use a method as a function, you might wonder when you should declare a function and when you should use a method. The differentiator is whether your function depends on other data. As I’ve covered several times, package-level state should be effectively immutable. Anytime your logic depends on values that are configured at startup or changed while your program is running, those values should be stored in a struct, and that logic should be imple‐ mented as a method. If your logic depends only on the input parameters, it should be a function.

Type Declarations Aren’t Inheritance In addition to declaring types based on built-in Go types and struct literals, you can also declare a user-defined type based on another user-defined type: type HighScore Score type Employee Person

Many concepts can be considered “object-oriented,” but one stands out: inheritance. With inheritance, the state and methods of a parent type are declared to be available on a child type, and values of the child type can be substituted for the parent type.1

1 For the computer scientists in the audience, I realize that subtyping is not inheritance. However, most

programming languages use inheritance to implement subtyping, so the definitions are often conflated in popular usage.

150

|

Chapter 7: Types, Methods, and Interfaces

Declaring a type based on another type looks a bit like inheritance but isn’t. The two types have the same underlying type, but that’s all. The types have no hierarchy. In languages with inheritance, a child instance can be used anywhere the parent instance is used. The child instance also has all the methods and data structures of the parent instance. That’s not the case in Go. You can’t assign an instance of type HighScore to a variable of type Score, or vice versa, without a type conversion, nor can you assign either of them to a variable of type int without a type conversion. Furthermore, any methods defined on Score aren’t defined on HighScore: // assigning untyped constants is valid var i int = 300 var s Score = 100 var hs HighScore = 200 hs = s // compilation error! s = i // compilation error! s = Score(i) // ok hs = HighScore(s) // ok

User-defined types whose underlying types are built-in types can be assigned literals and constants compatible with the underlying type. They can also be used with the operators for those types: var s Score = 50 scoreWithBonus := s + 100 // type of scoreWithBonus is Score

A type conversion between types that share an underlying type keeps the same underlying storage but associates different methods.

Types Are Executable Documentation While it’s well understood that you should declare a struct type to hold a set of related data, it’s less clear when you should declare a user-defined type based on other built-in types or one user-defined type that’s based on another user-defined type. The short answer is that types are documentation. They make code clearer by providing a name for a concept and describing the kind of data that is expected. It’s clearer for someone reading your code when a method has a parameter of type Percentage than of type int, and it’s harder for it to be invoked with an invalid value. The same logic applies when declaring one user-defined type based on another user-defined type. When you have the same underlying data, but different sets of operations to perform, make two types. Declaring one as being based on the other avoids some repetition and makes it clear that the two types are related.

Methods

|

151

iota Is for Enumerations—Sometimes Many programming languages have the concept of enumerations, which allow you to specify that a type can have only a limited set of values. Go doesn’t have an enumeration type. Instead, it has iota, which lets you assign an increasing value to a set of constants. The concept of iota comes from the programming language APL (which stood for “A Programming Language”). To generate a list of the first three positive integers in APL, you write ι3, where ι is the lowercase Greek letter iota. APL is famous for being so reliant on its own custom notation that it required computers with a special keyboard. For example, (~R∊R∘.×R)/R←1↓ιR is an APL program to find all the prime num‐ bers up to the value of the variable R. It may seem ironic that a language as focused on readability as Go would borrow a concept from a language that is concise to a fault, but this is why you should learn multiple programming languages: you can find inspiration everywhere.

When using iota, the best practice is to first define a type based on int that will represent all the valid values: type MailCategory int

Next, use a const block to define a set of values for your type: const ( Uncategorized MailCategory = iota Personal Spam Social Advertisements )

The first constant in the const block has the type specified, and its value is set to iota. Every subsequent line has neither the type nor a value assigned to it. When the Go compiler sees this, it repeats the type and the assignment to all the subsequent constants in the block, which is iota. The value of iota increments for each constant defined in the const block, starting with 0. This means that 0 is assigned to the first constant (Uncategorized), 1 to the second constant (Personal), and so on. When a new const block is created, iota is set back to 0.

152

|

Chapter 7: Types, Methods, and Interfaces

The value of iota increments for each constant in the const block, whether or not iota is used to define the value of a constant. The following code demonstrates what happens when iota is used intermittently in a const block: const ( Field1 Field2 Field3 Field4 Field5 )

= 0 = 1 + iota = 20 = iota

func main() { fmt.Println(Field1, Field2, Field3, Field4, Field5) }

You can run this code on The Go Playground and see the (perhaps unexpected) result: 0 2 20 20 4

Field2 is assigned 2 because iota has a value of 1 on the second line in the const block. Field4 is assigned 20 because it has no type or value explicitly assigned, so it gets the value of the previous line with a type and assignment. Finally, Field5 gets the value 4 because it is the fifth line and iota starts counting from 0.

This is the best advice I’ve seen on iota: Don’t use iota for defining constants where its values are explicitly defined (elsewhere). For example, when implementing parts of a specification and the specification says which values are assigned to which constants, you should explicitly write the constant values. Use iota for “internal” purposes only. That is, where the constants are referred to by name rather than by value. That way, you can optimally enjoy iota by inserting new constants at any moment in time / location in the list without the risk of breaking everything. —Danny van Heumen

The important thing to understand is that nothing in Go will stop you (or anyone else) from creating additional values of your type. Furthermore, if you insert a new identifier in the middle of your list of literals, all subsequent ones will be renumbered. This will break your application in a subtle way if those constants represented values in another system or in a database. Given these two limitations, iota-based enumera‐ tions make sense only when you care about being able to differentiate between a set of values and don’t particularly care what the value is behind the scenes. If the actual value matters, specify it explicitly.

iota Is for Enumerations—Sometimes

|

153

Because you can assign a literal expression to a constant, you’ll see sample code that suggests you should use iota for cases like this: type BitField int const ( Field1 BitField = 1 v1.0.0

The go get command lets you work with modules, updating the versions of your dependencies. Now if you look at go.mod, you’ll see that the version has been changed: module github.com/learning-go-book-2e/region_tax go 1.20 require ( github.com/learning-go-book-2e/simpletax v1.0.0 github.com/shopspring/decimal v1.3.1 )

You also see in go.sum that it contains both versions of simpletax: github.com/learning-go-book-2e/simpletax github.com/learning-go-book-2e/simpletax github.com/learning-go-book-2e/simpletax github.com/learning-go-book-2e/simpletax

v1.0.0 h1:KZU8aXRCHkvgFmBWkV... v1.0.0/go.mod h1:lR4YYZwbDTI... v1.1.0 h1:sG83gscauX/b8yKKY9... v1.1.0/go.mod h1:lR4YYZwbDTI...

This is fine; if you change a module’s version or even remove a module from your module, an entry for it might still remain in go.sum. This doesn’t cause problems. When you build and run the code again, the bug is fixed: $ go build $ ./region_tax 99.99 12345 107.99

Semantic Versioning Software has had version numbers from time immemorial, but there has been little consistency in what version numbers mean. The version numbers attached to Go modules follow the rules of semantic versioning, also known as SemVer. By requiring

246

| Chapter 10: Modules, Packages, and Imports

semantic versioning for modules, Go makes its module management code simpler while ensuring that users of a module understand what a new release promises. If you aren’t familiar with SemVer, check out the full specification. The very short explanation is that semantic versioning divides a version number into three parts: the major version, the minor version, and the patch version, which are written as major.minor.patch and preceded by a v. The patch version number is incremented when fixing a bug; the minor version number is incremented (and the patch version is set back to 0) when a new, backward-compatible feature is added; and the major version number is incremented (and minor and patch are set back to 0) when making a change that breaks backward compatibility.

Minimal Version Selection At some point, your module will depend on two or more modules that all depend on the same module. As often happens, these modules declare that they depend on different minor or patch versions of that module. How does Go resolve this? The module system uses the principle of minimal version selection: you will always get the lowest version of a dependency that is declared to work in all the go.mod files across all your dependencies. Let’s say that your module directly depends on modules A, B, and C. All three of these modules depend on module D. The go.mod file for module A declares that it depends on v1.1.0, module B declares that it depends on v1.2.0, and module C declares that it depends on v1.2.3. Go will import module D only once, and it will choose version v1.2.3, as that, in the words of the Go Modules Reference, is the minimum version that satisfies all requirements. You can see this in action with your sample program from “Importing Third-Party Code” on page 240. The command go mod graph shows the dependency graph of your module and all its dependencies. Here are a few lines of its output: github.com/learning-go-book-2e/money github.com/fatih/[email protected] github.com/learning-go-book-2e/money github.com/mattn/[email protected] github.com/learning-go-book-2e/money github.com/mattn/[email protected] github.com/fatih/[email protected] github.com/mattn/[email protected] github.com/fatih/[email protected] github.com/mattn/[email protected] github.com/mattn/[email protected] github.com/mattn/[email protected]

Each line lists two modules, the first being the parent and the second being the dependency and its version. You’ll notice the github.com/fatih/color mod‐ ule is declared to depend on version v0.0.14 of github.com/mattn/go-isatty, while github.com/mattn/go-colorable depends on v0.0.12. The Go compiler selects version v0.0.14 to use, because it is the minimal version that meets all requirements. This occurs even though, as of this writing, the latest version of github.com/mattn/go-isatty is v0.0.16. Your minimal version requirement is met with v0.0.14, so that’s what is used. Working with Modules

|

247

This system isn’t perfect. You might find that while module A works with version v1.1.0 of module D, it does not work with version v1.2.3. What do you do then? Go’s answer is that you need to contact the module authors to fix their incompatibilities. The import compatibility rule says, “If an old package and a new package have the same import path, the new package must be backward compatible with the old package.” All minor and patch versions of a module must be backward compatible. If they aren’t, it’s a bug. In our hypothetical example, either module D needs to be fixed because it broke backward compatibility, or module A needs to be fixed because it made a faulty assumption about the behavior of module D. You might not find this answer satisfying, but it’s honest. Some build systems, like npm, will include multiple versions of the same package. This can introduce its own set of bugs, especially when there is package-level state. It also increases the size of your application. In the end, some things are better solved by community than code.

Updating to Compatible Versions What if you explicitly want to upgrade a dependency? Let’s assume that after writing the initial program, there are three more versions of simpletax. The first fixes problems in the initial v1.1.0 release. Since it’s a bug patch release with no new func‐ tionality, it would be released as v1.1.1. The second keeps the current functionality but also adds a new function. It would get the version number v1.2.0. Finally, the third fixes a bug that was found in version v1.2.0. It has the version number v1.2.1. To upgrade to the bug patch release for the current minor version, use the command go get -u=patch github.com/learning-go-book-2e/simpletax. Since you had downgraded to v1.0.0, you would remain on that version, because there is no patch version with the same minor version. Upgrade to version v1.1.0 by using go get github.com/learning-go-book-2e/sim [email protected] and then run go get -u=patch github.com/learning-go-book-2e/ simpletax. This upgrades the version to v1.1.1. Finally, use the command go get -u github.com/learning-go-book-2e/simpletax to get the most recent version of simpletax. That upgrades you to version v1.2.1.

Updating to Incompatible Versions Let’s go back to the program. You’re expanding to Canada, and luckily, a version of the simpletax module handles both the US and Canada. However, this version has a slightly different API than the previous one, so its version is v2.0.0.

248

| Chapter 10: Modules, Packages, and Imports

To handle incompatibility, Go modules follow the semantic import versioning rule. This rule has two parts: • The major version of the module must be incremented. • For all major versions besides 0 and 1, the path to the module must end in vN, where N is the major version. The path changes because an import path uniquely identifies a package. By definition, incompatible versions of a package are not the same package. Using different paths means that you can import two incompatible versions of a package into different parts of your program, allowing you to upgrade gracefully. Let’s see how this changes the program. First, change the import of simpletax to this: "github.com/learning-go-book-2e/simpletax/v2"

This changes your import to refer to the v2 module. Next, change the code in main to the following: func main() { amount, err := decimal.NewFromString(os.Args[1]) if err != nil { log.Fatal(err) } zip := os.Args[2] country := os.Args[3] percent, err := simpletax.ForCountryPostalCode(country, zip) if err != nil { log.Fatal(err) } total := amount.Add(amount.Mul(percent)).Round(2) fmt.Println(total) }

The program is now reading a third parameter from the command line, which is the country code. The program also calls a different function in the simpletax package. When you run go get ./..., the dependency is automatically updated: $ go get ./... go: downloading github.com/learning-go-book-2e/simpletax/v2 v2.0.0 go: added github.com/learning-go-book-2e/simpletax/v2 v2.0.0

Build and run the program to see the new output: $ go build $ ./region_tax 99.99 M4B1B4 CA 112.99 $ ./region_tax 99.99 12345 US 107.99

Working with Modules

|

249

When you look at the go.mod file, you’ll see that the new version of simpletax is included: module github.com/learning-go-book-2e/region_tax go 1.20 require ( github.com/learning-go-book-2e/simpletax v1.0.0 github.com/learning-go-book-2e/simpletax/v2 v2.0.0 github.com/shopspring/decimal v1.3.1 )

And go.sum has been updated as well: github.com/learning-go-book-2e/simpletax v1.0.0 h1:KZU8aXRCHkvgFmBWkV... github.com/learning-go-book-2e/simpletax v1.0.0/go.mod h1:lR4YYZwbDTI... github.com/learning-go-book-2e/simpletax v1.1.0 h1:sG83gscauX/b8yKKY9... github.com/learning-go-book-2e/simpletax v1.1.0/go.mod h1:lR4YYZwbDTI... github.com/learning-go-book-2e/simpletax/v2 v2.0.0 h1:EUFWy1BBA2omgkm... github.com/learning-go-book-2e/simpletax/v2 v2.0.0/go.mod h1:yGLh6ngH... github.com/shopspring/decimal v1.3.1 h1:2Usl1nmF/WZucqkFZhnfFYxxxu8LG... github.com/shopspring/decimal v1.3.1/go.mod h1:DKyhrW/HYNuLGql+MJL6WC...

The old versions of simpletax are still referenced, even though they are no longer used. Use go mod tidy to remove those unused versions. Then you’ll see only v2.0.0 of simpletax referenced in go.mod and go.sum.

Vendoring To ensure that a module always builds with identical dependencies, some organiza‐ tions like to keep copies of their dependencies inside their module. This is known as vendoring. It’s enabled by running the command go mod vendor. This creates a directory called vendor at the top level of your module that contains all your module’s dependencies. These dependencies are used in place of the module cache stored on your computer. If new dependencies are added to go.mod or versions of existing dependencies are upgraded with go get, you need to run go mod vendor again to update the vendor directory. If you forget to do this, go build, go run, and go test will display an error message and refuse to run. Older Go dependency management systems required vendoring, but with the advent of Go modules and proxy servers (see “Module Proxy Servers” on page 259 for details), the practice is falling out of favor. One reason you might still want to vendor is that it can make building your code faster and more efficient when working with some CI/CD (continuous integration/continuous delivery) pipelines. If a pipeline’s build servers are ephemeral, the module cache may not be preserved. Vendoring dependencies allows these pipelines to avoid making multiple network 250

|

Chapter 10: Modules, Packages, and Imports

calls to download dependencies every time a build is triggered. The downside is that it dramatically increases the size of your codebase in version control.

Using pkg.go.dev While there isn’t a single centralized repository of Go modules, there is a single ser‐ vice that gathers together documentation on Go modules. The Go team has created a site called pkg.go.dev that automatically indexes open source Go modules. For each module, the package index publishes the godocs, the license used, the README, the module’s dependencies, and what open source modules depend on the module. You can see the info that pkg.go.dev has on your simpletax module in Figure 10-2.

Figure 10-2. Use pkg.go.dev to find and learn about third-party modules

Publishing Your Module Making your module available to other people is as simple as putting it in a version control system. This is true whether you are releasing your module as open source on a public version control system like GitHub or a private one that’s hosted by you or

Publishing Your Module

|

251

within a company. Since Go programs build from source code and use a repository path to identify themselves, there’s no need to explicitly upload your module to a central library repository, as you do for Maven Central or npm. Make sure you check in both your go.mod file and your go.sum file. While most Go developers use Git for version control, Go also sup‐ ports Subversion, Mercurial, Bazaar, and Fossil. By default, Git and Mercurial can be used for public repositories, and any of the sup‐ ported version control systems can be used for private repositories. For details, check out the version control system documentation for Go modules.

When releasing an open source module, you should include a file named LICENSE in the root of your repository that specifies the open source license under which you are releasing your code. It’s FOSS is a good resource for learning more about the various kinds of open source licenses. Roughly speaking, you can divide open source licenses into two categories: permis‐ sive (which allows users of your code to keep their code private) and nonpermissive (which requires users of your code to make their code open source). While the license you choose is up to you, the Go community favors permissive licenses, such as BSD, MIT, and Apache. Since Go compiles third-party code directly into every application, the use of a nonpermissive license like the GPL would require people who use your code to release their code as open source as well. For many organizations, this is not acceptable. One final note: do not write your own license. Few people will trust that it has been properly reviewed by a lawyer, and they can’t tell what claims you are making on their module.

Versioning Your Module Whether your module is public or private, you should properly version your module so that it works correctly with Go’s module system. As long as you are adding functionality or patching bugs, the process is simple. Store your changes in your source code repository, then apply a tag that follows the semantic versioning rules I discussed in “Semantic Versioning” on page 246. Go’s semantic versioning supports the concept of pre-releases. Let’s assume that the current version of your module is tagged v1.3.4. You are working on version 1.4.0, which is not quite done, but you want to try importing it into another module. What you should do is append a hyphen (-) to the end of your version tag, followed by an identifier for the pre-release build. In this case, use a tag like v1.4.0-beta1 to indicate beta 1 of version 1.4.0 or v1.4.0-rc2 to indicate release candidate 2. If you 252

|

Chapter 10: Modules, Packages, and Imports

want to depend on a pre-release candidate, you must specify its version explicitly in go get, as Go will not automatically select a pre-release version. If you reach a point where you need to break backward compatibility, the process is more complicated. As you saw when importing version 2 of the simpletax module, a backward-breaking change requires a different import path. There are a few steps to take. First you need to choose a way to store your new version. Go supports two ways for creating the different import paths: • Create a subdirectory within your module named vN, where N is the major version of your module. For example, if you are creating version 2 of your module, call this directory v2. Copy your code into this subdirectory, including the README and LICENSE files. • Create a branch in your version control system. You can put either the old code or the new code on the new branch. Name the branch vN if you are putting the new code on the branch, or vN-1 if you are putting the old code there. For example, if you are creating version 2 of your module and want to put version 1 code on the branch, name the branch v1. After you decide how to store your new code, you need to change the import path in the code in your subdirectory or branch. The module path in your go.mod file must end with /vN, and all the imports within your module must use /vN as well. Going through all your code can be tedious, but Marwan Sulaiman has created a tool that automates the work. Once the paths are fixed, go ahead and implement your changes. Technically, you could just change go.mod and your import state‐ ments, tag your main branch with the latest version, and not bother with a subdirectory or versioned branch. However, this is not a good practice, as it makes it unclear where to find older major versions of your module.

When you are ready to publish your new code, place a tag on your repository that looks like vN.0.0. If you are using the subdirectory system or keeping the latest code on your main branch, tag the main branch. If you are placing your new code on a different branch, tag that branch instead. You can find more details on updating your code to an incompatible version in the post “Go Modules: v2 and Beyond” on The Go Blog and in “Developing a Major Version Update” on the Go developer website.

Versioning Your Module

|

253

Overriding Dependencies Forks happen. While there’s a bias against forking in the open source community, sometimes a module stops being maintained or you want to experiment with changes that aren’t accepted by the module’s owner. A replace directive redirects all refer‐ ences to a module across all your module’s dependencies and replaces them with the specified fork of the module. It looks like this: replace github.com/jonbodner/proteus => github.com/someone/my_proteus v1.0.0

The original module location is specified on the left side of the => and the replace‐ ment on the right. The right side must have a version specified, but specifying a version is optional for the left side. If a version is specified on the left side, only that specific version will be replaced. If the version is not specified, any version of the original module will be replaced with the specific version of the fork. A replace directive can also refer to a path on your local filesystem: replace github.com/jonbodner/proteus => ../projects/proteus

With a local replace directive, the module version must be omitted from the right side. Avoid using local replace directives. They provided a way to modify multiple modules simultaneously before the invention of Go workspaces, but now they are a potential source of broken modules. (I will cover workspaces shortly.) If you share your mod‐ ule via version control, a module with local references in replace directives will probably not build for anyone else, since you cannot guarantee that other people will have the replacement modules in the same locations on their drive.

You also might want to block a specific version of a module from being used. Perhaps it has a bug or is incompatible with your module. Go provides the exclude directive to prevent a specific version of a module from being used: exclude github.com/jonbodner/proteus v0.10.1

When a module version is excluded, any mentions of that module version in any dependent module are ignored. If a module version is excluded and it’s the only version of that module that’s required in your module’s dependencies, use go get to add an indirect import of a different version of the module to your module’s go.mod file so that your module still compiles.

254

|

Chapter 10: Modules, Packages, and Imports

Retracting a Version of Your Module Sooner or later, you will accidentally publish a version of your module that you don’t want anyone to use. Perhaps it was released by accident before testing was complete. Maybe after its release, a critical vulnerability is discovered and no one should use it any more. No matter the reason, Go provides a way for you to indicate that certain versions of a module should be ignored. This is done by adding a retract directive to the go.mod file of your module. It consists of the word retract and the semantic version that should no longer be used. If a range of versions shouldn’t be used, you can exclude all versions in that range by placing the upper and lower bounds within brackets, separated by a comma. While it’s not required, you should include a comment after a version or version range to explain the reason for the retraction. If you wish to retract multiple nonsequential versions, you can specify them with multiple retract directives. In the examples shown, version 1.5.0 is excluded, as are all versions from 1.7.0 to 1.8.5, inclusive: retract v1.5.0 // not fully tested retract [v1.7.0, v.1.8.5] // posts your cat photos to LinkedIn w/o permission

Adding a retract directive to go.mod requires you to create a new version of your module. If the new version contains only the retraction, you should retract it as well. When a version is retracted, existing builds that specified the version will continue to work, but go get and go mod tidy will not upgrade to them. They will no longer appear as options when you use the go list command. If the most recent version of a module is retracted, it will no longer be matched with @latest; the highest unretracted version will match instead. While retract can be confused with exclude, there’s a very impor‐ tant difference. You use retract to prevent others from using specific versions of your module. An exclude directive blocks you from using versions of another module.

Using Workspaces to Modify Modules Simultaneously There’s one drawback to using your source code repository and its tags as a way to track your dependencies and their versions. If you want to make changes to two (or more) modules simultaneously, and you want to experiment with those changes across modules, you need a way to use a local copy of a module instead of the version in the source code repository.

Versioning Your Module

|

255

You can find obsolete advice online to try to solve this issue with temporary replace directives in go.mod that point to local directo‐ ries. Do not do this! It’s too easy to forget to undo these changes before committing and pushing your code. Workspaces were intro‐ duced to avoid this antipattern.

Go uses workspaces to address this issue. A workspace allows you to have multiple modules downloaded to your computer, and references between those modules will automatically resolve to the local source code instead of the code hosted in your repository. This section assumes that you have a GitHub account. If you don’t, you can still follow along. I’m going to use the organization name learning-go-book-2e, but you should replace it with your GitHub account name or organization.

Let’s start with two sample modules. Create a my_workspace directory and in that directory, create two more directories, workspace_lib and workspace_app. In the workspace_lib directory, run go mod init github.com/learning-go-book-2e/work space_lib. Create a file called lib.go with the following content: package workspace_lib func AddNums(a, b int) int { return a + b }

In the workspace_app directory, run go mod init github.com/learning-go -book-2e/workspace_app. Create a file called app.go with the following contents: package main import ( "fmt" "github.com/learning-go-book-2e/workspace_lib" ) func main() { fmt.Println(workspace_lib.AddNums(2, 3)) }

In previous sections, you used go get ./... to add require directives to go.mod. Let’s see what happens if you try it here: $ go get ./... github.com/learning-go-book-2e/workspace_app imports github.com/learning-go-book-2e/workspace_lib: cannot find module providing package github.com/learning-go-book-2e/workspace_lib

256

|

Chapter 10: Modules, Packages, and Imports

Since workspace_lib hasn’t been pushed to GitHub yet, you can’t pull it in. If you try to run go build, you will get a similar error: $ go build app.go:5:2: no required module provides package github.com/learning-go-book-2e/workspace_lib; to add it: go get github.com/learning-go-book-2e/workspace_lib

Let’s take advantage of workspaces to allow workplace_app to see the local copy of workspace_lib. Go to the my_workspace directory and run the following commands: $ go work init ./workspace_app $ go work use ./workspace_lib

This creates a go.work file in my_workspace with the following contents: go 1.20 use ( ./workspace_app ./workspace_lib )

The go.work file is meant for your local computer only. Do not commit it to source control!

Now if you build workspace_app, everything works: $ cd workspace_app $ go build $ ./workspace_app 5

Now that you are sure that workspace_lib does the right thing, it can be pushed to GitHub. In GitHub, create an empty public repository called workspace_lib and then run the following commands in the workspace_lib directory: $ $ $ $ $ $

git git git git git git

init add . commit -m "first commit" remote add origin [email protected]:learning-go-book-2e/workspace_lib.git branch -M main push -u origin main

After running these commands, go to https://github.com/learning-go-book-2e/work space_lib/releases/new (replacing “learning-go-book-2e” with your account or organi‐ zation), and create a new release with the tag v0.1.0.

Versioning Your Module

|

257

Now if you go back to the workspace_app directory and run go get ./..., the require directive is added, because there is a public module that can be downloaded: $ go get ./... go: downloading github.com/learning-go-book-2e/workspace_lib v0.1.0 go: added github.com/learning-go-book-2e/workspace_lib v0.1.0 $ cat go.mod module github.com/learning-go-book-2e/workspace_app go 1.20 require github.com/learning-go-book-2e/workspace_lib v0.1.0

You can validate that your code is working with the public module by setting the environment variable GOWORK=off and building your application: $ rm workspace_app $ GOWORK=off go build $ ./workspace_app 5

Even though there is now a require directive referring to the public module, you can continue to make updates in our local workspace and they will be used instead. In workspace_lib, modify the lib.go file and add the following function: func SubNums(a, b int) int { return a - b }

In workspace_app, modify the app.go file and add the following line to the end of the main function: fmt.Println(workspace_lib.SubNums(2,3))

Now run go build and see it use the local module instead of the public one: $ go build $ ./workspace_app 5 -1

Once you have made the edits and want to release your software, you need to update the version information in your modules’ go.mod files to refer to the updated code. This requires you to commit your modules to source control in dependency order: 1. Choose a modified module that has no dependencies on any of the modified modules in your workspace. 2. Commit this module to your source code repository. 3. Create a new version tag on the newly committed module in your source code repository.

258

|

Chapter 10: Modules, Packages, and Imports

4. Use go get to update the version specified in go.mod in the modules that depend on the newly committed module. 5. Repeat the first four steps until all modified modules are committed. If you have to make changes to workspace_lib in the future and want to test them in workspace_app without pushing back to GitHub and creating lots of temporary versions, you can git pull the latest versions of the modules into your workspace again and make your updates.

Module Proxy Servers Rather than relying on a single, central repository for libraries, Go uses a hybrid model. Every Go module is stored in a source code repository, like GitHub or GitLab. But by default, go get doesn’t fetch code directly from source code repositories. Instead, it sends requests to a proxy server run by Google. When the proxy server receives the go get request, it checks its cache to see if there has been a request for this module version before. If so, it returns the cached information. If a module or a version of a module isn’t cached on the proxy server, it downloads the module from the module’s repository, stores a copy, and returns the module. This allows the proxy server to keep copies of every version of virtually all public Go modules. In addition to the proxy server, Google also maintains a checksum database. It stores information on every version of every module cached by the proxy server. Just as the proxy server protects you from a module or a version of a module being removed from the internet, the checksum database protects you against modifications to a version of a module. This could be malicious (someone has hijacked a module and slipped in malicious code), or it could be inadvertent (a module maintainer fixes a bug or adds a new feature and reuses an existing version tag). In either case, you don’t want to use a module version that has changed because you won’t be building the same binary and don’t know what the effects are on your application. Every time you download a module via go get or go mod tidy, the Go tools calculate a hash for the module and contact the checksum database to compare the calculated hash to the hash stored for that module’s version. If they don’t match, the module isn’t installed.

Specifying a Proxy Server Some people object to sending requests for third-party libraries to Google. There are a few options: • You can disable proxying entirely by setting the GOPROXY environment variable to direct. You’ll download modules directly from their repositories, but if you

Module Proxy Servers

|

259

depend on a version that’s removed from the repository, you won’t be able to access it. • You can run your own proxy server. Both Artifactory and Sonatype have Go proxy server support built into their enterprise repository products. The Athens Project provides an open source proxy server. Install one of these products on your network and then point GOPROXY to the URL.

Using Private Repositories Most organizations keep their code in private repositories. If you want to use a private module in another Go module, you can’t request it from Google’s proxy server. Go will fall back to checking the private repository directly, but you might not want to leak the names of private servers and repositories to external services. If you are using your own proxy server, or if you have disabled proxying, this isn’t an issue. Running a private proxy server has some additional benefits. First, it speeds up downloading of third-party modules, as they are cached in your company’s network. If accessing your private repositories requires authentication, using a private proxy server means that you don’t have to worry about exposing authentication information in your CI/CD pipeline. The private proxy server is configured to authenticate to your private repositories (see the authentication configuration documentation for Athens), while the calls to the private proxy server are unauthenticated. If you are using a public proxy server, you can set the GOPRIVATE environment variable to a comma-separated list of your private repositories. For example, if you set GOPRIVATE to: GOPRIVATE=*.example.com,company.com/repo

any module stored in a repository that’s located at any subdomain of example.com or at a URL that starts with company.com/repo will be downloaded directly.

Additional Details The Go Team has a complete Go Modules Reference available online. In addition to the content in this chapter, the Module Reference also covers topics like using version control systems besides Git, the structure and API of the module cache, additional environment variables for controlling module lookup behavior, and the REST API for the module proxy and checksum database.

260

|

Chapter 10: Modules, Packages, and Imports

Exercises 1. Create a module in your own public repository. This module has a single func‐ tion named Add with two int parameters and one int return value. This function adds the two parameters together and returns them. Make this version v1.0.0. 2. Add godoc comments to your module that describe the package and the Add function. Be sure to include a link to https://www.mathsisfun.com/numbers/addi tion.html in your Add function godoc. Make this version v1.0.1. 3. Change Add to make it generic. Import the golang.org/x/exp/constraints package. Combine the Integer and Float types in that package to create an interface called Number. Rewrite Add to take in two parameters of type Number and return a value of type Number. Version your module again. Because this is a backward-breaking change, this should be v2.0.0 of your module.

Wrapping Up In this chapter, you’ve learned how to organize code and interact with the ecosystem of Go source code. You’ve seen how modules work, how to organize your code into packages, how to use third-party modules, and how to release modules of your own. In the next chapter, you’re going to take a look at more of the development tools that are included with Go, learn about some essential third-party tools, and explore some techniques to give you better control over your build process.

Wrapping Up

|

261

CHAPTER 11

Go Tooling

A programming language doesn’t exist in isolation. For it to be useful, tools must help the developer turn source code into an executable. Since Go is intended to address the problems that software engineers face today and to help them build quality software, careful thought has been put into tooling that simplifies tasks that are often difficult with other development platforms. This includes improvements in how you build, format, update, validate, and distribute, and even how your users will install your code. I have already covered many of the bundled Go tools: go vet, go fmt, go mod, go get, go list, go work, go doc, and go build. The testing support provided by the go test tool is so extensive, it is covered by itself in Chapter 15. In this chapter, you will explore additional tools that make Go development great, both from the Go team and from third parties.

Using go run to Try Out Small Programs Go is a compiled language, which means that before Go code is run, it must be converted into an executable file. This is in contrast to interpreted languages like Python or JavaScript, which allow you to write a quick script to test an idea and execute it immediately. Having that rapid feedback cycle is important, so Go provides similar functionality via the go run command. It builds and executes a program in one step. Let’s go back to the first program from Chapter 1. Put it in a file called hello.go:

263

package main import "fmt" func main() { fmt.Println("Hello, world!") }

(You can also find this code in the Chapter 11 repository in the sample_code/gorun directory.) Once the file is saved, use the go run command to build and execute it: go run hello.go Hello, world!

If you look inside the directory after running the go run command, you see that no binary has been saved there; the only file in the directory is the hello.go file you just created. Where did the executable go (no pun intended)? The go run command does, in fact, compile your code into a binary. However, the binary is built in a temporary directory. The go run command builds the binary, executes the binary from that temporary directory, and then deletes the binary after your program finishes. This makes the go run command useful for testing out small programs or using Go like a scripting language. Use go run when you want to treat a Go program like a script and run the source code immediately.

Adding Third-Party Tools with go install While some people choose to distribute their Go programs as precompiled binaries, tools written in Go can also be built from source and installed on your computer via the go install command. As you saw in “Publishing Your Module” on page 251, Go modules are identified via their source code repositories. The go install command takes an argument, which is the path to the main package in a module’s source code repository, followed by an @ and the version of the tool you want (if you just want to get the latest version, use @latest). It then downloads, compiles, and installs the tool.

264

|

Chapter 11: Go Tooling

Always be sure to include the @version or @latest after the name of the package that you are installing! If you don’t, it triggers a variety of confusing behaviors and is almost certainly not what you want to do. You either get an error message (if the current directory is not in a module, or if the current directory is a module, but the package isn’t referenced in the module’s go.mod file), or it installs the package version mentioned in go.mod.

By default, go install places binaries into the go/bin directory within your home directory. Set the GOBIN environment variable to change this location. It is strongly recommended that you add the go install directory to your executable search path (this is done by modifying the PATH environment variable on both Unix and Windows). For simplicity, all the examples in this chapter assume that you’ve added this directory. Other environment variables are recognized by the go tool. You can get a complete list, along with a brief description of each variable, using the go help environment command. Many of them control low-level behavior that can be safely ignored. I’ll point out the relevant ones as needed. Some online resources tell you to set the GOROOT or GOPATH envi‐ ronment variables. GOROOT specifies the location where your Go development environment is installed, and GOPATH was used to store all Go source code, both your own and third-party dependen‐ cies. Setting these variables is no longer necessary; the go tool figures out GOROOT automatically, and GOPATH-based development has been superseded by modules.

Let’s look at a quick example. Jaana Dogan created a great Go tool called hey that load tests HTTP servers. You can point it at the website of your choosing or an application that you’ve written. Here’s how to install hey with the go install command: $ go install github.com/rakyll/hey@latest go: downloading github.com/rakyll/hey v0.1.4 go: downloading golang.org/x/net v0.0.0-20181017193950-04a2e542c03f go: downloading golang.org/x/text v0.3.0

This downloads hey and all its dependencies, builds the program, and installs the binary in your Go binary directory.

Adding Third-Party Tools with go install

|

265

As I covered in “Module Proxy Servers” on page 259, the contents of Go repositories are cached in proxy servers. Depending on the repository and the values in your GOPROXY environment variable, go install may download from a proxy or directly from a repository. If go install downloads directly from a repository, it relies on command-line tools being installed on your computer. For exam‐ ple, you must have Git installed to download from GitHub.

Now that you have built and installed hey, you can run it with the following: $ hey https://go.dev Summary: Total: Slowest: Fastest: Average: Requests/sec:

2.1272 secs 1.4227 secs 0.0573 secs 0.3467 secs 94.0181

If you have already installed a tool and want to update it to a newer version, rerun go install with the newer version specified or with @latest: $ go install github.com/rakyll/hey@latest

Of course, you don’t need to leave programs installed via go install in the go/bin directory; they are regular executable binaries and can be stored anywhere on your computer. Likewise, you don’t have to distribute programs written in Go using go install; you can put a binary up for download. However, go install is convenient for Go developers, and it has become the method of choice for distributing thirdparty developer tools.

Improving Import Formatting with goimports An enhanced version of go fmt called goimports also cleans up your import state‐ ments. It puts them in alphabetical order, removes unused imports, and attempts to guess any unspecified imports. Its guesses are sometimes inaccurate, so you should insert imports yourself. You can download goimports with the command go install golang.org/x/ tools/cmd/goimports@latest. You run it across your project with this command: $ goimports -l -w .

The -l flag tells goimports to print the files with incorrect formatting to the console. The -w flag tells goimports to modify the files in place. The . specifies the files to be scanned: everything in the current directory and all its subdirectories.

266

|

Chapter 11: Go Tooling

The packages under golang.org/x are part of the Go Project but outside the main Go tree. While useful, they are developed under looser compatibility requirements than the Go standard library and may introduce backward-breaking changes. Some packages in the standard library, such as the context package that is covered in Chapter 14, started out in golang.org/x. The pkgsite tool that was covered in “Documenting Your Code with Go Doc Comments” on page 231 is also located there. You can see the other packages in the “Sub-repositories” section.

Using Code-Quality Scanners Back in “go vet” on page 7, you looked at the built-in tool go vet, which scans source code for common programming errors. Many third-party tools can check code style and scan for potential bugs that are missed by go vet. These tools are often called linters.1 In addition to likely programming errors, some of the changes suggested by these tools include properly naming variables, formatting error messages, and placing comments on public methods and types. These aren’t errors since they don’t keep your programs from compiling or make your program run incorrectly, but they do flag situations where you are writing nonidiomatic code. When you add linters to your build process, follow the old maxim “trust, but verify.” Since the kinds of issues that linters find are fuzzier, they sometimes have false positives and false negatives. This means you don’t have to make the changes that they suggest, but you should take the suggestions seriously. Go developers expect code to look a certain way and follow certain rules, and if your code does not, it sticks out. If you find a linter’s suggestion to be unhelpful, each linting tool allows you to add a comment to your source code that blocks the errant result (the format of the comment varies from linter to linter; check each tool’s documentation to learn what to write). Your comment should also include an explanation of why you are ignoring the linter’s finding, so code reviewers (and future you, when you look back on your source code in six months) understand your reasoning.

1 The term “linter” comes from the original lint program written by Steve Johnson when he was on the Unix

team at Bell Labs and described in his 1978 paper. The name comes from the tiny bits of fabric that come off clothes in a dryer and are captured by a filter. He saw his program as being like a filter capturing small errors.

Using Code-Quality Scanners

|

267

staticcheck If you had to pick one third-party scanner, use staticcheck. It is supported by many companies that are active in the Go community, includes more than 150 code-quality checks, and tries to produce few to no false positives. It is installed via go install honnef.co/go/tools/cmd/staticcheck@latest. Invoke it with staticcheck ./... to examine your module. Here’s an example of something that staticcheck finds that go vet does not: package main import "fmt" func main() { s := fmt.Sprintf("Hello") fmt.Println(s) }

(You can also find this code in the sample_code/staticcheck_test directory in the Chapter 11 repository.) If you run go vet on this code, it doesn’t find anything wrong. But, staticcheck notices a problem: $ staticcheck ./... main.go:6:7: unnecessary use of fmt.Sprintf (S1039)

Pass the code in parentheses to staticcheck with the -explain flag for an explana‐ tion of the issue: $ staticcheck -explain S1039 Unnecessary use of fmt.Sprint Calling fmt.Sprint with a single string argument is unnecessary and identical to using the string directly. Available since 2020.1 Online documentation https://staticcheck.io/docs/checks#S1039

Another common issue that staticcheck finds is unused assignments to variables. While the Go compiler requires all variables to be read once, it doesn’t check that every value assigned to a variable is read. It is a common practice to reuse an err variable when there are multiple function calls within a function. If you forget to write if err != nil after one of those function invocations, the compiler won’t be able to help you. However, staticcheck can. This code compiles without a problem:

268

|

Chapter 11: Go Tooling

func main() { err := returnErr(false) if err != nil { fmt.Println(err) } err = returnErr(true) fmt.Println("end of program") }

(This code is in the Chapter 11 repository in the sample_code/check_err directory.) Running staticcheck finds the mistake: $ staticcheck ./... main.go:13:2: this value of err is never used (SA4006) main.go:13:8: returnErr doesn't have side effects and its return value is ignored (SA4017)

There are two related issues on line 13. The first is that the error returned by returnErr is never read. The second is that the returnErr function’s output (the error) is being ignored.

revive Another good linting option is revive. It is based on golint, a tool that used to be maintained by the Go team. Install revive with the command go install github.com/mgechev/revive@latest. By default, it enables only the rules that were present in golint. It can find style and code-quality issues like exported identifiers that don’t have comments, variables that don’t follow naming conventions, or error return values that aren’t the last return value. With a configuration file, you can turn on many more rules. For example, to enable a check for shadowing of universe block identifiers, create a file named built_in.toml with the following contents: [rule.redefines-builtin-id]

If you scan the following code: package main import "fmt" func main() { true := false fmt.Println(true) }

you’ll get this warning:

Using Code-Quality Scanners

|

269

$ revive -config built_in.toml ./... main.go:6:2: assignment creates a shadow of built-in identifier true

(You can also find this code in the Chapter 11 repository in the sample_code/ revive_test directory.) Other rules that can be enabled are focused on opinionated code organization, like limiting the number of lines in a function or number of public structs in a file. There are even rules for evaluating the complexity of the logic in a function. Check out the revive documentation and its supported rules.

golangci-lint Finally, if you’d rather take the buffet approach to tool selection, there’s golangcilint. It is designed to make it as efficient as possible to configure and run over 50 code-quality tools, including go vet, staticcheck, and revive. While you can use go install to install golangci-lint, it is recommended that you download a binary version instead. Follow the installation instructions on the website. Once it is installed, you run golangci-lint: $ golangci-lint run

Back in “Unused Variables” on page 32, you looked at a program with variables set to values that were never read, and I mentioned that go vet and the go compiler were unable to detect these issues. Neither staticcheck nor revive catches this problem. However, one of the tools bundled with golangci-lint does: $ golangci-lint run main.go:6:2: ineffectual assignment to x (ineffassign) x := 10 ^ main.go:9:2: ineffectual assignment to x (ineffassign) x = 30 ^

You can also use golangci-lint to provide shadowing checks that go beyond what revive can do. Configure golangci-lint to detect shadowing of both universe block identifiers and identifiers within your own code by putting the following configura‐ tion into a file named .golangci.yml in the directory where you run golangci-lint: linters: enable: - govet - predeclared linters-settings: govet: check-shadowing: true settings:

270

|

Chapter 11: Go Tooling

shadow: strict: true enable-all: true

With these settings, running golangci-lint on this code package main import "fmt" var b = 20 func main() { true := false a := 10 b := 30 if true { a := 20 fmt.Println(a) } fmt.Println(a, b) }

detects the following issues: $ golangci-lint run main.go:5:5: var `b` is unused (unused) var b = 20 ^ main.go:10:2: shadow: declaration of "b" shadows declaration at line 5 (govet) b := 30 ^ main.go:12:3: shadow: declaration of "a" shadows declaration at line 9 (govet) a := 20 ^ main.go:8:2: variable true has same name as predeclared identifier (predeclared) true := false ^

(You can find both golangci-lint code samples in the Chapter 11 repository in the sample_code/golangci-lint_test directory.) Because golangci-lint runs so many tools (as of this writing, it runs 7 different tools by default and allows you to enable more than 50 more), it’s inevitable that your team may disagree with some of its suggestions. Review the documentation to understand what each tool can do. Once you come to agreement on which linters to enable, update the .golangci.yml file at the root of your module and commit it to source control. Check out the documentation for the file format.

Using Code-Quality Scanners

|

271

While golangci-lint allows you to have a configuration file in your home directory, don’t put one there if you are working with other developers. Unless you enjoy adding hours of silly arguments to your code reviews, you want to make sure that everyone is using the same code-quality tests and formatting rules.

I recommend that you start using go vet as a required part of your automated build process. Add staticcheck next since it produces few false positives. When you are interested in configuring tools and setting code-quality standards, look at revive, but be aware that it might have false positives and false negatives, so you can’t require your team to fix every issue it reports. Once you are used to their recommendations, try out golangci-lint and tweak its settings until it works for your team.

Using govulncheck to Scan for Vulnerable Dependencies One kind of code quality isn’t enforced by the tools you’ve looked at so far: software vulnerabilities. Having a rich ecosystem of third-party modules is fantastic, but clever hackers find security vulnerabilities in libraries and exploit them. Developers patch these bugs when they are reported, but how do you ensure that the software that uses a vulnerable version of a library is updated to the fixed version? The Go team has released a tool called govulncheck to address this situation. It scans through your dependencies and finds known vulnerabilities in both the standard library and in third-party libraries imported into your module. These vulnerabilities are reported in a public database maintained by the Go team. You can install it with this: $ go install golang.org/x/vuln/cmd/govulncheck@latest

Let’s take a look at a small program to see the vulnerability checker in action. First, download the repository. The source code in main.go is very simple. It imports a third-party YAML library and uses it to load a small YAML string into a struct: func main() { info := Info{} err := yaml.Unmarshal([]byte(data), &info) if err != nil { fmt.Printf("error: %v\n", err) os.Exit(1) } fmt.Printf("%+v\n", info) }

The go.mod file contains the required modules and their versions: module github.com/learning-go-book-2e/vulnerable

272

|

Chapter 11: Go Tooling

go 1.20 require gopkg.in/yaml.v2 v2.2.7 require gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect

Let’s see what happens when you run govulncheck on this project: $ govulncheck ./... Using go1.21 and [email protected] with vulnerability data from https://vuln.go.dev (last modified 2023-07-27 20:09:46 +0000 UTC). Scanning your code and 49 packages across 1 dependent module for known vulnerabilities... Vulnerability #1: GO-2020-0036 Excessive resource consumption in YAML parsing in gopkg.in/yaml.v2 More info: https://pkg.go.dev/vuln/GO-2020-0036 Module: gopkg.in/yaml.v2 Found in: gopkg.in/[email protected] Fixed in: gopkg.in/[email protected] Example traces found: #1: main.go:25:23: vulnerable.main calls yaml.Unmarshal Your code is affected by 1 vulnerability from 1 module.

This module is using an old and vulnerable version of the YAML package. govuln check helpfully gives the exact line in the codebase that calls the problematic code. If govulncheck knows there a vulnerability in a module that your code uses, but can’t find an explicit call to the buggy part of the module, you’ll get a less severe warning. The message informs you of the library’s vulnerability and what version resolves the issue, but it will also let you know that your module is likely not affected.

Let’s update to a fixed version and see if that solves the problem: $ go get -u=patch gopkg.in/yaml.v2 go: downloading gopkg.in/yaml.v2 v2.2.8 go: upgraded gopkg.in/yaml.v2 v2.2.7 => v2.2.8 $ govulncheck ./... Using go1.21and [email protected] with vulnerability data from https://vuln.go.dev (last modified 2023-07-27 20:09:46 +0000 UTC). Scanning your code and 49 packages across 1 dependent module for known vulnerabilities... No vulnerabilities found.

Using govulncheck to Scan for Vulnerable Dependencies

|

273

Remember, you should always strive for the smallest possible change to your project’s dependencies since that makes it less likely that a change in a dependency breaks your code. For that reason, update to the most recent patch version for v2.2.x, which is v2.2.8. When govulncheck is run again, there are no known issues. While govulncheck currently requires a go install to download it, it likely will be added to the standard toolset eventually. In the meantime, be sure to install and run it against your projects as a regular part of your builds. You can learn more about it in the blog post that announced it.

Embedding Content into Your Program Many programs are distributed with directories of support files; you might have web page templates or some standard data that’s loaded when a program starts. If a Go program needs support files, you could include a directory of files, but this takes away one of the advantages of Go: its ability to compile to a single binary that’s easy to ship and distribute. However, there’s another option. You can embed the contents of the files within your Go binary by using go:embed comments. You can find a program demonstrating embedding on GitHub in the sam‐ ple_code/embed_passwords directory in the Chapter 11 repository. It checks to see if a password is one of the 10,000 most commonly used passwords. Rather than write that list of passwords directly into the source code, you’re going to embed it. The code in main.go is straightforward: package main import ( _ "embed" "fmt" "os" "strings" ) //go:embed passwords.txt var passwords string func main() { pwds := strings.Split(passwords, "\n") if len(os.Args) > 1 { for _, v := range pwds { if v == os.Args[1] { fmt.Println("true") os.Exit(0) } } fmt.Println("false")

274

|

Chapter 11: Go Tooling

} }

You must do two things to enable embedding. First, the embed package must be imported. The Go compiler uses this import as a flag to indicate that embedding should be enabled. Because this sample code isn’t referring to anything exported from the embed package, you use a blank import, which was discussed in “Avoiding the init Function if Possible” on page 239. The only symbol exported from embed is FS. You’ll see it in the next example. Next, you place a magic comment directly before each package-level variable that holds the contents of a file. This comment must start with go:embed, with no space between the slashes and go:embed. The comment must also be on the line directly before the variable. (Technically, it is legal to have blank lines or other, nonmagic comments between the embedding comment and the variable declaration, but don’t do it.) In this sample, you are embedding the contents of passwords.txt into the package-level variable named passwords. It is idiomatic to treat a variable with an embedded value as immutable. As mentioned earlier, you can embed into only a package-level variable. The variable must be of type string, []byte, or embed.FS. If you have a single file, it’s simplest to use string or []byte. If you need to place one or more directories of files into your program, use a variable of type embed.FS. This type implements three interfaces defined in the io/fs pack‐ age: FS, ReadDirFS, and ReadFileFS. This allows an instance of embed.FS to represent a virtual filesystem. The following program provides a simple command-line help system. If you don’t provide a help file, it lists all available files. If you specify a file that’s not present, it returns an error: package main import ( "embed" "fmt" "io/fs" "os" "strings" ) //go:embed help var helpInfo embed.FS func main() { if len(os.Args) == 1 { printHelpFiles() os.Exit(0) } data, err := helpInfo.ReadFile("help/" + os.Args[1]) if err != nil {

Embedding Content into Your Program

|

275

fmt.Println(err) os.Exit(1) } fmt.Println(string(data)) }

You can find this code, along with the sample help files, in the sample_code/help_sys‐ tem directory in the Chapter 11 repository. Here’s the output when you build and run this program: $ go build $ ./help_system contents: advanced/topic1.txt advanced/topic2.txt info.txt $ ./help_system advanced/topic1.txt This is advanced topic 1. $ ./help_system advanced/topic3.txt open help/advanced/topic3.txt: file does not exist

You should notice a couple of things. First, you no longer need to use a blank import for embed, since you are using embed.FS. Second, the directory name is part of the filesystem that’s embedded. The users of this program don’t enter the “help/” prefix, so you have to prepend it in the call to ReadFile. The printHelpFiles function shows how you can treat an embedded virtual filesys‐ tem just like a real one: func printHelpFiles() { fmt.Println("contents:") fs.WalkDir(helpInfo, "help", func(path string, d fs.DirEntry, err error) error { if !d.IsDir() { _, fileName, _ := strings.Cut(path, "/") fmt.Println(fileName) } return nil }) }

You use the WalkDir function in io/fs to walk through the embedded filesystem. WalkDir takes in an instance of fs.FS, a path to start at, and a function. This function is called for every file and directory in the filesystem, starting from the specified path. If the fs.DirEntry is not a directory, you print out its full pathname, removing the help/ prefix by using strings.Cut.

276

|

Chapter 11: Go Tooling

There are a few more things to know about file embedding. While all the examples have been text files, you can embed binary files as well. You can also embed multiple files or directories into a single embed.FS variable by specifying their names, separa‐ ted by spaces. When embedding a file or directory that has a space in its name, put the name in quotes. In addition to exact file and directory names, you can use wildcards and ranges to specify the names of the files and directories you want to embed. The syntax is defined in the documentation for the Match function in the path package in the standard library, but it follows common conventions. For example, * matches 0 or more characters, and ? matches a single character. All embedding specifications, whether or not they use match patterns, are checked by the compiler. If they aren’t valid, compilation fails. Here are the ways a pattern can be invalid: • If the specified name or pattern doesn’t match a file or directory • If you specify multiple filenames or patterns for a string or []byte variable • If you specify a pattern for a string or []byte variable and it matches more than one file

Embedding Hidden Files Including files in a directory tree that start with . or _ is a little complicated. Many operating systems consider these to be hidden files, so they are not included by default when a directory name is specified. However, you can override this behavior in two ways. The first is to put /* after the name of a directory you want to embed. This will include all hidden files within the root directory, but it will not include hidden files in its subdirectories. To include all hidden files in all subdirectories, put all: before the name of the directory. This sample program (which you can find in the sample_code/embed_hidden direc‐ tory in the Chapter 11 repository) makes this easier to understand. In the sample, the directory parent_dir contains two files, .hidden and visible, and one subdirectory, child_dir. The child_dir subdirectory contains two files, .hidden and visible. Here is the code for the program: //go:embed parent_dir var noHidden embed.FS //go:embed parent_dir/* var parentHiddenOnly embed.FS //go:embed all:parent_dir var allHidden embed.FS

Embedding Hidden Files

|

277

func main() { checkForHidden("noHidden", noHidden) checkForHidden("parentHiddenOnly", parentHiddenOnly) checkForHidden("allHidden", allHidden) } func checkForHidden(name string, dir embed.FS) { fmt.Println(name) allFileNames := []string{ "parent_dir/.hidden", "parent_dir/child_dir/.hidden", } for _, v := range allFileNames { _, err := dir.Open(v) if err == nil { fmt.Println(v, "found") } } fmt.Println() }

The output of the program is shown here: noHidden parentHiddenOnly parent_dir/.hidden found allHidden parent_dir/.hidden found parent_dir/child_dir/.hidden found

Using go generate The go generate tool is a little different, because it doesn’t do anything by itself. When you run go generate, it looks for specially formatted comments in your source code and runs programs specified in those comments. While you could use go generate to run anything at all, it is most commonly used by developers to run tools that (unsurprisingly, given the name) generate source code. This could be from analyzing existing code and adding functionality or processing schemas and making source code out of it. A good example of something that can be automatically converted to code are Pro‐ tocol Buffers, sometimes called protobufs. Protobuf is a popular binary format that is used by Google to store and transmit data. When working with protobufs, you write a schema, which is a language-independent description of the data structure. Developers who want to write programs to interact with data in protobuf format run

278

| Chapter 11: Go Tooling

tools that process the schema and produce language-specific data structures to hold the data and language-specific functions to read and write data in protobuf format. Let’s see how this works in Go. You can find a sample module in the proto_generate repo. The module contains a protobuf schema file called person.proto: syntax = "proto3"; message Person { string name = 1; int32 id = 2; string email = 3; }

While making a struct that implements Person would be easy, writing the code to convert back and forth from the binary format is difficult. Let’s use tools from Google to do the hard work and invoke them with go generate. You need to install two things. The first is the protoc binary for your computer (see the installation instructions. Next, use go install to install the Go protobuf plug-ins: $ go install google.golang.org/protobuf/cmd/[email protected]

In main.go, there is the magic comment that’s processed by go generate: //go:generate protoc -I=. --go_out=. --go_opt=module=github.com/learning-go-book-2e/proto_generate --go_opt=Mperson.proto=github.com/learning-go-book-2e/proto_generate/data person.proto

(If you look at the source code in GitHub, you’ll see this should be a single line. It’s wrapped to fit the constraints of a printed page.) Run go generate by typing the following: $ go generate ./...

After running go generate, you’ll see a new directory called data that contains a file named person.pb.go. It contains the source code for the Person struct, and some methods and functions that are used by the Marshal and Unmarshal functions in the google.golang.org/protobuf/proto module. You call these functions in your main function: func main() { p := &data.Person{ Name: "Bob Bobson", Id: 20, Email: "[email protected]", } fmt.Println(p) protoBytes, _ := proto.Marshal(p) fmt.Println(protoBytes) var p2 data.Person

Using go generate

|

279

proto.Unmarshal(protoBytes, &p2) fmt.Println(&p2) }

Build and run the program as usual: $ go build $ ./proto_generate name:"Bob Bobson" id:20 email:"[email protected]" [10 10 66 111 98 32 66 111 98 115 111 110 16 20 26 14 98 111 98 64 98 111 98 115 111 110 46 99 111 109] name:"Bob Bobson" id:20 email:"[email protected]"

Another tool commonly used with go generate is stringer. As I discussed in “iota Is for Enumerations—Sometimes” on page 152, enumerations in Go lack many of the features that are found in other languages with enumerations. One of those features is automatically generating a printable name for each value in the enumeration. The stringer tool is used with go generate to add a String method to your enumera‐ tion’s values so they can be printed. Install stringer with go install golang.org/x/tools/cmd/stringer@latest. The sample_code/stringer_demo directory in the Chapter 11 repository provides a very simple example of how to use stringer. Here’s the source in main.go: type Direction int const ( _ Direction = iota North South East West ) //go:generate stringer -type=Direction func main() { fmt.Println(North.String()) }

Run go generate ./... and you’ll see a new file generated called direction_string.go. Use go build to build the string_demo binary and when you run it, you’ll get the output: North

You can configure stringer and its output in multiple ways. Arjun Mahishi has written a great blog post describing how to use stringer and customize its output.

280

| Chapter 11: Go Tooling

Working with go generate and Makefiles Since the job of go generate is to run other tools, you might wonder if it’s worth using when you have a perfectly good Makefile in your project. The advantage of go generate is that it creates a separation of responsibilities. Use go generate commands to mechanically create source code, and use the Makefile to validate and compile source code. It is a best practice to commit the source code created by go generate to version control. (The sample projects in the Chapter 11 repository don’t include generated source code so you can see go generate work.) This allows people browsing your source code to see everything that’s invoked, even the generated parts. It also means they don’t need to have tools like protoc installed in order to build your code. Checking in your generated source code technically means that you don’t need to run go generate unless it will produce different output, such as processing a modified protobuf definition or an updated enumeration. However, it’s still a good idea to automate calling go generate before go build. Relying on a manual process is asking for trouble. Some generator tools, like stringer, include clever tricks to block compilation if you forget to rerun go generate, but that’s not universal. You’ll inevitably waste time during testing trying to understand why a change didn’t show up before realizing that you forgot to invoke go generate. (I made this mistake multiple times before I learned my lesson.) Given this, it is best to add a generate step to your Makefile and make it a dependency of your build step. However, I would disregard this advice in two situations. The first is if invoking go generate on identical input produces source files with minor differences, such as a timestamp. A well-written go generate tool should produce identical output every

time it’s run on the same input, but there are no guarantees that every tool you need to use is well written. You don’t want to keep on checking in new versions of files that are functionally identical, as they will clutter your version control system and make your code reviews noisier.

The second situation is if go generate takes a very long time to complete. Fast builds are a feature of Go, because they allow developers to stay focused and get rapid feedback. If you are noticeably slowing down a build to generate identical files, the loss in developer productivity is not worth it. In both cases, all you can do is leave lots of comments to remind people to rebuild when things change and hope that everyone on your team is diligent.

Reading the Build Info Inside a Go Binary As companies develop more of their own software, it is becoming increasingly com‐ mon for them to want to understand exactly what they have deployed to their data Reading the Build Info Inside a Go Binary

|

281

centers and cloud environments, down to the version and the dependencies. You might wonder why you’d want to get this information from compiled code. After all, a company already has this information in version control. Companies with mature development and deployment pipelines can capture this information right before deploying a program, allowing them to be sure that the information is accurate. However, many, if not most, companies don’t track the exact version of internal software that’s deployed. In some cases, software can be deployed for years without being replaced, and no one remembers much about it. If a vulnerability is reported in a version of a third-party library, you need to either find some way to scan your deployed software and figure out what versions of third-party libraries are deployed, or redeploy everything just to be safe. In the Java world, this exact problem happened when a serious vulnerability was discovered in the popular Log4j library. Luckily, Go solves this problem for you. Every Go binary you create with go build automatically contains build information on what versions of what modules make up the binary, and also what build commands were used, what version control system was used, and what revision the code was at in your version control system. You can view this information with the go version -m command. The following shows the output for the vulnerable program when built on an Apple Silicon Mac: $ go build go: downloading gopkg.in/yaml.v2 v2.2.7 $ go version -m vulnerable vulnerable: go1.20 path github.com/learning-go-book-2e/vulnerable mod github.com/learning-go-book-2e/vulnerable (devel) dep gopkg.in/yaml.v2 v2.2.7 h1:VUgggvou5XRW9mHwD/yXxIYSMtY0zoKQf/v... build -compiler=gc build CGO_ENABLED=1 build CGO_CFLAGS= build CGO_CPPFLAGS= build CGO_CXXFLAGS= build CGO_LDFLAGS= build GOARCH=arm64 build GOOS=darwin build vcs=git build vcs.revision=623a65b94fd02ea6f18df53afaaea3510cd1e611 build vcs.time=2022-10-02T03:31:05Z build vcs.modified=false

Because this information is embedded into every binary, govulncheck is capable of scanning Go programs to check for libraries with known vulnerabilities: $ govulncheck -mode binary vulnerable Using [email protected] with vulnerability data from https://vuln.go.dev (last modified 2023-07-27 20:09:46 +0000 UTC).

282

|

Chapter 11: Go Tooling

Scanning your binary for known vulnerabilities... Vulnerability #1: GO-2020-0036 Excessive resource consumption in YAML parsing in gopkg.in/yaml.v2 More info: https://pkg.go.dev/vuln/GO-2020-0036 Module: gopkg.in/yaml.v2 Found in: gopkg.in/[email protected] Fixed in: gopkg.in/[email protected] Example traces found: #1: yaml.Unmarshal Your code is affected by 1 vulnerability from 1 module.

Be aware that govulncheck can’t track down exact lines of code when inspecting a binary. If govulncheck finds a problem in a binary, use go version -m to find out the exact deployed version, check the code out of version control, and run it again against the source code to pinpoint the issue. If you want to build your own tools to read the build information, look at the debug/ buildinfo package in the standard library.

Building Go Binaries for Other Platforms One of the advantages of a VM-based language like Java, JavaScript, or Python is that you can take your code and get it to run on any computer where the virtual machine has been installed. This portability makes it easy for developers using these languages to build programs on a Windows or Mac computer and deploy it on a Linux server, even though the operating system and possibly the CPU architecture are different. Go programs are compiled to native code, so the generated binary is compatible with only a single operating system and CPU architecture. However, that doesn’t mean that Go developers need to maintain a menagerie of machines (virtual or otherwise) to release on multiple platforms. The go build command makes it easy to cross-compile, or create a binary for a different operating system and/or CPU. When go build is run, the target operating system is specified by the GOOS environment variable. Similarly, the GOARCH environment variable specifies the CPU architecture. If you don’t set them explicitly, go build defaults to using the values for your current computer, which is why you’ve never had to worry about these variables before. You can find the valid values and combinations for GOOS and GOARCH (sometimes pronounced “GOOSE” and “GORCH”) in the installation documentation. Some of the supported operating systems and CPUs are a bit esoteric, and others might require some translation. For example, darwin refers to macOS (Darwin is the name of the macOS kernel), and amd64 means 64-bit Intel-compatible CPUs.

Building Go Binaries for Other Platforms

|

283

Let’s go back to the vulnerable program one last time. When using an Apple Silicon Mac (which has an ARM64 CPU), running go build defaults to darwin for GOOS and arm64 for GOARCH. You can confirm this using the file command: $ go build $ file vulnerable vulnerable: Mach-O 64-bit executable arm64

Here is how to build a binary for Linux on 64-bit Intel CPUs: $ GOOS=linux GOARCH=amd64 go build $ file vulnerable vulnerable: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=IDHVCE8XQPpWluGpMXpX/4VU3GpRZEifN 8TzUrT_6/1c30VcDYNVPfSSN-zCkz/JsZSLAbWkxqIVhPkC5p5, with debug_info, not stripped

Using Build Tags When writing programs that need to run on multiple operating systems or CPU architectures, you sometimes need different code for different platforms. You also might want to write a module that takes advantage of the latest Go features but is still backward compatible with older Go compilers. You can create targeted code in two ways. The first is to use the name of the file to indicate when the file should be included in the build. You do this by adding the target GOOS and GOARCH, separated by _, to the filename before .go. For example, if you have a file that you want to be compiled only on Windows, you’d name the file something_windows.go, but if you wanted it to be compiled only when building for ARM64 Windows, name the file something_windows_arm64.go. A build tag (also called a build constraint) is the other option you can use to specify when a file is compiled. Like embedding and generating, build tags take advantage of a magic comment. In this case, it’s //go:build. This comment must be placed on the line before the package declaration in your file. Build tags use boolean operators (||, &&, and !) and parentheses to specify exact build rules for architectures, operating systems, and Go versions. The build tag //go:build (!darwin && !linux) || (darwin && !go1.12)—which really appears in the Go standard library—specifies that the file should not be compiled on Linux or macOS, except it’s OK to compile it on macOS if the Go version is 1.11 or earlier. Some meta build constraints are also available. The constraint unix matches any Unix-ish platform, and cgo matches if cgo is supported by the current platform and is enabled. (I cover cgo in “Cgo Is for Integration, Not Performance” on page 433.) The question becomes when you should use filenames to indicate where to run code and when you should use build tags. Because build tags allow binary operators, you 284

|

Chapter 11: Go Tooling

can specify a more specific set of platforms with them. The Go standard library sometimes takes a belt-and-suspenders approach. The package internal/cpu in the standard library has platform-specific source code for CPU feature detection. The file internal/cpu/cpu_arm64_darwin.go has a name that indicates that it is meant only for computers using Apple CPUs. It also has a //go:build arm64 && darwin && !ios line in the file to indicate that it should be compiled only when building for Apple Silicon Macs and not for iPhones or iPads. The build tags are able to specify the target platform with more detail, but following the filename convention makes it easy for a person to find the right file for a given platform. In addition to the built-in build tags that represent Go versions, operating systems and CPU architectures, you can also use any string at all as a custom build tag. You can then control compilation of that file with the -tags command-line flag. For example, if you put //go:build gopher on the line before the package declaration in a file, it will not be compiled unless you include a -tags gopher flag as part of the go build, go run, or go test command. Custom build tags are surprisingly handy. If you have a source file that you don’t want to build right now (perhaps it doesn’t compile yet, or it’s an experiment that’s not ready to be included), it is idiomatic to skip over the file by putting //go:build ignore on the line before the package declaration. You will see another use for custom build tags when looking at integration tests in “Using Integration Tests and Build Tags” on page 405. When writing your build tags, make sure there isn’t any whitespace between the // and go:build. If there is, Go will not consider it a build tag.

Testing Versions of Go Despite Go’s strong backward-compatibility guarantees, bugs do happen. It’s natural to want to make sure that a new release doesn’t break your programs. You also might get a bug report from a user of your library saying that your code doesn’t work as expected on an older version of Go. One option is to install a secondary Go environment. For example, if you wanted to try out version 1.19.2, you would use the following commands: $ go install golang.org/dl/go1.19.2@latest $ go1.19.2 download

Testing Versions of Go

|

285

You can then use the command go1.19.2 instead of the go command to see if version 1.19.2 works for your programs: $ go1.19.2 build

Once you have validated that your code works, you can uninstall the secondary environment. Go stores secondary Go environments in the sdk directory within your home directory. To uninstall, delete the environment from the sdk directory and the binary from the go/bin directory. Here’s how to do that on macOS, Linux, and BSD: $ rm -rf ~/sdk/go.19.2 $ rm ~/go/bin/go1.19.2

Using go help to Learn More About Go Tooling You can learn more about Go’s tooling and runtime environment with the go help command. It contains exhaustive information about all the commands mentioned here, as well as things like modules, import path syntax, and working with nonpublic source code. For example, you can get information on import path syntax by typing go help importpath.

Exercises These exercises cover some of the tools that you’ve learned about in this chapter. You can find the solutions in the exercise_solutions directory in the Chapter 11 repository. 1. Go to the UN’s Universal Declaration of Human Rights (UDHR) page and copy the text of the UDHR into a text file called english_rights.txt. Click the Other Languages link and copy the document text in a few additional languages into files named LANGUAGE_rights.txt. Create a program that embeds these files into package-level variables. Your program should take in one command-line parameter, the name of a language. It should then print out the UDHR in that language. 2. Use go install to install staticcheck. Run it against your program and fix any problems it finds. 3. Cross-compile your program for ARM64 on Windows. If you are using an ARM64 Windows computer, cross-compile for AMD64 on Linux.

Wrapping Up In this chapter, you learned about the tools that Go provides to improve software engineering practices and third-party code-quality tools. In the next chapter, you’re going to explore one of the signature features in Go: concurrency.

286

|

Chapter 11: Go Tooling

CHAPTER 12

Concurrency in Go

Concurrency is the computer science term for breaking up a single process into inde‐ pendent components and specifying how these components safely share data. Most languages provide concurrency via a library using operating system–level threads that share data by attempting to acquire locks. Go is different. Its main concurrency model, arguably Go’s most famous feature, is based on Communicating Sequential Processes (CSP). This style for concurrency was described in 1978 in a paper by Tony Hoare, the man who invented the Quicksort algorithm. The patterns implemented with CSP are just as powerful as the standard ones but are far easier to understand. In this chapter, you are going to quickly review the features that are the backbone of concurrency in Go: goroutines, channels, and the select keyword. Then you are going to look at some common Go concurrency patterns and learn about the situations where lower-level techniques are a better approach.

When to Use Concurrency Let’s start with a word of caution. Be sure that your program benefits from concur‐ rency. When new Go developers start experimenting with concurrency, they tend to go through a series of stages: 1. This is amazing; I’m going to put everything in goroutines! 2. My program isn’t any faster. I’m adding buffers to my channels. 3. My channels are blocking and I’m getting deadlocks. I’m going to use buffered channels with really big buffers.

287

4. My channels are still blocking. I’m going to use mutexes. 5. Forget it, I’m giving up on concurrency. People are attracted to concurrency because they believe concurrent programs run faster. Unfortunately, that’s not always the case. More concurrency doesn’t automati‐ cally make things faster, and it can make code harder to understand. The key is understanding that concurrency is not parallelism. Concurrency is a tool to better structure the problem you are trying to solve. Whether concurrent code runs in parallel (at the same time) depends on the hard‐ ware and whether the algorithm allows it. In 1967, Gene Amdahl, one of the pioneers of computer science, derived Amdahl’s law. It is a formula for figuring out how much parallel processing can improve performance, given how much of the work must be performed sequentially. If you want to dive into the details on Amdahl’s law, you can learn more in The Art of Concurrency by Clay Breshears (O’Reilly). For our purposes, all you need to understand is that more concurrency does not mean more speed. Broadly speaking, all programs follow the same three-step process: they take data, transform it, and then output the result. Whether you should use concurrency in your program depends on how data flows through the steps in your program. Some‐ times two steps can be concurrent because the data from one is not required for the other to proceed, and at other times two steps must happen in series because one depends on the other’s output. Use concurrency when you want to combine data from multiple operations that can operate independently. Another important thing to note is that concurrency isn’t worth using if the process that’s running concurrently doesn’t take a lot of time. Concurrency isn’t free; many common in-memory algorithms are so fast that the overhead of passing values via concurrency overwhelms any potential time savings you’d gain by running concur‐ rent code in parallel. This is why concurrent operations are often used for I/O; read‐ ing or writing to a disk or network is thousands of times slower than all but the most complicated in-memory processes. If you are not sure if concurrency will help, first write your code serially and then write a benchmark to compare performance with a concurrent implementation. (See “Using Benchmarks” on page 393 for information on how to benchmark your code.) Let’s consider an example. Say you are writing a web service that calls three other web services. Your program sends data to two of those services, and then takes the results of those two calls and sends them to the third, returning the result. The entire process must take less than 50 milliseconds, or an error should be returned. This is a good use of concurrency, because there are parts of the code that need to perform I/O that can run without interacting with one another, there’s a part where the results are combined, and there’s a limit on how long the code needs to run. At the end of this chapter, you’ll see how to implement this code. 288

|

Chapter 12: Concurrency in Go

Goroutines The goroutine is the core concept in Go’s concurrency model. To understand gorou‐ tines, let’s define a couple of terms. The first is process. A process is an instance of a program that’s being run by a computer’s operating system. The operating system associates some resources, such as memory, with the process and makes sure that other processes can’t access them. A process is composed of one or more threads. A thread is a unit of execution that is given some time to run by the operating system. Threads within a process share access to resources. A CPU can execute instructions from one or more threads at the same time, depending on the number of cores. One of the jobs of an operating system is to schedule threads on the CPU to make sure that every process (and every thread within a process) gets a chance to run. Think of a goroutine as a lightweight thread, managed by the Go runtime. When a Go program starts, the Go runtime creates a number of threads and launches a single goroutine to run your program. All the goroutines created by your program, including the initial one, are assigned to these threads automatically by the Go run‐ time scheduler, just as the operating system schedules threads across CPU cores. This might seem like extra work, since the underlying operating system already includes a scheduler that manages threads and processes, but it has several benefits: • Goroutine creation is faster than thread creation, because you aren’t creating an operating system–level resource. • Goroutine initial stack sizes are smaller than thread stack sizes and can grow as needed. This makes goroutines more memory efficient. • Switching between goroutines is faster than switching between threads because it happens entirely within the process, avoiding operating system calls that are (relatively) slow. • The goroutine scheduler is able to optimize its decisions because it is part of the Go process. The scheduler works with the network poller, detecting when a goroutine can be unscheduled because it is blocking on I/O. It also integrates with the garbage collector, making sure that work is properly balanced across all the operating system threads assigned to your Go process. These advantages allow Go programs to spawn hundreds, thousands, even tens of thousands of simultaneous goroutines. If you try to launch thousands of threads in a language with native threading, your program will slow to a crawl.

Goroutines

|

289

If you are interested in learning more about how the scheduler does its work, watch the talk Kavya Joshi gave at GopherCon 2018 called “The Scheduler Saga”.

A goroutine is launched by placing the go keyword before a function invocation. Just as with any other function, you can pass it parameters to initialize its state. However, any values returned by the function are ignored. Any function can be launched as a goroutine. This is different from JavaScript, where a function runs asynchronously only if the author of the function declared it with the async keyword. However, it is customary in Go to launch goroutines with a closure that wraps business logic. The closure takes care of the concurrent bookkeeping. The following sample code demonstrates the concept: func process(val int) int { // do something with val } func processConcurrently(inVals []int) []int { // create the channels in := make(chan int, 5) out := make(chan int, 5) // launch processing goroutines for i := 0; i < 5; i++ { go func() { for val := range in { out