Generic pipelines using Docker: the DevOps guide to building reusable, platform agnostic CI/CD frameworks 9781484236543, 9781484236550, 1484236548

Citation preview

Generic Pipelines Using Docker The DevOps Guide to Building Reusable, Platform Agnostic CI/CD Frameworks — Brandon Atkinson Dallas Edwards

Generic Pipelines Using Docker The DevOps Guide to Building Reusable, Platform Agnostic CI/CD Frameworks

Brandon Atkinson Dallas Edwards

Generic Pipelines Using Docker Brandon Atkinson North Chesterfield, VA, USA

Dallas Edwards Midlothian, VA, USA

ISBN-13 (pbk): 978-1-4842-3654-3 https://doi.org/10.1007/978-1-4842-3655-0

ISBN-13 (electronic): 978-1-4842-3655-0

Library of Congress Control Number: 2018962365

Copyright © 2018 by Brandon Atkinson, Dallas Edwards This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Joan Murray Development Editor: Laura Berendson Coordinating Editor: Jill Balzano Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail [email protected], or visit www.apress.com/ rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book’s product page, located at www.apress.com/9781484236543. For more detailed information, please visit www.apress.com/source-code. Printed on acid-free paper

Table of Contents About the Authors�������������������������������������������������������������������������������vii About the Technical Reviewer�������������������������������������������������������������ix Acknowledgments�������������������������������������������������������������������������������xi Introduction���������������������������������������������������������������������������������������xiii Chapter 1: Recognizing You Are Stuck in the Past�������������������������������1 Monolithic Applications�����������������������������������������������������������������������������������������2 One Pipeline per Application���������������������������������������������������������������������������������5 Bad Actors�������������������������������������������������������������������������������������������������������7 Pipeline as Gatekeeper����������������������������������������������������������������������������������10 Agile Can’t Die; It Was Never Born����������������������������������������������������������������������12 Overview�������������������������������������������������������������������������������������������������������������14

Chapter 2: Setting the Stage for the Present��������������������������������������17 Microservices������������������������������������������������������������������������������������������������������17 Developer Best Practices Are Key�����������������������������������������������������������������19 Dedicated Pipeline per Language�����������������������������������������������������������������������21 Standards Are Key�����������������������������������������������������������������������������������������26 Taking Code Reuse a Step Further����������������������������������������������������������������28 Microservices and Shared Pipelines are More Agile�������������������������������������������31 Overview�������������������������������������������������������������������������������������������������������������32

iii

Table of Contents

Chapter 3: Getting it Right with Docker and Scripts���������������������������33 One Pipeline to Rule Them All�����������������������������������������������������������������������������34 Shell Scripts��������������������������������������������������������������������������������������������������������39 Configuration Files�����������������������������������������������������������������������������������������43 Docker at the Core����������������������������������������������������������������������������������������������47 Platform Agnostic������������������������������������������������������������������������������������������������56 Overview�������������������������������������������������������������������������������������������������������������61 Shell Scripts��������������������������������������������������������������������������������������������������61 Docker�����������������������������������������������������������������������������������������������������������62 Build Containers��������������������������������������������������������������������������������������������62

Chapter 4: A Practical Example����������������������������������������������������������63 An Overview of Our Applications�������������������������������������������������������������������������63 Spring Boot����������������������������������������������������������������������������������������������������64 ASP.NET Core Web API�����������������������������������������������������������������������������������65 Angular 5�������������������������������������������������������������������������������������������������������66 A Deep Dive into the Pipeline������������������������������������������������������������������������������67 The Pipeline Configuration File����������������������������������������������������������������������67 The Clone Stage��������������������������������������������������������������������������������������������71 The Build Stage���������������������������������������������������������������������������������������������72 The Test Stage�����������������������������������������������������������������������������������������������74 The Archive Stage������������������������������������������������������������������������������������������75 The Deploy Stage�������������������������������������������������������������������������������������������77 A Look at Our Build Containers����������������������������������������������������������������������78 Running the Pipeline�������������������������������������������������������������������������������������������81 Using the Command Line������������������������������������������������������������������������������82 Using IntelliJ IDEA CE������������������������������������������������������������������������������������83

iv

Table of Contents

Moving to the Cloud��������������������������������������������������������������������������������������������89 Moving the Pipeline to Travis CI���������������������������������������������������������������������89 Running the Pipeline in CircleCI��������������������������������������������������������������������96 Overview�����������������������������������������������������������������������������������������������������������104

Chapter 5: Moving Beyond the Basics����������������������������������������������105 Additional Stages and Steps�����������������������������������������������������������������������������105 Other Scripting Languages�������������������������������������������������������������������������������108 PowerShell��������������������������������������������������������������������������������������������������109 Python���������������������������������������������������������������������������������������������������������112 Custom Script Hooks����������������������������������������������������������������������������������������115 Pre- and Postpipeline Hooks�����������������������������������������������������������������������116 Stage Hooks������������������������������������������������������������������������������������������������119 Where to Go from Here�������������������������������������������������������������������������������������122

Index�������������������������������������������������������������������������������������������������125

v

About the Authors Brandon Atkinson is a software engineer with more than 14 years of industry experience encompassing analysis, design, development, and implementation of enterprise-level solutions. His passion is building scalable teams and enterprise architecture that can transform businesses and alleviate pain points. He has extensive experience in various technologies/methodologies, including Azure, AWS, .NET, DevOps, Cloud, JavaScript, Angular, Node.js, and more. Brandon lives in Richmond, Virginia, USA with his wife and two daughters. Dallas Edwards has more than ten years of experience as a software engineer. He thrives on creating solutions that are pragmatic, scale easily, and that are easy to test and maintain. His experience encompasses a wide range of expertise, including software development, iOS application development, and DevOps. Dallas lives in Richmond, Virginia, USA with his wife and is an avid scuba diver.

vii

About the Technical Reviewer Alex Fabian is a professional software engineer with experience building enterprisescale Java systems, APIs, and web applications. He has hands-­on experience implementing Docker-based deployment pipelines for microservices and has spent considerable time integrating with various offerings from Amazon Web Services. Alex has had the opportunity to work on large- and small-scale projects for Fortune 500 companies, technology startups, government contractors, and small organizations. He holds certifications from Amazon Web Services and a B.S. in Computer Science from The University of Virginia.

ix

Acknowledgments Brandon Atkinson Writing a book is very hard work, and very time consuming. I could not have done this without the support of my family, especially my amazing wife Jennie and my wonderful kids. I also want to thank Dallas for coming on this journey with me. I did my best to take over your life; you’re about to get it back! I also want to thank Chris Bowers for setting me on this DevOps path, and providing a lot of the vision that made this pattern come true. Dallas Edwards There’s no way this could’ve happened without the constant support and encouragement of my family. I am so grateful for the role each of you has played in my life. I’d like to thank Brandon for giving me this opportunity. I can’t wait to see what you drag me into next! Finally, to my wife Cierra: thank you so much for all of your advice and sticking with me through all the late nights. I already proved it.

xi

Introduction DevOps (a clipped compound of “development” and “operations”) is a software engineering culture and practice that aims at unifying software development (Dev) and software operation (Ops). The main characteristic of the DevOps movement is to strongly advocate automation and monitoring at all steps of software construction, from integration, testing, and releasing to deployment and infrastructure management. That definition pretty much sums up what most people think of when they hear the term DevOps. Not because it perfectly describes what it is, however; it leaves you more confused than before you asked the question “What is DevOps?”. DevOps comes in so many forms it can be hard to nail down exactly what it is. I’ve worked in a lot of different shops in my time and DevOps is always something different at each stop. For most it involves developers writing code that is checked into source control, which immediately kicks off a build pipeline that deploys your application. That pipeline will perform various steps that may include stages like building, testing, and deploying. From my experience, if you’re just doing these simple steps through automation, you’re way ahead of some. However, for many this is not enough to have your organization be considered as fully embracing DevOps. You may also want to include things like automated infrastructure creation, security scans, static code analysis, and more. In an ideal situation everything you do in a software shop is stored as code in source control. This includes your application code, infrastructure scripts, database scripts, networking setup, etc. With a push of a button everything your application needs to run can be created on the fly. For a lot shops, this is what it means to truly embrace DevOps.

xiii

Introduction

Some organizations are more mature than others when it comes to DevOps practices. In my experience you’re well on your way to maturity if you’re doing the following items: •

You deploy your application via an automated pipeline.

•

Once built and tested, application code binaries can be promoted via an automated pipeline.

I realize a lot of people will look at that list and exclaim “WHAT?!”. There are only two items, and your organization may be way beyond those. However, most are not and would see immense improvements by just doing these two things. If you’re one of the folks who thinks this list is crazy small, then count yourself lucky. I know many people who would kill for just these two items. Automated pipelines in my opinion are the lifeblood of good DevOps practices. They provide so many benefits to both the team of developers as well as the business that relies on their code. A well-crafted pipeline gives you a repeatable process for building, testing, and deploying your application. It can be used to create artifacts that, once built, are simply promoted, ensuring that what makes it to Production has been tested and vetted. Pipelines can also be a pain point for organizations as well. You may have multiple applications, each written in different languages, and each with their own finicky way of being built. It can be a nightmare at times jumping between technologies, debugging the various stages, and keeping the lights on. Luckily modern technology is helping us get around some of these issues. Technology like Docker has given us an opportunity to standardize our platforms. By utilizing Docker in your pipeline, you can give your developers some peace of mind that if they can build it, so can you. Combine this with cloud technology like Amazon ECS or Azure Container Service and now we can extend that piece of mind all the way

xiv

Introduction

to deployment. If it runs locally in your Docker daemon, it will run in the cloud. This even applies if you’re running an enterprise cloud platform like Nutanix or Dell; if the destination is a container orchestration service you’re golden. Now, consider that .NET Core is open source and can run on Linux, and you’ve pretty much covered all your bases. It’s a great time to be in technology! This book aims to show how you can use all this technology to simplify your pipelines and make them truly generic. Imagine a single pipeline that deploys all your code regardless of the tech stack it is written in. It’s not a dream; we’ll show you how.

Who This Book is For This book was written with the DevOps professional in mind who may be struggling with writing and maintaining multiple pipelines. However, it’s also for anyone in technology who is interested in learning about building generic pipelines. You don’t have to be a DevOps master to get a lot from this book; however, you will benefit more with experience in the following areas: •

Docker: Being familiar with Docker from creating Dockerfiles, building and running images, etc. We won’t get very deep into Docker, but a working knowledge is key.

•

Bash Scripts: Most of the examples in this book are written in Bash. You should have at least some minimal experience with scripts.

•

CI/CD Platform Experience: You’ll be much better off if you already have experience with a platform like Circle CI or Jenkins. However, throughout the book we’ll walk you through working with these platforms. xv

Introduction

As you can see, having some DevOps and pipeline experience will certainly benefit you as you read this book, but it’s not a hard requirement. If you’re a developer of any type, you should have no issues jumping right into the examples in this book. As stated, the future of DevOps and pipelines is everything is code. If you’re comfortable writing code, you’ll be fine.

Why We Wrote This Book This book is the culmination of a lot of hard work our team has done over the last year. We were tasked with creating a generic, reusable pipeline that any development team could utilize regardless of the language they were writing in. Our organization was going through a transformation from monolithic applications to microservices, and the old way of delivering code wasn’t going to cut it. After we decided on our plan of attack and began implementing, we realized we had something special on our hands. Talking with other groups around us, no one was tackling CI/CD pipelines in quite the same way. Most were still building a pipeline per application. Some were taking a step further and building tech stack–specific pipelines that could work with teams using the same language. However, no one was going the extra step and building a single pipeline for all. Now that we’ve been iterating on our code for a while we’ve seen some amazing results. Our development teams can quickly produce microservices and on-board them to the pipeline in a matter of minutes. By utilizing Docker locally on the developer’s machines and in the pipeline, we can ensure that the experience is extremely similar. Using Docker in the pipeline gives us full control over the process regardless of what the underlying platform has installed. We’ve seen this pattern take hold in our group and how it’s transformed the way we deliver code. This book is our effort to share what xvi

Introduction

we’ve learned and hopefully pass along some of that knowledge. If you come away from this book with even a few things you find useful, we’ll take that as a success. If you end up taking this pattern and applying it in your organization—even better. We’ve seen its successes and believe in it 100%!

What We Won’t Cover in This Book This book is meant to show you how to build generic pipelines utilizing Docker. We won’t be covering any CI/CD principles (except by reference), or talking about the benefits of DevOps and how it can transform your organization. We’re going to assume you are either already on-board with these concepts or at least know about them in some way. There are many well-written books out there that extoll the benefits of proper continuous integration and delivery. We don’t need to hash all that out here, nor could we do a better job than those before us. We won’t be covering Agile principles or how to write better software. We won’t argue which CI/CD platform is better or which language produces the best code. While you’re in this book we’re going to focus on why monolithic applications and their pipelines are bad. How microservices and smaller pipelines are good. We’ll follow that up with why generic reusable pipelines are even better. Then we’ll wrap all that up with a lot of examples. So, if you want or need a deep dive into the benefits of DevOps and proper CI/CD practices, search out another book. If you want to build leaner more efficient pipelines, read on.

What You’ll Need Before moving on we’ll need to cover the items you’ll need to work with the examples in this book. The first few chapters deal with why you want to look at generic pipelines, and later chapters present implementing an example. So, you will not immediately need all of these. If you like, you can xvii

Introduction

read ahead and deal with these later. If you prefer to be prepared ahead of time, here is what you’ll need: •

Mac or Windows 10

•

Docker for Mac/Windows

•

IntelliJ IDEA (or another IDE)

•

CI/CD Platform

•

Docker Hub Account

•

GitHub Account

Let’s dig into each of these and walk through installing, signing up, and configuring each.

Mac or Windows 10 At first glance you may be thinking “Why include a section on this? Of course we have one of these.” You would not be entirely wrong for thinking this, but there is a good reason. If you’re on a Mac you’re all set, you are set up well for what’s about to come. If you’re on a Windows machine, we need to talk. We highly recommend using Windows 10 if your only option is Windows. The main reason for this is the Docker experience. In Windows 10, Docker is a first-class citizen and the experience closely matches that on a Mac. In addition, Windows 10 now supports the Linux subsystem, which allows you to use Bash natively. We also realize not everyone can be on the latest hardware and operating systems. You should be able to complete all the examples in this book with any operating system that you can install Docker on. However, this means you may have to jump through more hoops to get everything working correctly. As we move forward, we will assume you are either on a Mac or Windows 10. xviii

Introduction

Caution Realistically, Windows 7 is about as low as you can go with an earlier Windows release. While it is possible to install Docker on Windows 7, some of the other items we’ll use in this book, like Bash, will be outside your reach natively on that platform. You can still utilize them; it will just need to be within the context of a Docker container or virtual machine.

D  ocker for Mac/Windows It goes without saying that you’ll need Docker for this book—it’s in the title! Luckily Docker provides a free Community Edition for both Mac and Windows 10 users. Navigate to www.docker.com/docker-mac or www.docker.com/docker-windows as shown in Figure I-1.

Figure I-1.  Download page from docker.com Click the “Download from Docker Store” button to be taken to the official Docker Store download page as shown in Figure I-2.

xix

Introduction

Figure I-2.  Docker Community Edition download page for Mac. The Windows download page is almost identical. Click the “Get Docker” link to download Docker. Follow the instructions to install Docker on your machine. Once installed, you will see the Docker icon in the top left of your screen as shown in Figure I-3. Clicking the icon will open the menu.

Figure I-3.  Docker icon and menu after a successful installation

xx

Introduction

On Windows 10 the icon will appear in the bottom right tray. You can also verify the installation by running a docker -v command in the terminal, as shown in Figure I-4. If you’re on Windows you can run this command from PowerShell.

Figure I-4.  Verifying your Docker installation via the terminal

Note  When downloading Docker, make note of the supported versions for your operating system. As of this writing, Mac OS Yosemite 10.10.3 or above and Windows 10 Professional or Enterprise 64-bit are supported. If you are using an earlier release of your OS, you can opt to install Docker Toolbox ­(https://docs. docker.com/toolbox/overview/). However, please be aware that Toolbox is not a full-fledged Docker experience.

I ntelliJ IDEA In this book we’ll be using JetBrains IntelliJ IDEA as our IDE of choice. However, any IDE can be used and if you have one already installed and are comfortable with it, feel to use that. If you want to use IntelliJ, then continue reading; otherwise jump to the next section. xxi

Introduction

Note Any screenshots taken of an IDE in this book will be of IntelliJ on Mac. However, there is nothing stopping you from using a different IDE. All code examples will work the same regardless of what you are using. JetBrains offers a free Community Edition of IntelliJ that can be downloaded from the following URL: www.jetbrains.com/idea/ download/. There are two available downloads: a trial edition and the Community Edition as shown in Figure I-5.

Figure I-5.  IntelliJ download page on the JetBrains website The Community Edition has more than enough features to get you through this book and beyond. Download this edition, and follow the normal installation. Once installed and launched, you’ll be greeted with

xxii

Introduction

the “Welcome to IntelliJ IDEA” screen as shown in Figure I-6. If you see this screen, your install was successful and you can move on to the next section.

Figure I-6.  The welcome screen upon launching IntelliJ for the first time

C  I/CD Platform To showcase how you can build platform agnostic pipelines, we’ll need some platforms to work with. We’ll work with two of the more popular platforms on the market: Circle CI and Travis CI. Outside of being popular options for building pipelines, they also provide free subscriptions via Software as a Service (SaaS) offerings. This was important so you could work through the examples without spending any money. Also, being SaaS,

xxiii

Introduction

you don’t need to worry about downloading and installing software, which is extremely nice! Another reason we chose these two platforms is how different they are in their UIs and setup. Given that this book will demonstrate a platform agnostic approach to building pipelines, these two vendors showcase this quite well. We won’t cover setup here; that will come later in Chapter 4. For now we recommend visiting their sites and poking around a little bit. We’ll come back to these soon!

C  ircle CI Circle CI is a great option as it provides a totally free option when signing up. To learn more, navigate to the site located at the following URL: https://circleci.com/.

T ravis CI Travis CI is another great option with a free tier. You can sign up and learn more here: https://travis-ci.org/.

O  ther Options If you’ve been working in DevOps for a while, you know there are many platforms out there to choose from. If you’re new to DevOps, have no fear. If you don’t like an offering from a vendor, you’ll have plenty of options to find something different. What we present in this book will work with any platform that offers running commands inside a Docker container. If you don’t care for the platforms we’re using in this book, there is nothing stopping you from using something different. These vendors were chosen because they offer

xxiv

Introduction

a great product at a great price: Free! Also, these options are completely managed with SaaS solutions. This frees you up from having to deal with installations. However, if you like, here are some other options you can also use: •

JetBrains Teamcity TestDrive (www.jetbrains.com/ teamcity/): JetBrains offers a 60-day trial of Teamcity TestDrive in the cloud. However, TestDrive is meant to be a trial in the web, and not meant for long-term use. At the end of the trail you are expected to download the product if you like it.

•

CloudBees Jenkins (www.cloudbees.com/): Jenkins is a powerhouse in the DevOps world, so we’d be remiss to leave them out of this book. CloudBees offers a seven-­ day trial of their Enterprise product, which is hosted. There are some drawbacks. You need a company-­ provided email address to sign up, and did I mention the seven days?

Docker Hub Account You’ll also need an account on Docker Hub. In the examples you’ll be building custom Docker images to perform your build and unit test stages in, so we’ll need somewhere to store those. Docker provides free accounts on Docker Hub. To get started, navigate to https://hub.docker.com/ as shown in Figure I-7.

xxv

Introduction

Figure I-7.  Docker Hub landing page Fill out the form on the landing page and click the “Sign Up” page and you’ll be all set. Once you’re logged in you can create new repositories via the “Create Repository” button, as shown in Figure I-8. You’ll need to create a new repository before pushing any images to Hub.

Figure I-8.  Docker Hub page after you’ve logged in

G  itHub Account Finally, you’ll need a GitHub account. As you noticed in the signup for Circle CI and VSTS, you’ll eventually be connecting a GitHub account to them. To create an account, navigate to https://github.com/ and fill in the form located on the homepage, as seen in Figure I-9.

xxvi

Introduction

Figure I-9.  Sign up form on the GitHub landing page Once you sign up, you’ll be taken to the main landing page where you can create new projects. For now you don’t need to do anything else; you’re all setup for the examples to come.

Overview This introduction has gotten you prepared for the code examples that come later in the book. You should now have all the software installed and have created all the accounts needed. While actual coding and deployments come later in the book, I would encourage you to poke around and explore Circle CI and VSTS ahead of time. There are a lot of options available in each, and it would be good to build even a simple “Hello World” deployment before jumping into examples. Next, we’ll begin to look at how traditional monolithic applications and pipelines are holding you back. We’ll explore why these came to be and how toxic they are for rapid delivery of applications. xxvii

CHAPTER 1

Recognizing You Are Stuck in the Past It can seem sometimes that the world has lost its mind. You’re plugging away writing code, building new features for your business users, and deploying code like a pro without a care in the world. Then along comes a new pattern or a blogger talking about new ways to write software and efficient means to deploy code. They talk about how they’ve observed that the way your organization does things is just plain wrong, if not barbaric. You might take it personally or get offended, and rightly so! Who are they to tell you that what you’re doing is wrong; “we’ve always done it this way.” Recognizing you are stuck in the past can be difficult. Some of us refuse to acknowledge it or are comfortable in our ways of doing things, while others of us use it as an excuse to get upset or vent frustration about a project or the software we’re building. Oftentimes, especially when you work for a larger company, you have no control over the situation you’re in. I’ve worked for companies that have had feature requests pile up, which prevented any discussions of refactoring an application. When the business needs new features to bring in more money from demanding clients, it’s tough to argue any other way. In this chapter we’ll look at how some organizations are truly stuck in the past. We’ll explore why applications tend to evolve this way, or are born that way. We’ll also discuss how these applications can lead to stress and © Brandon Atkinson, Dallas Edwards 2018 B. Atkinson and D. Edwards, Generic Pipelines Using Docker, https://doi.org/10.1007/978-1-4842-3655-0_1

1

Chapter 1

Recognizing You Are Stuck in the Past

discontent among teams, and some simple things you can do to mitigate stress and improve morale. This is not meant to be doom and gloom, but rather to help bring to light that things can be better.

Monolithic Applications Monolith—something having a uniform, massive, redoubtable, or inflexible quality or character. ("monolith". Dictionary.com Unabridged. Random House, Inc. Jan 30, 2018. ) I think this sums up quite nicely some of the monolithic applications I have worked on in the past. Massive and inflexible are great ways to describe these applications. If you’re not a software developer by trade and have never worked on an application like this, count yourself lucky. Those of us who have understand these definitions and how accurate they are. A traditional monolithic application looks like the design as shown in Figure 1-1.

Figure 1-1.  Traditional monolithic application design 2

Chapter 1

Recognizing You Are Stuck in the Past

Info The word redoubtable is fitting as well, defined as “that is to be feared; formidable.” (“redoubtable”. Dictionary.com Unabridged. Random House, Inc. Jan 30, 2018. ) Here we have a huge application that connects to a database and has multiple configurations as it moves through various deployment environments. Sometimes the configuration isn’t even separate, it’s baked into the source code. In this scenario the user interface (UI) lives with, and is usually packaged with, the application code. Applications like this are usually built on MVC frameworks that bring front-end rendering and server-side code all in one neat package. These frameworks are great for rapid application development, but usually result in poor decisions in code structure, separation of concerns, and flow. Some teams will logically make the next jump to an architecture that breaks the UI away from server-side code, much like the design shown in Figure 1-2.

Figure 1-2.  Monolith that has been split into two layers 3

Chapter 1

Recognizing You Are Stuck in the Past

While this is certainly a step in the right direction, it can introduce much more complexity. You’ve now effectively doubled up your deployment concerns. Even though the UI and server-side code are split from each other, they cannot be deployed independently. You’ll now have two deployments of large code bases that must be coordinated. You may also now need to juggle configuration files: one for the UI and one for the server-side code. In an ideal world you’re deploying the application into multiple environments that may include: Development, QA, UAT, Staging, and Production. At each stage the configuration must change to point to the correct database. Also, presumably the database must be updated with any data or schema changes to support the code. In almost all scenarios changes in the application code or database are tightly coupled. So, a change in either concern forces a complete deployment of all the concerns. This is a lot of work if all you need to do is add a link to a page! Now imagine if you have two or three of these applications. The deployment process becomes so much more complicated. It would not be unheard of that these applications have dependencies on each other as well. Now every deployment must be coordinated across multiple applications, or at the very least, all applications must be tested even if only one is being deployed. This is a sure-fire way to drive your IT teams insane. Nothing about this process makes you feel good. Monoliths become a nightmare to maintain, debug, or add new features to. From a developer point of view, you never want to touch the code, especially in business-critical areas. As a system team (DevOps or Sys Admins), you dread deployments, as they are error prone and take a lot of time. If you’re lucky to have a CI/CD pipeline in place to support your application, it may not be much better on the other side of the fence.

4

Chapter 1

Recognizing You Are Stuck in the Past

One Pipeline per Application A lot of monolithic applications are still deployed manually, even in 2018! I’ve worked with several clients who had a system team that consisted of DBAs and Sys Admins. This was the de facto DevOps team, as they were responsible for deploying application code. The deployment process involved the developers manually building and packaging the code, and placing it on a shared drive or FTP folder on the network. A Sys Admin would then copy the compiled code and paste it on the production servers. You might be reading this and thinking to yourself there is no way this is still going on. How can organizations deploy applications like this? Well, these practices exist for many of the same reasons as monoliths. Here’s the short list: •

We’ve always done it this way.

•

The deployment can’t be automated.

•

The business won’t give us time to build it out.

•

We don’t have DevOps expertise.

The list can go on and on, and some of these may be valid; however, most simply reflect fear of doing something new or moving away from a method that works. If they are valid, they most likely need to be addressed and overcome. If your monolith is a commercial off-the-shelf product, you may not be able to automate the deployment. The business may have other very real and critical needs that need immediate attention. You may not want to take on building a pipeline if you don’t have the expertise, or the time to dedicate to it.

5

Chapter 1

Recognizing You Are Stuck in the Past

Note  Commercial off-the-shelf (COTS) applications can pose a real challenge. While there are many well-written and maintained COTS products, some are a real nightmare. A quality COTS application can be easily deployed via a pipeline. Others require custom deployment processes that make using a pipeline nearly impossible. If you’re lucky enough to have a pipeline for your monolith, it can at times add to your headaches. In almost all scenarios, you will build one that is tightly coupled to your application. In fact, this practice has nothing to do with monoliths per se. Most CI/CD pipelines and processes are built around a single application. They are custom built to do everything needed by the application and can include things like: •

Building the application

•

Automated tests

•

Running security or static code analysis

•

Provisioning/configuring infrastructure

•

Publishing artifacts

•

Injecting configuration

•

Deploying the application

These pipelines do everything they should to get the application where it needs to go, and the state it needs to be in. You may even be able to reuse some sections of the pipeline elsewhere, like security or static code scans. However, in most cases the pipeline is very coupled to the specific needs of the application being deployed. For instance, if you’re provisioning infrastructure, configuring it, and adding application configuration, the pipeline becomes very tightly coupled to the application.

6

Chapter 1

Recognizing You Are Stuck in the Past

If you begin adding more applications in your organization, your overhead starts to grow quickly. You now need to build and maintain multiple pipelines, each with their own eccentricities. If the platform you build your CI/CD efforts on top of needs to be updated or a new version is released, you may be in the position of rebuilding all your pipelines. This is not an ideal situation to be in, especially if you don’t have the staff to handle it.

Bad Actors Having a single pipeline per application can introduce some bad behavior. Since the pipeline is tied directly to a single codebase, it’s not uncommon to receive requests for it to be modified to do nonstandard things. After all, it’s a pipeline dedicated to a single application; it must support the needs of the application. If you make a change to it, it only affects this one application. However, these ad hoc changes can quickly put you in a very fragile spot. Take, for instance, the flow as shown in Figure 1-3.

Figure 1-3.  Source control with multiple branches, hooked up to a deployment pipeline 7

Chapter 1

Recognizing You Are Stuck in the Past

In this example we have application code stored in source control with multiple branches. The “master” branch is tied to the deployment pipeline. Any commits to “master” will trigger a deployment of the code. Developers can maintain feature or development branches that will eventually get merged into “master.” In this scenario “master” is the source of truth, and the only branch that ever gets deployed. This follows proper CI/CD principles and is a very good practice. As is often the case with larger code bases that have a dedicated pipeline, bad practices begin to emerge. It’s not uncommon for a feature branch to become so large that it becomes difficult to easily merge it back into “master.” Developers work hard on this branch for an upcoming release, and the clock is ticking. Eventually the request will come down to have the pipeline reconfigured so it points to a feature branch, as shown in Figure 1-4.

Figure 1-4.  Source control with multiple branches, hooked up to a deployment pipeline on a nonmaster branch

8

Chapter 1

Recognizing You Are Stuck in the Past

Now we have our pipeline deploying off a feature branch, which is not good practice. If you have multiple teams working on several features, which branch is now the source of truth? The team is very quickly put into a place where “Feature Branch #1,” in this example, is the de facto “master” branch. This very quickly gets out of hand, especially if another team needs to begin working on a new feature at this very moment. They will be forced to create a feature branch off a feature branch! Another common bad practice is a request to turn off certain stages in the pipeline, as shown in Figure 1-5.

Figure 1-5.  Pipeline that has been configured to bypass unit tests Now the pipeline is deploying code that is in a feature branch, and the testing stage has been bypassed. If your pipeline is configured like this, it should raise alarm bells—not for the DevOps team, but for the teams you support. These requests can come over very quickly and are usually the result of tight deadlines, or needing to provide a hotfix for production. Whatever the reason, they introduce risk in the process of delivering 9

Chapter 1

Recognizing You Are Stuck in the Past

quality software. The result most often is that these changes persist past the initial implementation, as technical debt has a way of being created but never resolved.

Info These examples are from real-world scenarios I’ve encountered throughout my career. It takes a village to deliver quality software, and the pipeline needs just as much attention at times as your application.

Pipeline as Gatekeeper The CI/CD pipeline plays a huge role in the software delivery lifecycle. Not only is it responsible for delivering your application to an environment in an automated and predictable way, it also plays a role in the quality of that software. Quality pipelines include various stages to check not only the quality of the code but also if it has any security or performance issues. Typical stages include:

10

•

Build: This stage does exactly what it says; it builds the application from source code. The binaries created here are what is deployed later.

•

Test: This stage will run unit tests against the code. This is your first quality check and it ensures any changes made by the development team did not break expected functionality.

•

Static Code/Security Analysis: This is a scan of the source files for the application. This scan will look for vulnerabilities, technical debt, and may verify code coverage. This stage could be broken into multiple steps, given your choice of tools to perform the scanning.

Chapter 1

Recognizing You Are Stuck in the Past

•

Storing Artifacts: A good process takes the binaries created in the Build stage and saves them for later deployments. This ensures that code that was tested and successfully deployed can be promoted later with confidence it passed all your checks.

•

Deployment: A simple (hopefully) deployment of the binaries to the specified environment, i.e., Dev, QA, Production, etc.

•

End to End/Performance Tests: Once the code is deployed, you may run automated tests to ensure it performs properly. This can be via a browser or hitting API endpoints. This stage may also be broken up into multiple stages, depending on your needs.

As you can see in this scenario, a CI/CD pipeline can do a lot of work to ensure the code being delivered is a quality product. Bypassing some of these steps is just impossible; you can’t deploy if you haven’t built the application. However, some could absolutely be skipped, but shouldn’t. Teams may find themselves up against tight timelines and realize that their code coverage is beginning to slip. A request may come down to bypass the static code analysis stage. They may import a new code library that has a security vulnerability in it. Rather than address it, another request is received to bypass another stage. Soon you’re left with the absolute barebones pipeline.

Note There is an argument to be made that the CI/CD pipeline is not the place to enforce good behaviors. It’s really the responsibility of the development teams to ensure the code is properly tested and secure. However, there is no way to escape the fact that the pipeline is often where these checks are applied.

11

Chapter 1

Recognizing You Are Stuck in the Past

This behavior is tied directly to the fact that the pipeline serves a single application. The team believes it’s in their best interest to ignore some best practices, as they are being pressured to deliver faster. However, if the pipeline is not there as a proper check on what’s being deployed, the result could be much worse.

Agile Can’t Die; It Was Never Born CI/CD pipelines are the perfect complement to any team using agile methodologies. In an ideal situation development teams are building features that are committed to source control often. Each commit kicks off a build via the pipeline and your entire test suite is executed. The code is deployed to a lower environment for user acceptance testing, and artifacts are created for later promotion. Once the feature is complete, or maybe the sprint, the new feature is deployed to production and your users are happy. Even if you’re not using agile, these practices still make sense. You want your code built and tested as often as possible. Performing deployments often will also highlight any problems long before you get to push to production. The problem you will most often face is an extremely long lead time to deployment. With a typical monolithic application, the time it takes to deploy from the moment work begins on a new feature can be weeks or months, as shown in Figure 1-6.

12

Chapter 1

Recognizing You Are Stuck in the Past

Figure 1-6.  Feature development timeline on a typical monolithic application In this example it’s over a month from the time development begins on a new feature before it’s deployed to production. This cycle time introduces a lot of risk from many different angles: 1. Developers: This is one of the main reasons that long-living feature branches exist. Your “master” branch is pristine and you’re hesitant to mess with it. However, the long cycle time means your feature branch must also live that long. If multiple features are being developed concurrently, by the time you need to merge back into “master” it becomes a real problem.

13

Chapter 1

Recognizing You Are Stuck in the Past

2. Testers: In this scenario, testers don’t see the new features for quite some time. In some cases they don’t see compete functionality until the code is ready to go to production. This introduces even more risk if a bug is found. 3. DevOps: One of the selling points of pipelines is an automated process that is repeatable; you know what you’re getting every time you request a new build. If you’re only deploying once a month, that process is not performed enough to become routine. If you’re doing manual deployments, this means the subtle nuances that come with that process are forgotten. Each deployment then becomes as if it’s the first time you’re doing it. The longer the cycle time to develop and deploy a feature, the more out of sync your feature branch becomes with a “master” branch. In many scenarios changes are being introduced into the “master” branch during this time. UI updates, bug fixes, or other features may get merged. This puts you in a position of merging down from “master” into your feature branch, then merging back up. In either direction you’ll most likely encounter conflicts that need to be resolved.

Overview In this chapter we looked at how a monolithic application with a dedicated pipeline can introduce risk for your deployments. This combination can lead to bad behaviors that often result in cutting corners to meet deadlines. However, it’s not all doom and gloom. You can absolutely have a dedicated pipeline that performs all the tasks needed and runs smoothly. This quite

14

Chapter 1

Recognizing You Are Stuck in the Past

often requires strong leaders who are willing to say no to a risky request. The reality is that if the business needs software out the door on a certain date, you do what is required. In the next chapter we’ll look at how microservices can help ease some of these concerns. We’ll explore how these services allow you to break away from a dedicated pipeline, into a more standardized approach. You’ll begin to see how we can look at the pipeline in more generic ways and get more from your deployments.

15

CHAPTER 2

Setting the Stage for the Present When you sat down with this book, you were probably fully caffeinated and ready to dive into learning about the world of generic pipelines. You made it through the introduction and rolled up your sleeves for Chapter 1 when bam, you were served with a healthy dose of “this is how it is.” Sorry to rain on your parade, but a lot of companies still work that way. Maybe you are beyond that, on a journey of exploring microservices and how they can improve writing and deploying applications. Alternatively, maybe you work in a shop that is already developing microservices, but struggle with how to manage all the pipelines. It is all very individual, but the good news is we’re here to help. In this chapter we’ll begin to explore how you can simplify code deployments using microservices. We’ll explore the beginnings of a truly generic pipeline. You’ll see how this can improve your agile maturity as well, by lowering risk and delivery time.

M  icroservices Microservices represent the true antithesis to monolithic applications. Small and lightweight, these services usually provide specific and very limited functionality. This allows them to focus on singular tasks, which © Brandon Atkinson, Dallas Edwards 2018 B. Atkinson and D. Edwards, Generic Pipelines Using Docker, https://doi.org/10.1007/978-1-4842-3655-0_2

17

Chapter 2

Setting the Stage for the Present

makes developing them fast, debugging issues easier, and deployments nimbler. A true microservice can be deployed independently of other services and is self-contained. A typical microservice architecture is shown in Figure 2-1.

Figure 2-1.  Multiple microservices with dedicated databases In this example, each microservice has its own database. This type of architecture allows developers to easily make changes to their services without the need to check with other teams. The service is the source of truth for the data and no other service should manipulate it. If another service needs to get to this data, an API is exposed for that purpose. This way if the developers don’t change the API, changes can move forward that don’t break other services that consume it. This can also be accomplished with versioned endpoints in the API. In this architecture, teams can move as fast as they want to and can deploy even faster. Unlike in the monolithic application, which must be deployed as a single unit, microservices can be deployed at different times. This gives teams the flexibility to work on multiple features at the same time, without worrying about stepping on each other’s check-ins. Each

18

Chapter 2

Setting the Stage for the Present

microservice is in its own repository, so teams are safe to make changes as needed. Adopting microservices is not a magic bullet, however. Teams will need to ensure they use proper API versioning and dependency management.

Note  While microservices can make developers and DevOps lives’ easier, they also come with baggage. Usually you end up with a lot of services, since they focus on more singular workloads. This increases your deployments, and the complexity does not necessarily go away. Microservices are not a silver bullet, but they do provide many benefits, which we’ll explore in detail.

Developer Best Practices Are Key One of the tenants of continuous delivery is that teams produce software, or new features, in short cycles. Microservices provide a great way to do this. Since teams can focus solely on their service and adding new features to it, they can move much faster. One of the benefits to this is smaller and quicker check-ins to the “master” branch. In most cases your deployment pipeline is configured to react to commits to the “master” branch in your source control. A commit will trigger a build, which creates an artifact that can be deployed. Microservices give development teams an opportunity to practice true CI/CD patterns. By this I mean each commit to “master” will trigger a pipeline execution. The pipeline will build, test, and deploy the application each time, all while producing an artifact that can be promoted to Production. Each commit produces a potential production release.

19

Chapter 2

Setting the Stage for the Present

Depending on how mature your organization is, this may seem like madness! Every commit produces a production-ready artifact?! If you’re new to microservices, this does sound strange—and even more so if you don’t have a sophisticated pipeline. However, consider the following scenario. You are a development team working in agile. You have two-week sprints and plan to produce a feature during each sprint. You have a CI/CD pipeline that performs the following steps: •

Build

•

Unit Tests

•

Security Scan

•

Static Code Analysis

•

End to End/Integration Tests

•

Artifact Creation

•

Deployment

•

Performance Tests

In this scenario when your team has finished development and tested locally, they can commit the new feature to the “master” branch in source control. The commit will trigger the pipeline, where a full suite of tests is run to ensure the code is production ready. At the end you have an artifact of the application you can now promote to production. Microservices allow this type of rapid deployment. While microservices are the gateway to faster release cycles, it takes much more to truly achieve this goal. This architecture gives you isolation from other teams and data stores, which allows for more rapid development. Developers need to ensure their application code is properly unit tested. End to end tests must be written so user acceptance testing

20

Chapter 2

Setting the Stage for the Present

can be automated. The same goes for performance tests. Security scans and static code analysis is usually performed by third-party tools, but developers must review the results and potentially update code based on those results. A great pipeline can get your code deployed in a matter of minutes, fully scanned and tested. However, it’s a team effort to build ­production-­ ready code. Simply developing the application is no longer enough. Proper testing and scanning must be in place to ensure the application being deployed works and is safe for your business.

Note There is debate about how small or large a microservice should be. Should it cover all the possible needs of a certain business process, for instance “Billing”? Or, should it be much more granular, for instance one for “Send Bill” and another for “Process Payment”. At the end of the day, this is a business decision. However, a good rule of thumb is if you can’t re-build it in a sprint (roughly 2-3 weeks), then it’s too big.

Dedicated Pipeline per Language Microservices give us a better way of delivering application code. In the previous chapter we looked at how a monolithic application has a dedicated pipeline for the application itself. Let’s review what the application looked like, as shown in Figure 2-2.

21

Chapter 2

Setting the Stage for the Present

Figure 2-2.  Monolith application While this application is a monolith, it would most likely have three pipelines to deploy it: one for the server-side application, one for the UI, and one for the database. Even though you have split the deployment into three components, they are all still tied to a single application. There is little to no reuse of code in those pipelines. Microservices provide an opportunity to improve this. Most business applications will have some sort of UI. Let’s look at an example business application that uses microservices, as shown in Figure 2-3.

22

Chapter 2

Setting the Stage for the Present

Figure 2-3.  Application that uses microservices with a UI In this scenario we still have a UI that users interact with, but now the business logic is separated out into multiple microservices. As more functionality is added, more microservices are added as well. Following the monolithic approach, we’d build a pipeline per UI and/or microservice. This may be OK if you only plan on having a few services, but what happens if your application and business take off and you end up with a dozen microservices. What about 20, 50, or 100? You may think those numbers are high, but you’ll be surprised at how fast adoption grows once you start down the microservices path.

Caution  Don’t take for granted how fast microservice adoption can take place or how many services you may end up with. Netflix runs hundreds of microservices to power its operation, reportedly over 500. You can be assured they do not have dedicated pipelines for each service.

23

Chapter 2

Setting the Stage for the Present

Microservices allow for a more generic approach to building your pipelines. It’s not uncommon for an application built with microservices to use multiple languages such as Angular for the front end, .NET Core for services interacting with SQL Server, and perhaps Golang for orchestration services. Figure 2-3 showed an application that had a UI powered by three microservices. Figure 2-4 expands on this with the addition of languages.

Figure 2-4.  Application that uses microservices with a UI, with languages added for reference In this example our UI is Angular, but it could be React or something similar. We have a .NET Core service and two Golang services. The languages here are not important at all. You can easily substitute Java for .NET, or Python for Golang. Whatever language is prominent in your organization, you can count on microservices written in it, plus at least one more. One of the benefits microservices provide is the ability to choose the language that is right for the job. As mentioned, you may want to use .NET to interact with SQL Server due to its frameworks custom built for

24

Chapter 2

Setting the Stage for the Present

that purpose. You may choose Golang as an orchestration service due to its speed and small footprint. You may choose an entirely different language simply because you have a shop full of developers who already know it. The list goes on and on. However, as you build your pipelines you must be prepared for a new language popping up at any time. Now that we are looking at applications built with one or more languages, we can start to look at the pipeline as being language specific rather than application specific. This is beneficial for a variety of reasons, including: •

Faster on-boarding: If your pipeline is built for a language rather than a specific application, others can use it if they write their applications in the same language. Teams can on-board to the pipeline faster and begin providing business value much sooner.

•

Reuse of code: You’re not spending your time writing custom code for each application. Your pipeline code is reused across multiple applications.

•

Ease of maintenance: If a new feature is required of the pipeline or you need to perform a bug fix, everyone gets those changes. Imagine you have an Angular 4 pipeline and Angular 5 is released. Simply update your Angular pipeline, and all UIs benefit at the same time.

Language-specific pipelines provide a lot of code reuse, which will make your life easier. This becomes very apparent once you build out a language-specific pipeline and teams begin to on-board. In the monolith world, a team would build an application and put in a request for a new pipeline. With language-specific pipelines built for microservices, you can build it once, then simply have teams utilize it. Figure 2-5 illustrates this concept.

25

Chapter 2

Setting the Stage for the Present

Figure 2-5.  Multiple services and UIs utilizing language-specific pipelines Now you’re no longer writing and rewriting code for your pipelines. Teams using the same language will also use the same pipeline. However, this can only be achieved with proper standards in place. Everyone must agree on a certain baseline set of rules and patterns that the pipeline will follow. You can’t have shared pipelines if everyone wants to do something different.

Note Examples that show .NET or .NET Core are referencing .NET Core running on Linux. One of the benefits of shared pipelines is they all run in Linux environments, and usually the same environment. Not jumping between Windows and Linux is a true benefit here, and you should take advantage of it.

Standards Are Key We’ll be exploring standards much more in depth later in the book as we begin to take shared pipelines even further. However, it’s an important area that we’ll look at now because it’s crucial to making these pipelines work. We’ll scratch the surface now, and in future chapters we’ll really dig in. 26

Chapter 2

Setting the Stage for the Present

In the previous section we talked about the need for consensus on how things work in the pipeline. For a shared pipeline to function correctly for all, a few key standards must be defined: •

How you build

•

How you test

•

How you deploy

Some of these may seem trivial, but once you begin working with multiple teams you’ll find folks have their own ways of doing things. It’s important that the pipeline performs the exact same commands for all applications for a specific language. Take .NET Core as an example: you may settle on the build command shown in Listing 2-1.

Listing 2-1.  .NET Core Build Command dotnet build -c Release -o /build-output This command will build the application in the current directory in Release (Production) mode and store the binaries in an output folder called “build-output.” Listing 2-2 shows an example test command.

Listing 2-2.  .Net Core Test Command dotnet test -c Release -o /build-output This command will unit test the application in the current directory in Release (Production) mode and look for binaries in the folder named “build-output.” Both commands are perfectly fine and would run without any issues. However, what if you have a development team that wants to run a dotnet build command, but with different arguments? Or you have a request not to build in “Release” mode, but rather in “Debug” mode? If you accommodate each team’s request, your pipeline will need to react differently based on the code it is building and testing. If you’re dealing with dozens of services, this would not be sustainable. 27

Chapter 2

Setting the Stage for the Present

The goal of a shared pipeline is to greatly decrease cycle time and release code faster. To do this the pipeline must be opinionated, in that it runs certain commands and only those commands. If a team wishes to deploy their application with your pipeline, they must play by the rules. At first glance this does not seem like a huge ask of your development teams. However, you may find that opinions differ on the best way to build and test applications. People may have certain tools or patterns they prefer over others. The key is getting a standard in place and sticking to it. The best way to accomplish this is to interview your developers. Find out exactly which commands they run locally to build and test. In most cases you’ll find the commands are the same or differ only slightly. Once you have this information, you can work with the teams to settle on the standard commands the pipeline will run. After that, all teams must agree to use these commands going forward. This is the easy part, as you enforce those via the pipeline. If the application cannot be built and tested using the standard commands, it won’t be deployed.

Taking Code Reuse a Step Further Once you have a language-specific pipeline in place, you may notice that some of your steps are identical. These tend to be tasks dedicated to various things like security scans or static code analysis. Submitting code for scans usually involves zipping up the source code or binaries, and uploading them to a service that performs the scan. There are many commercial platforms and software packages that perform static code and security analysis. They usually work with a long list of languages. What this means for your pipeline is a generic step that any language can use. Figure 2-6 illustrates a shared step across all your pipelines.

28

Chapter 2

Setting the Stage for the Present

Figure 2-6.  Code reuse across common areas As you can see, not only are we pushing more applications through fewer pipelines, now we’re beginning to reuse code across pipelines as well. This illustration should begin to solidify how shared pipelines can reduce your workload. This also illustrates how generic steps can be utilized anywhere you need them. Now, let’s assume that your pipeline is using static code analysis for code quality. Let’s also assume the business has decided to utilize containers and the cloud for deployments. Your pipeline can be further simplified, as shown in Figure 2-7. With these changes you can share even more code across multiple pipelines. Uploading code for a static code analysis and deploying to the cloud are not trivial matters. However, the underlying mechanics don’t change very much between applications. Some simple configuration data is all that is needed for each of these steps to operate properly.

29

Chapter 2

Setting the Stage for the Present

Figure 2-7.  Pipeline utilizing shared steps for static code analysis and cloud deploy In this example half of the pipeline is now shared code. The only steps that are not generic are the ones tied directly to a language like build and unit test. As your development practices begin to mature, you’ll find additional steps that can be shared, like performance testing, artifact storage, logging, etc. The idea with this pattern is to make steps as generic as possible. Most CI/CD platforms allow for code reuse either directly or indirectly. Jenkins, for instance, allows you to pull in code from various repositories. Circle CI and Drone CI allow you to configure scripts to run for each step, which can easily be reused. This code reuse provides so many benefits as your microservices begin to grow. The more code you can reuse the better off you’ll be.

30

Chapter 2

Setting the Stage for the Present

 icroservices and Shared Pipelines are M More Agile Whether you like agile practices or hate them, they provide a process to release code much faster. Microservices and shared pipelines can greatly facilitate this process. With microservices you can begin to focus on smaller releases, generally tied around a new feature. Figure 2-8 illustrates a release cycle of two weeks, which could match up to the development team’s sprint.

Figure 2-8.  Two-week release cycle As the example demonstrates, development takes approximately two weeks, at which point the feature code is committed to the “master” branch. The pipeline is executed based on the commit and a full suite of tests are kicked off. A release candidate artifact is created and can be

31

Chapter 2

Setting the Stage for the Present

promoted to Production at any time. This image looks drastically different than the one shown in the previous chapter. Microservices give you the opportunity for rapid development, testing, and deployments. A shared pipeline by itself does not necessarily contribute to rapid deployments. It’s the benefits of this pattern where you see the speed: •

Teams can on-board faster: Gone are the days of requesting a custom pipeline, meeting to discuss needs, and waiting for it to be built and tested. Now development teams simply apply the standard and can on-board much faster.

•

Enhancements are provided to all at the same time: If a new feature is added to the language-specific pipeline, everyone benefits immediately.

•

Maintenance is done all at the same time: Bug fixes don’t need to be applied to multiple pipelines. If a new version of Node.js comes out, simply update the Node. js pipeline and you’re done.

Overview In this chapter we explored how moving into a microservice architecture can speed up development and releases, but also increase the number of applications that need to be deployed. We looked at how a language-­ specific pipeline can ease the DevOps workload by providing a reusable shared pipeline that all teams can use. We also reviewed how code reuse is possible in this pattern, further lightening the load and increasing output. In the next chapter we’ll go one step further, and begin exploring the truly generic pipeline. Not only generic from the application’s point of view but also from the underlying platform perspective. We’ll see how Docker can help us take our pipelines anywhere and how scripts take code reuse to the next level. 32

CHAPTER 3

Getting it Right with Docker and Scripts Up until now we’ve looked at pipelines that work with monoliths and microservices. We’ve explored the challenges that come with both, but have seen how microservices can ease your pipeline workload. In the previous chapter we saw how you can build language-specific pipelines. These implementations allow for multiple teams to take advantage of a single pipeline. It also makes you begin to set and enforce development standards, which allows for code reuse across your pipelines. We’ll explore this concept much deeper in later chapters with code examples. Language specific pipelines are great; they allow the DevOps team to focus on fewer pipelines by giving feature teams a shared implementation. It lowers maintenance efforts, allows for all teams to share in upgrades and improvements, and sets standards for how applications are deployed. While all this is great, we can do better. Imagine if you could build a single pipeline that could deploy any application regardless of the language it was written in. How much time could your DevOps team get back if they only had to support one implementation? In this chapter we’ll begin laying the foundations of the generic pipeline using Docker. You’ll learn the pattern and process that allows a truly generic pipeline to work. We’ll explore how this pattern is driven by Docker and frees you from the underlying platform the pipeline runs on. At the end of this chapter you’ll be ready to start coding an implementation. © Brandon Atkinson, Dallas Edwards 2018 B. Atkinson and D. Edwards, Generic Pipelines Using Docker, https://doi.org/10.1007/978-1-4842-3655-0_3

33

Chapter 3

Getting it Right with Docker and Scripts

One Pipeline to Rule Them All Language-specific pipelines are a great first start. They allow you to break free of application-specific implementations and begin to serve a larger base of teams simultaneously. However, as your feature teams begin to fully embrace microservices, it stands to reason they will begin to embrace other languages as well. You may start off as a 100% .NET or Java shop, and slowly turn into a true polyglot shop. This may not be a huge concern if you’re supporting a few languages, but what about five, six, or seven languages? This may seem like a lot, but it’s not. I have worked at organizations that had the following languages to support: •

User Interface: Angular and React

•

Server Side: .NET Core, Java, Node.js, Python, Golang

Microservices give feature teams the flexibility to try out new languages and patterns. They allow for choosing the right language for the job as well. Take .NET and SQL Server for instance. Let’s imagine you’ve been writing back-end services in Node.js and now must write a new API that connects to SQL Server. You could use Node.js for this task; however, .NET has builtin functionality to do just that. You can very quickly spin up a .NET Core microservice that handles CRUD operations against your SQL Server. It would be faster to write and less error prone, since it’s inside the .NET ecosystem. Another scenario involves hiring of new talent. Suppose you hire an amazing developer. When they join, they talk about writing their microservices in Golang, since it’s easy to write, easy to learn, and compiles and deploys very fast. You try it out and like it. Suddenly Golang services are springing up all over the place. The moral of the story is with microservices developers, feel free to try out new languages or stray from the standard if another language provides benefits. As more and more languages come on the scene, you’ll find yourself writing more and more pipeline implementations. If you’re in an organization that supports seven languages, that would be seven pipelines. 34

Chapter 3

Getting it Right with Docker and Scripts

Even in a simple scenario, you’d have two languages to support, one for the UI and one for the server side. That’s one too many! A better solution is a single pipeline that is built to handle any language that your teams work with. In the previous chapter we looked at shared steps of a pipeline, as shown in Figure 3-1.

Figure 3-1.  Shared steps of a CI/CD pipeline A typical pipeline may consist of the following steps: •

Build

•

Unit Test

•

Static Code Scan/Security Scan

•

Packaging/Publishing of Artifacts

•

Deploying

•

End to End Tests

•

Performance Tests

35

Chapter 3

Getting it Right with Docker and Scripts

How mature your organization and CI/CD efforts will determine how many steps you have, but these are a good target. Upon close examination, most of these steps are not specific to a language. For instance, if you're using commercial software for static code analysis it will usually work with a wide variety of languages. Packaging of artifacts usually involves zipping up binaries and shouldn’t be any different across applications. The main differences in applications occur during the first two steps: build and unit test. Given this, we can reuse most of our code across steps, and consolidate code for the remaining ones using logic. For instance, Figure 3-2 shows a fully shared code base across all steps.

Figure 3-2.  Pipeline stages using shared code In this scenario, the entire pipeline is now shared across all languages. Logic is included in the language-specific steps to perform the appropriate commands. This can be accomplished via if/else blocks, switch/case statements, etc. At first glance this may seem like a lot of logic code, especially if you’re dealing with a lot of languages. However, earlier in the book we discussed implementing development standards in your shared

36

Chapter 3

Getting it Right with Docker and Scripts

pipeline. In this scenario you’d have set common build commands to run for each language. For instance, in .NET you’d use the standard “dotnet build” command. These commands generally will be the built-in, out of the box commands for each language. The goal is to set a standard that all teams can follow without being overly complicated. Listing 3-1 shows an example build step.

Listing 3-1.  Sample Build Step in a Generic Pipeline if [ $language = "dotnetcore" ] then     dotnet build -c Release -o /output elif [ $language = "java" ] then     mvn package -s settings.xml -f / elif [ $language = "angular" ] then     npm run build else     echo "Error: No valid language provided." fi This example shows how you can pass in a simple variable, “language” in this case, to inform the step which language it is working with. An if/else statement then allows you to perform the build command for that language. Listing 3-2 shows the same example using a switch/case statement.

37

Chapter 3

Getting it Right with Docker and Scripts

Listing 3-2.  Sample Build Step Using a Switch/Case Statement case $language in "dotnetcore")       dotnet build -c Release -o /output       ;; "java")       mvn package -s settings.xml -f /       ;; "angular")       npm run build       ;; *)       echo "Error: No valid language provided."       ;; esac In this example we’re covering three languages in the “build” step: Angular, .NET Core, and Java. The commands in each section are admittedly short and probably not a full set of the commands you may run. For instance, a proper Angular build section may look more like Listing 3-3.

Listing 3-3.  A More Complete Angular Build Step "angular")       npm set progress=false       npm install –-registry https://registry.npmjs.org/       npm run build       ;; This case statement has three times the amount of code in it, but it’s still small for all intents and purposes. Even if you need to do more complex things for a specific language, you will most likely not end up with so many lines that it’s unmanageable. This is where setting standards for 38

Chapter 3

Getting it Right with Docker and Scripts

languages becomes important. To build a truly language-agnostic pipeline that all teams can use, you need high-quality standards. We’ll continue to explore this throughout the book; however, in this case your Angular build standard would be: •

Run “npm set progress=false” to turn off progress bars.

•

Run “npm install” to restore packages.

•

Run “npm build” to build the project.

•

No other commands will be run.

With this build standard in place, all teams would need to conform to it or in order to use your pipeline. This may seem harsh, but there are a lot of benefits to it. Teams that share common standards on how they build and test can more rapidly debug issues and assist other teams with issues, and it simplifies your work in the pipeline. Now of course, there will always be outliers who need to do things their own way. In those cases it’s important not to deviate from the standard. You could explore things like a custom pipeline for that application, or provide hooks into the pipeline to allow teams to override your steps. In most cases you will need to hold strong on the standards and say no to a lot of requests.

Shell Scripts For your pipeline to be truly generic and run anywhere, it needs to be written in a way that is portable. So far we’ve explored writing shared steps. In the previous section we saw how to combine commands from different languages into a single step. In case you haven’t noticed, those commands were all written in Shell. For this book we’ve chosen Shell scripts to execute all our commands.

39

Chapter 3

Getting it Right with Docker and Scripts

Shell scripts were an easy choice to make for a variety of reasons. All our examples are written in languages that run in a Linux environment. This includes .NET Core, Angular, and Java. This was also an easy choice, since this book is about writing pipelines with Docker. While Docker can run on Windows, I would argue most organizations don’t utilize this option—especially those running on Amazon Web Services using Elastic Container Service (ECS) or Google Cloud Platform Kubernetes Engine. A Shell environment will be present in all our containers without any additional installations, which is nice! No matter the container we’re working in, we’re confident a Shell environment is available. However, this is not the case with all Shell languages. Bash, for instance, is a very popular Shell language, but if you’re new to Docker you’ll quickly find out that it’s not always available. For instance, Alpine containers, which are lightweight Linux distribution based on “musl libc” and “busybox”, have a Shell environment but do not come with Bash installed. While we have chosen Shell scripts for this book, there is no reason you cannot deviate from this and use another scripting language. For instance, you can use Bash or Python just as easily. The only caveat is you will need to ensure those runtimes are installed in your Docker image. In fact, if you happen to be using Windows containers, there is no reason you can’t follow along with us. You’ll just be using PowerShell in your containers, most likely. Let’s take the example from the previous section and reimagine it as a Shell script for the build step. Listing 3-4 shows what this might look like.

40

Chapter 3

Getting it Right with Docker and Scripts

Listing 3-4.  Build Step as a Shell Script #!/usr/bin/env sh case $1 in "dotnetcore")       dotnet build -c Release -o /output       ;; "java")       mvn package -s settings.xml -f /       ;; "angular")       npm run build       ;; *)       echo "Error: No valid language provided."       ;; esac This example looks almost identical to Listing 3-2, with a couple of small but important changes. First, we’ve include a shebang as the first line in the file to indicate this is a Shell script. This would change based on the language you’re using. For instance, if you wanted to use Python 3 your shebang would be #!/usr/bin/python3. Next, we’ve replaced the “$language” variable with a “$1” indicating we’re passing it in as an argument. While this is still a very simple implementation of a build step, these two changes make it a fully functional step. Just save it as "build.sh" and you’re ready to use it in your pipeline. This also opens the door for making your pipeline logic much more modular. Take Listing 3-3 as an example. This was a much more detailed build command for Angular. Granted it is all of three lines, but imagine you have a language that requires 30 lines or more. This is probably more code

41

Chapter 3

Getting it Right with Docker and Scripts

than you care to have in a switch/case statement. Now that we’re using Shell scripts, we can reimagine that code as a separate script as shown in Listing 3-5.

Listing 3-5.  Angular Build Commands in Their Own Shell Script #!/usr/bin/env sh npm set progress=false npm install –-registry https://registry.npmjs.org/ npm run build Now, this script can be saved as "angular_build.sh". All your build logic is now consolidated into a separate script. If you needed to go crazy and have dozens of lines of code, it’s isolated here. This makes writing, maintenance, and debugging much easier. It also begins to open the door for sharing code across multiple pipelines. If we take all our build commands and put them into separate Shell scripts, our build step could be simplified as shown in Listing 3-6.

Listing 3-6.  Simplified Build Step Shell Script #!/usr/bin/env sh case $1 in "dotnetcore")       dotnet_build.sh       ;; "java")       maven_build.sh       ;; "angular")       angular_build.sh       ;; *) 42

Chapter 3

Getting it Right with Docker and Scripts

      echo "Error: No valid language provided."       ;; esac While we haven’t reduced the line count of the code in the file, we’ve greatly simplified it, making it easier to read and follow. The script is no longer cluttered with code from the various languages. If we need to add another language, simply write the appropriate build script and then add another case statement to the build step script. This method introduces shared scripts that can be executed from a step on your CI/CD platform. For instance, let’s imagine that you have multiple lines of business in your organization, each with their own DevOps team. Each LOB has development teams building microservices, and each runs their own CI/CD platform. We’ll also say that all those teams are writing microservices in Node.js. It’s not hard to imagine that each DevOps group has its own pipeline that can build, test, and deploy Node.js services. Each pipeline is essentially doing the same thing, and most likely using almost identical code to do it! If both teams adopted using Shell scripts to build their Node.js services, then they could then easily share code. In fact, the code in Listing 3-5 could simply be renamed “npm_build.sh” and used for all Node.js applications! Even if each team was using different platforms for their pipelines, running Shell scripts is supported in every major platform.

C  onfiguration Files For Shell scripts to properly handle multiple languages, you must have some way to inform your pipeline about the application you want to build. You want to be explicit about what you are doing. A configuration file can solve this issue for you. Development teams can place this file in their repo and it would be cloned along with the application when the pipeline executes. It would contain all the information the pipeline would need to execute. Listing 3-7 shows what a simple configuration file may look like. 43

Chapter 3

Getting it Right with Docker and Scripts

Note This book focuses on building applications that are deployed via Docker containers to an orchestration service like Amazon ECS or Kubernetes. As such, the configuration file shown in this chapter is specific to that. A configuration file for your applications may look drastically different. Listing 3-7.  A Sample Configuration File {   "application": {     "name": "Hello App",     "language": "dotnetcore"   },   "build": {     "path": "",     "outputPath": "HelloApp/bin/Release/netcoreapp2.0"   },   "test": {     "enabled": true,     "path": "HelloTests/"   },   "archive": {     "registry": "docker.io",     "namespace": "YOUR-NAMESPACE",     "repository": "YOUR-REPO"   },   "deploy": {     "containerPort": 5000   } }

44

Chapter 3

Getting it Right with Docker and Scripts

Let’s break this down by each section to better understand its makeup: •

•

•

Application: Contains basic information about the application •

Name: This is a friendly name for the application. This may be its identifier in the UI of the platform or used for reporting.

•

Language: This is the language the application is written in. This is the most important, if not only, variable the pipeline may care about.

Build: Contains information about how to build the application •

Path: This would be the directory path in the cloned repo, in case your application is located somewhere other than root.

•

OutputPath: This tells the pipeline where the built binaries should be placed. It is useful if other stages require the binaries to be placed in specific locations.

Tests: Contains information about how to execute unit tests •

Enabled: Would allow the application to bypass a stage

•

Path: Used if the unit tests are not located in the same directory as the application

45

Chapter 3

•

•

Getting it Right with Docker and Scripts

Archive: Contains information about how to archive the built application •

Registry: The URL to the Docker registry where the application will be pushed

•

Namespace: The namespace in the registry

•

Repository: The repository name under the namespace

Deploy: Contains information pertaining to the deployment •

ContainerPort: The container port number

As you can see, even a simple configuration file can get complex very fast. However, this file contains just enough information that our pipeline can execute its stages and deploy our application. A lot of thought needs to go into these files to make them flexible for future changes. Additional sections may include things like: •

Security Scans

•

Static Code Analysis

•

Performance Tests

•

ATDD Tests

This could go on and on. The main takeaway here is that for a generic pipeline that uses shared code to function properly, you need a way to instruct it on which paths to take while executing. We will explore configuration files in more detail, as well as use them in later chapters.

46

Chapter 3

Getting it Right with Docker and Scripts

Docker at the Core At the core of the generic pipeline is Docker. It is the glue that holds everything together, as well as the magic that makes it all possible. Docker provides a mechanism where we can isolate our pipeline from the underlying platform. It also allows us to create an environment that is specific to the needs of the application being deployed. For instance, if we’re building a .NET Core application, we don’t need to worry about having the Java runtime installed. It also allows for us to easily target specific runtime. If you’re in a larger organization, it’s not uncommon to have an enterprise CI/CD platform that you must use. In these scenarios you are often forced to use the runtimes that are installed on the platform, or face long lead times to get new ones installed. Continuing with our .NET Core example, imagine we have an enterprise platform with various runtimes installed: •

Java Runtime Environment 8

•

.NET Core 1.0

•

Python 2.6

•

.NET Standard 4.5

•

Sonar Scanner 2.0

If we’re using .NET Core 1.0 we’re in good shape. It’s installed on the platform, and we can build and deploy to our heart’s content. Figure 3-3 shows what this platform may look like.

47

Chapter 3

Getting it Right with Docker and Scripts

Figure 3-3.  Enterprise CI/CD platform Things are humming along just fine; however, our development team has begun working on a new version of their application and they are using .NET Core 2.0. Well that’s not going to work; we need to get .NET Core upgraded on the platform. In most cases this is not a quick process. For an enterprise to run a tight ship, they need to vet installations in lower environments first. Only after proper testing has taken place can the upgraded version of the runtime be scheduled for a Production deployment. That also takes time, as you need to secure a change order and downtime window. Figure 3-4 shows the updated platform after the installation.

48

Chapter 3

Getting it Right with Docker and Scripts

Figure 3-4.  Enterprise CI/CD platform with new runtimes installed If your team happens to be agile, they are most likely putting you on their impediment list! At this point the enterprise platform has become a bottleneck slowing down the development teams. In a worst-case scenario, you’re stifling progress and innovation because teams cannot move as fast as they need to. Imagine that an early access release of .NET Core comes out and a team would like to use it for their application. This would be even more of a challenge given it’s not a release candidate! In addition to the slowdowns that this can present, there are also a ton of runtimes installed that most developers don’t need. To put it another way, the Java developers don’t need .NET Core and vice versa. Now most of the time this is not a problem, as multiple runtimes can be installed side by side without issue. But larger platforms, like Jenkins, also come with a lot of plugins that sometimes don’t play nice with each other. As the platform’s popularity grows inside the organization, more and more requests for plugins flow in. At some point there may be a conflict between plugins, and someone will have to lose out on functionality they were counting on.

49

Chapter 3

Getting it Right with Docker and Scripts

These scenarios can continue to play out in many different forms. For instance, given our platform and the runtimes we have installed, if a team wanted to use Golang for their project they’d have to wait for that to be installed. Luckily all these problems can be solved with Docker. The only requirement is that it is installed on the enterprise platform. From there we have total control over what our applications need in the pipeline. Figure 3-5 shows the ideal platform in its purest form, with only Docker installed.

Figure 3-5.  Enterprise CI/CD platform with only Docker installed Now at this point you may be thinking to yourself, “this is insane.” Why would I have an entire platform and only put Docker on it. Well, you’re still going to have a lot of other things installed/configured for your enterprise. However, you can begin to break free of installing individual runtimes and tooling to support all the applications your platform supports. Docker allows for teams to control their environments and only install things specific to their applications.

50

Chapter 3

Getting it Right with Docker and Scripts

In the previous figures we had a platform with many runtimes installed to support many different development teams. Let’s see what that looks like with Docker in the mix. Figure 3-6 shows the platform once we begin to utilize Docker.

Figure 3-6.  Enterprise CI/CD platform utilizing Docker during builds In this scenario, each application is built, tested, and deployed inside a container that is isolated from other containers. The container is responsible for what components and runtimes are installed, thus relieving the platform team of being responsible for installing and maintaining multiple runtimes on the platform. With this approach, the previous scenario of a team switching to a new runtime version becomes trivial. In fact, a team wanting to be bleeding edge and use an Early Access release is not a concern for the enterprise platform team anymore. Figure 3-7 shows an updated .NET Core 2.0 container and an Early Access .NET Core container being utilized.

51

Chapter 3

Getting it Right with Docker and Scripts

Figure 3-7.  Enterprise CI/CD platform with updated containers In this pattern everything the application needs to be built, tested, scanned, and deployed is inside the “build” container. It’s called a build container, as all the work in the pipeline will be performed inside it. We can illustrate this with a simple Dockerfile example. Listing 3-8 shows a sample Dockerfile for a .NET Core 1.0 application.

Listing 3-8.  Sample Dockerfile for a .NET Core 1.0 Build Container FROM microsoft/1.0-runtime:latest RUN apt-get update && apt-get install -y \           unzip RUN wget http://repo1.maven.org/.../sonar-runner-dist-2.4.zip && \           unzip sonar-runner-dist-2.4.zip -d /opt ENV PATH $PATH:/opt/sonar-runner-dist-2.4/bin In this example our build container is based on a .NET Core base image. We update “apt-get” and install “unzip”. After that we download Sonar Runner, unzip it, and update the “PATH” so we can easily run it.

52

Chapter 3

Getting it Right with Docker and Scripts

At this point we have a build container that can build, test, and scan .NET Core 1.0 applications. Now if the development team decides they want to move to .NET Core 2.0 (why wouldn’t they) we simply create another build container using the Dockerfile shown in Listing 3-9.

Listing 3-9.  Sample Dockerfile for a .NET Core 2.0 Build Container FROM microsoft/aspnetcore-build:2.0 RUN apt-get update && apt-get install -y \           unzip RUN wget http://repo1.maven.org/.../sonar-runner-dist-2.4.zip && \           unzip sonar-runner-dist-2.4.zip -d /opt ENV PATH $PATH:/opt/sonar-runner-dist-2.4/bin As you can see, the only thing that changed was the base image of the Dockerfile. In this scenario we now have two build containers, all from changing a single line of code. Contrast that with what it would take to install a new runtime in the platform, fully test it in lower environments, and then promote it to Production. With Docker, we can simply make a change to a Dockerfile and test it. If it works, great! If not, you’ve literally wasted about ten minutes of work. Earlier in the chapter we saw an example architecture where the pipeline was made up of shared components. Figure 3-8 shows us this concept again.

53

Chapter 3

Getting it Right with Docker and Scripts

Figure 3-8.  Pipeline stages using shared code In this scenario we used Shell scripts to encapsulate shared commands, which can be used across all of the application deployments. Now, we can envision this taking place inside Docker containers, as shown in Figure 3-9.

Figure 3-9.  Pipeline stages using shared code inside build containers 54

Chapter 3

Getting it Right with Docker and Scripts

Now we have common shared pipeline code loaded into build containers that are isolated from each other. The same code is copied into each container, but changes to a container no longer have any side effects on the others. This is drastically different than if the CI/CD platform is responsible for all the runtimes and plugins. As illustrated earlier, a team can quickly and safely jump to a new version of a runtime without fear of affecting other teams on the previous runtime. Loading the shared shell scripts is extremely easy. All we need to do is add a COPY command to our build container Dockerfile, as shown in Listing 3-10.

Listing 3-10.  Adding Shell Scripts to Our Build Container FROM microsoft/aspnetcore-build:2.0 COPY stages stages RUN apt-get update && apt-get install -y \           unzip RUN wget http://repo1.maven.org/.../sonar-runner-dist-2.4.zip && \           unzip sonar-runner-dist-2.4.zip -d /opt ENV PATH $PATH:/opt/sonar-runner-dist-2.4/bin In this example we assume there is a directory called “stages” that holds all the Shell scripts for our pipeline. By copying the scripts into each build container, we have further isolated changes they may impact other teams. A change to a script will only be reflected once the build container is rebuilt. This is certainly not the only way to get your shell scripts into your build container. You could copy them in as part of an application clone, or if using a language like Python you could include them as modules and perform a PIP installation. The main takeaway is that the scripts are part of the build container and isolated from other code.

55

Chapter 3

Getting it Right with Docker and Scripts

P  latform Agnostic Running a CI/CD pipeline inside a Docker container provides you with so many benefits. We’ve already seen how it can provide your development teams isolation from other runtimes, and give them total flexibility on what is installed in the container. Combine this with scripts loaded into the container to execute your pipeline code, and now you’re decoupled from the underlying platform. This is a very big deal. Many enterprise grade CI/CD platforms already allow you to share code across pipelines; this is not a new concept. However, in most cases the mechanisms in place to do so are clunky and not ideal. Take for instance Jenkins, which is a very large player in this space. Jenkins allows you to configure libraries that contain shared code for use in your pipeline. These can be configured at the local (folder) level or globally. Figure 3-10 shows a sample configuration.

Figure 3-10.  Folder level configuration of a shared library 56

Chapter 3

Getting it Right with Docker and Scripts

In this scenario, we have a GitHub repo called “my-library” with some shared code in it. We reference this library in our Jenkinsfile for the pipeline to load it. Jenkins has a very specific directory structure to allow these files to be shared. The “my-library” repo would be structured like Listing 3-11.

Listing 3-11.  Directory Structure of a Shared Library (root – "my-library") +- vars |   +- foo.groovy |   +- bar.groovy |   +- baz.groovy There may be many more files in this directory but there must be a folder named “vars,” which only contains Groovy files. While this is a fine method for sharing code across pipelines, it leaves a lot to be desired. This pattern also makes roper unit testing of the code more difficult, albeit still possible. It also locks us into using Groovy for our shared code, or at least using it as an entry point and then calling something else under the covers. By using more generic scripts like Shell, we can already begin to break free of the platform even without Docker in the equation. If you’re on Windows you can use PowerShell, or even now you can consider Bash, which would not only decouple you from the platform but also the operating system. Putting Docker into the mix gives you the same capability; the only requirement now is that Docker be installed. You could also explore a language like Python, which pretty much runs anywhere. If we were to move all the code in “my-library” into Shell scripts, our Jenkins configuration would be much simpler. In fact, all the steps from earlier would be gone. We’d have a Docker container loaded with our scripts, which we would simply run on Jenkins. Listing 3-12 shows a sample Docker image run in Jenkins. This example is in Groovy, and the Jenkins platform has the Docker plugin installed. 57

Chapter 3

Getting it Right with Docker and Scripts

Listing 3-12.  Running a Docker cContainer in Jenkins docker.withRegistry('YOUR-DOCKER-REGISTRY') {        docker.image('YOUR-CUSTOM-IMAGE').inside('ARGS') {              stage('Clone') {                    sh("/stages/01_clone.sh")              }              stage('Build') {                    sh("/stages/02_build.sh")              }              stage("Test") {                    sh("/stages/03_test.sh")              }              stage("Archive") {                    sh("/stages/04_archive.sh")              }        } }

Note The preceding stages are the actual stages we’ll be building in upcoming chapters. This example would assume that you have cloned or copied your scripts into the root of the container that is mapped to the Jenkins workspace. Now that we have all our shared code in scripts in a GitHub repo, we can copy that in via our Dockerfile and use the preceding command to execute the container. You have the option to either “bake” your scripts into the container when it’s built, or you could even do a git clone command to bring the scripts down when the container runs. The point here is you have some options.

58

Chapter 3

Getting it Right with Docker and Scripts

Now let’s consider that you have your pipeline built in a Docker container using Shell scripts, and you’re up and running on Jenkins. Everything is going great, and then your boss comes to you and proclaims that Jenkins is out and Circle CI is in! In most organizations around the world this would be a major event. All your pipelines need to be rewritten on Circle CI. It will take months to migrate everything over, not to mention the testing and deployments and, wait a minute! We have everything isolated in Docker containers. This won’t be so bad. We can take the same Docker image we’ve used in Jenkins and easily run it on another platform. Listing 3-13 shows what the same setup may look like in Circle CI.

Listing 3-13.  Running the Docker Container in Circle CI version: 2 jobs:   build:     docker:       - image: YOUR-DOCKER-REGISTRY/YOUR-CUSTOM-IMAGE     steps:       - setup_remote_docker:           docker_layer_caching: true       - run:           name: Clone           command: /stages/01_clone.sh       - run:           name: Build           command: /stages/02_build.sh       - run:           name: Test           command: /stages/03_test.sh

59

Chapter 3

Getting it Right with Docker and Scripts

      - run:           name: Archive           command: /stages/04_archive.sh At this point the beauty of the pattern should be clear. While we are just looking at configuration files, you should readily see the similarities. On either platform, we use our Docker image as the basis for running the pipeline. Stages or Steps are defined and given names, like Clone, Build, Test, etc. Next, a Shell script is executed in each section. Since everything runs inside a container, there are no surprises when we execute the pipeline. It will run the same on each platform!

Note At this point you may be thinking this is way too easy. While this pattern makes moving the code and executing it on another platform crazy easy, you still need to keep in mind the other aspects of running a CI/CD platform. Each will have its own ways of being configured, dealing with networking, etc. However, not having to worry about how to port a pipeline will provide a lot of breathing room if you need to make a switch. To further illustrate the point of being agnostic, let’s look at what this pipeline looks like on another platform. Listing 3-14 shows what the same setup may look like in Travis CI.

Listing 3-14.  Running the Docker Container in Travis CI services:   - docker before_script:   - |     docker run -it -d \       -v /var/run/docker.sock:/var/run/docker.sock \       -e DOCKER_USERNAME=${DOCKER_USERNAME} \ 60

Chapter 3

Getting it Right with Docker and Scripts

      -e DOCKER_PASSWORD=${DOCKER_PASSWORD} \       -e GITHUB_URL=${GITHUB_URL} \       --name MY-PIPELINE \       YOUR-DOCKER-REGISTRY/YOUR-CUSTOM-IMAGE script:   - docker exec MY-PIPELINE /stages/01_clone.sh   - docker exec MY-PIPELINE /stages/02_build.sh   - docker exec MY-PIPELINE /stages/03_test.sh   - docker exec MY-PIPELINE /stages/04_archive.sh In later chapters we’ll be using Circle CI and Travis CI to build a very simple pipeline. We’ll use the same Docker container and Shell scripts to build, test, and archive an application on both platforms. In fact, the Yaml files shown will be used in those examples!

Overview We covered a lot of material and patterns in this chapter! Let’s take a quick moment to review what we discussed.

Shell Scripts Shell scripts provide us a way to centralize our pipeline logic into smaller easy to share chunks of code. We can write scripts for very specific functionality and then use them inside “Stage” or “Step” scripts to stitch them together. By doing so, we create scripts that are easier to test and share across multiple pipeline implementations. Also, by using Shell we can be guaranteed they will run across any platform. This pattern also opens us up to using other scripts like Python or PowerShell.

61

Chapter 3

Getting it Right with Docker and Scripts

Docker Docker provides us the isolation needed to simplify the needs of the development team while running on the platform. In the past, the platform would need to have all the runtimes and plugins installed globally for anyone to use. With Docker, the only requirement is that Docker be installed for all to use.

Build Containers Build containers provide a level of isolation on the platform. Each team can install their own runtimes and dependencies in a container, thus isolating their needs from others. This also provides them with the flexibility to use anything they want, without fear of it affecting the platform. If a team wants to us a bleeding edge release of a runtime, they have the power to do it. They own the build container (or a DevOps team) and are responsible for what goes in it. This allows teams to move at a much more rapid pace than if they were dependent on the platform team. In the next chapters we’ll move out of theory and discussion and into actual implementation. We’ll create a simple demo application that we can use to deploy. We’ll look at building out our build container and how to implement it in Circle CI and Travis CI. You’ll take everything you have learned so far and apply it in a practical way. So let’s get started!

62

CHAPTER 4

A Practical Example In Chapter 3 we showed you an example of how we can execute a few simulated pipeline stages in a Docker container to create a pipeline that can be ported to any CI server. In this chapter we’re going to take it a step further and create a pipeline that can clone, build, test, archive, and deploy a set of working applications. Then, we’ll show you how to take the pipeline you’ve created and move it from your desktop to two popular CI platforms.

An Overview of Our Applications In this chapter we are focused on applications written using one of three distinct tech stacks; they are: 1. Spring Boot applications written in Java, using Maven as a build automation tool 2. ASP.NET Core Web APIs written in C# 3. Angular applications leveraging TypeScript and Node.js We’ve created three sample projects, each using one of these technologies. These projects include everything you need—code, configuration files, Dockerfiles, etc.—to follow along for the rest of the chapter. The next three sections provide a brief overview of what these © Brandon Atkinson, Dallas Edwards 2018 B. Atkinson and D. Edwards, Generic Pipelines Using Docker, https://doi.org/10.1007/978-1-4842-3655-0_4

63

Chapter 4

A Practical Example

sample applications do and the commands we execute to build, test, and run them. We won’t go into detail on how each of the sample applications work—there are other books better suited for that. These are barebones implementations and aren’t meant to be used as a model for writing high-­ quality applications.

S  pring Boot Note  You can find the source code for this project at https:// github.com/Apress/generic-pipelines-using-docker The first application is an API written in Java using the Spring Boot framework, and using Maven to build, test, and package the project. It’s based off the quick start example on the Spring Boot homepage and works with any relatively recent version of Java and Maven. For the purposes of this book, we’re using JDK 8 and Maven 3.

Note  If you want to learn more about the Spring Boot framework and how this project works, check out the quick start guide at https://projects.spring.io/spring-boot/#quick-­start. To run the application from the command line, first compile the source code using the mvn clean package, then execute java -jar target/hello.jar. After a few seconds the application will be up and running. Once it is, open your web browser and navigate to http:// localhost:8080. You’ll receive a simple message: “Hello World!” as shown in Figure 4-1.

64

Chapter 4

A Practical Example

Figure 4-1.  The Spring Boot application

ASP.NET Core Web API Note  You can find the source code for this project at ­https:// github.com/Apress/generic-pipelines-using-docker. The second application is also an API, but this time written on top of ASP.NET Core Web API. It has a single endpoint that returns an array containing two values. It’s based off of the ASP.NET Core Web API project template generated by Visual Studio. This project uses the commands built into the .NET Core CLI to build and test the project. Using the command line, run dotnet build ValueApi to build the project, followed by dotnet ValueApi/bin/Debug/ netcoreapp2.0/ValueApi.dll to start the API. If you open a browser and navigate to http://localhost:5000/api/values, you’ll get a response like the one in Figure 4-2.

Figure 4-2.  The .NET Core application

65

Chapter 4

A Practical Example

A  ngular 5 Note  You can find the source code for this project at https:// github.com/Apress/generic-pipelines-using-docker. Our last sample project is a web application built using Angular 5, TypeScript, and Node.js and relies on Chromium to run its test suite. It’s based on the Angular Quick Start example. To build and run the project, first download and install the project’s dependencies by running npm install on the command line. Next, use npm run build to compile the project. Finally, host the project in a lightweight web server by running npm start. Once the server is running, open a browser and navigate to http://localhost:4200. You will see a simple website like the one shown in Figure 4-3.

Figure 4-3.  The Angular 5 application

66

Chapter 4

A Practical Example

A Deep Dive into the Pipeline Note  You can find the source code for this project at ­https:// github.com/Apress/generic-pipelines-using-docker. Now that you’ve got a high-level overview of the sample projects we’re dealing with in this chapter, it’s time to explore the pipeline itself. In the following sections, we explore the configuration file that drives the behavior of the pipeline. Then we take a close look at each of the five stages of our pipeline: clone, build, test, archive, and deploy. Finally, we take a peek inside the build containers where all this takes place.

The Pipeline Configuration File Alongside the source code for each application, you’ll find a small JSON file named pipeline.json that contains some crucial information about the application. The pipeline will use this file to decide everything from how to build the application to where the resulting artifact should be stored for use later. The next three listings show the configuration file for each of our sample projects:

Listing 4-1.  The Pipeline Configuration File for the Sample Java Project {   "application": {     "name": "Sample Java Application",     "type": "java"   },   "build": {     "path": null,     "outputPath": null   }, 67

Chapter 4

A Practical Example

  "test": {     "enabled": true,     "path": null   },   "archive": {     "registry": "docker.io",     "repository": "edwardsdl/sample-java"   },   "deploy": {     "containerPort": 8080   } }

Listing 4-2.  The Pipeline Configuration File for the Sample .NET Core Project {   "application": {     "name": "Sample .NET Core App",     "type": "netcore"   },   "build": {     "path": null,     "outputPath": "ValueApi/bin/Release/netcoreapp2.0"   },   "test": {     "enabled": true,     "path": "ValueTests/"   },   "archive": {     "registry": "docker.io",     "repository": "edwardsdl/sample-netcore"   }, 68

Chapter 4

A Practical Example

  "deploy": {     "containerPort": 5000   } }

Listing 4-3.  The Pipeline Configuration File for the Sample Node Project {   "application": {     "name": "Sample Node App",     "type": "node"   },   "build": {     "path": null,     "outputPath": "dist/"   },   "test": {     "enabled": true,     "path": null   },   "archive": {     "registry": "docker.io",     "repository": " edwardsdl/sample-node"   },   "deploy": {     "containerPort": 5000   } } Our pipeline configuration file is broken into four sections: application, build, test, and archive. Some provide information about the application, while others provide fine-grained control over specific stages in the pipeline. Let’s take a closer look at each section. 69

Chapter 4

A Practical Example

The application section stores high-level information about the project. We use it to store the name and type of the application. In this book we deal with three types: node, java, and netcore. It’s up to you to decide what application types you want to support and what identifiers to assign them. Supporting a greater number of tech stacks will give your developers more flexibility but will require more work on their end. If a new version of the .NET Core framework is released and maintains backwards compatibility, netcore1, netcore2, and netcore3 applications can all share a single build image. This is a useful trick for keeping the number of images you have to maintain to a minimum. The build section contains two elements: path and outputPath. The path element is used to let the pipeline know where the applications code can be found when it pulls it from source control. For most projects the source code is located in the root of the repository, but we’ve found some teams appreciate the flexibility to put it elsewhere. The outputPath element lets the pipeline know where to put the compiled output of the build command. The test section contains configuration settings for the test stage of the pipeline. The first property, enabled, is used to determine whether this stage is run at all. As your pipeline becomes more robust with additional stages and features, you’ll likely find yourself adding this property to other sections too. It can be very useful to turn portions of the pipeline on and off due to unusual situations or for nonstandard projects. The path property in this section tells the pipeline where the tests are located relative to the project’s root directory.

Caution  If the enabled property is setting off alarm bells, that’s a good thing! This is useful for teams who don’t yet have unit tests or need to temporarily disable the stage while working through an issue. This has the potential to be used as a crutch!

70

Chapter 4

A Practical Example

The archive section lets the pipeline know where the build artifacts— in our case the Docker image containing one of our sample applications— should be stored. The registry property indicates which Docker registry will store the image. In this case, we’ll be storing the image on Docker Hub (Docker’s public registry). If your organization is hosting their own internal registry, you’d put that here instead. The repository property is the name of the image and should not include any tags; the pipeline will handle all of the tagging automatically. Your organization may require you to store application binaries and images separately. In that case you can modify this section to represent an array of artifact repositories.

The Clone Stage Typically, the first step in any pipeline is to download a copy of a project’s source, and this pipeline is no different. If you haven’t already, this would be a good time to clone or review the GitHub examples for the book. In the stages directory, you’ll find a file named 01_clone.sh with the following code:

Listing 4-4.  The Clone Stage Shell Script #!/usr/bin/env bash echo echo "Cloning Application" git clone "${GITHUB_URL:?}" . There’s not much happening in this stage. The script starts by outputting a brief description of the stage and then performing a git clone, which will place the application’s source code in the current working directory. Notice we’re using an environment variable here— GITHUB_URL. This—and others like it in subsequent stages—are expected to

71

Chapter 4

A Practical Example

exist wherever the pipeline is running. Stages will also source information from the pipeline.json file. You’ll see an example of this in the next stage. We’ll cover the pipeline.json file in detail later in this chapter.

The Build Stage In the second stage the pipeline will build the application. Each tech stack will be handled differently, but the end goal is the same: to create an artifact we can deploy inside a container. In the stages directory, you’ll find a file named 02_build.sh with the following code:

Listing 4-5.  The Netcore Build Stage Shell Script #!/usr/bin/env bash echo echo "Building Application" application_type=$(jq -r .application.type pipeline.json) case "${application_type}" in   "java")     mvn clean package     ;;   "netcore")     dotnet restore     dotnet build -c Release     ;;   "node")     npm install     ;;   *)

72

Chapter 4

A Practical Example

    echo "Unable to build application type ${application_type}"     exit 1     ;; esac This script uses jq to pull the value of application.type out of the pipeline.json file and assign it to application_type. This should be set to either java, netcore, or node if the pipline.json file is configured correctly. If application_type doesn’t equal one of these values, the script terminates with exit code 1.

Note  jq is a fantastic tool that’s packed with functionality! You can find a great tutorial on the official website at https://stedolan. github.io/jq/. For java applications, we’ll use Maven to clean the workspace, which ensures there aren’t any cached or outdated files lying around, and then compile the application. It’s important to remember that we require all java apps that come through the pipeline to support Maven. If the application_type is netcore, we’ll use the .NET Core CLI to perform a NuGet package restore by issuing the command dotnet restore app. Then, we’ll call dotnet build app -c Release to compile the source code using the Release configuration. Finally, if application_type is set to node, we’ll use the Node Package Manager to download and install any required dependencies. After that, we issue the npm run build command to compile everything.

73

Chapter 4

A Practical Example

The Test Stage The third stage of the pipeline handles test execution. You can find it in 03_test.sh.

Listing 4-6.  The Test Stage Shell Script #!/usr/bin/env bash echo echo "Testing Application" application_type=$(jq -r .application.type pipeline.json) enabled=$(jq -r .test.enabled pipeline.json) test_path=$(jq -r .test.path pipeline.json) if "${enabled}" then   echo "Skipped"   exit 0 fi case "${application_type}" in   "java")     mvn test     ;;   "netcore")     # The path to the test project must be set until     # https://github.com/Microsoft/vstest/issues/1129 is     # resolved.     dotnet test "${test_path}"     ;;   "node")

74

Chapter 4

A Practical Example

    npm run test     ;;   *)     echo "Unable to test application type ${application_type}"     exit 1     ;; esac The test stage uses the same pattern as our build stage, with one exception. Before kicking off any tests, it checks the test.enabled property in the pipeline configuration file. If it’s set to false, the stage is skipped. Assuming the stage is enabled in the configuration file, application_ type is evaluated and the appropriate command is executed to run the test suite. As in the build stage, unknown application types cause the stage to fail with error code 1. Using our sample projects, implementing this stage turns out to be fairly trivial. In our experience though, it tends to grow in complexity and even spawn completely new stages. For example, your organization may want to report on code coverage or require a certain percentage of tests to pass. Perhaps your teams have various suites of tests in several different repositories. If your teams write both unit tests and end-to-end tests, it might make sense to keep them in separate stages.

The Archive Stage Now that the project has been built and tested successfully, it’s time to package it into an artifact and place it somewhere safe. In your organization this might be Artifactory, Nexus, GitHub, or any number of other repositories. For our sample projects, we are using Docker Hub. If you’re unfamiliar with it, Docker Hub is simply a free, public repository where anyone can store Docker images. The archive stage can be found in 05_archive.sh. 75

Chapter 4

A Practical Example

Listing 4-7.  The Archive Stage Shell Script #!/usr/bin/env bash echo echo "Archiving Application" registry=$(jq -r .archive.registry pipeline.json) repository=$(jq -r .archive.repository pipeline.json) image="${registry}/${repository}:latest" docker login \     -u "${DOCKER_USERNAME?:}" \     -p "${DOCKER_PASSWORD?:}" \     "${registry}" docker build -t "${image}" . docker push "${image}" Like the build and test stages, the archive stage starts off by pulling some information out of the pipeline configuration file. We get the registry and repository values and then combine them to get the desired name of the image containing the application. For now, we’re just applying the latest tag, but in the next chapter we’ll discuss versioning your artifacts so they won’t be overwritten and can be uniquely identified later. Now that we have a name, we can build and push the image. The first step is to log in to the registry specified in the pipeline configuration file. Like GITHUB_URL, the DOCKER_USERNAME and DOCKER_PASSWORD variables will be passed into the container as environment variables. Because these are credentials and thus sensitive information, they shouldn’t be stored in the pipeline configuration file. Assuming we were able to login successfully, the next step is to build the image. Like we mentioned earlier, each of our sample applications is designed to be deployed inside a container, so each of them has an 76

Chapter 4

A Practical Example

associated Dockerfile. Our pipeline expects each application’s Dockerfile to be located in the project’s root directory. Of course, you could always offer more flexibility by introducing a new docker.dockerfilePath variable in your pipeline configuration file. In our experience, however, that hasn’t proved to be necessary. To kick off the build, we issue the docker build command passing the name of the image and the build context.

The Deploy Stage Finally, we reach the deploy stage. There are far too many deployment targets to cover here; every organization is different. For the purposes of this book, 05_deploy.sh will “deploy” to your local machine by simply running the newly created image.

Listing 4-8.  The Deploy Stage Shell Script #!/usr/bin/env bash echo echo "Deploying Application" container_port=$(jq -r .deploy.containerPort pipeline.json) registry=$(jq -r .archive.registry pipeline.json) repository=$(jq -r .archive.repository pipeline.json) image="${registry}/${repository}:latest" docker run -dp "${container_port}:${container_port}" "${image}" As mentioned before, each of our example apps are designed to be deployed inside a container. To do that, we need a few pieces of information from the pipeline configuration file: archive.registry, archive.namespace, and archive.repository. We put these together to form an image name, for example, docker.io/edwardsdl/sample-­ netcore:latest. This is the image we created in the archive phase.

77

Chapter 4

A Practical Example

Next, we execute the command docker run -dp "${container_port}: ${container_port}" "${image}". This runs the latest version of the image containing our application in “detached” mode—meaning in the background—and publishes the port the application is listening on. Once this is done, the deploy script terminates, which, since it is the last of the stage scripts, causes the pipeline container to exit.

Tip  While containerizing your applications isn’t necessary to create a generic pipeline, it certainly makes things easier. If your organization hasn’t explored the idea of containerized applications, I highly recommend doing so.

A Look at Our Build Containers One of the key features of our pipeline is that it executes entirely inside a Docker container. This allows us total control over our build environment. We can add or update dependencies easily, set environment variables as needed, or install software in custom locations—all without interfering with other applications’ build environments or workflows. Of course, in order to have a build container you must have a Dockerfile. In this section we’ll take a look at the three Dockerfiles we use for each of our three tech stacks. First let’s look inside Dockerfile.java. As you may have guessed, this is the Dockerfile we use to construct the image for our Java build containers.

Listing 4-9.  The Dockerfile for the Java Build Container FROM maven:3-jdk-8 RUN curl -fsSL get.docker.com | sh RUN apt-get update && apt-get install -y jq zip 78

Chapter 4

A Practical Example

COPY stages stages WORKDIR /app The Dockerfile is relatively simple. We use maven:3-jdk-8 as our base image because it comes out of the box with both the Java 8 JDK and Maven 3. Admittedly this base image makes the container a little bloated, but in our experience these containers tend to get fairly large anyway, so it’s not worth worrying about a few extra megabytes. Next, we install Docker inside the container. That probably sounds strange—it did to us the first time too! The reason is simple: our applications are designed to be deployed as containers and thus have Dockerfiles themselves. That means we need to issue docker build and docker push commands from inside our build containers.

Caution  We chose to install Docker using this method because it’s concise and easy to understand. However, it’s never a good idea to run scripts without examining them first. You can find a more secure method for installing Docker at https://docs.docker.com/ install/. Note  Using Docker inside of Docker is becoming a very common scenario. However, if you are new to this concept you can learn more here: https://blog.docker.com/2013/09/docker-can-now-­ run-within-docker/. The third line of our Dockerfile installs two packages: jq and zip. The first, jq, is a command line tool that’s great for parsing and transforming JSON data. It’s used extensively inside our stage scripts, as you saw earlier in the chapter. The second is zip. I’m sure you can guess what that does.

79

Chapter 4

A Practical Example

Next, we copy the pipeline stages into the container. Be aware: by copying your code in now, you’ll be required to recreate your build images when your stage scripts change. In your implementation you may decide to clone your stages into your container when it starts up. You’ll always be running the latest code, but it’s more difficult to determine which version of the pipeline created a given artifact. Finally, we set our working directory to /app. The pipeline will use this as its primary workspace. Application code will be cloned, built, tested, and packaged all within this directory. We use Dockerfile.netcore to create the build container for .NET Core projects, which looks very similar to its Spring Boot counterpart. In this case we use microsoft/netcore:2-sdk as our base image instead of maven:3-jdk-8. Otherwise, this file is exactly the same.

Listing 4-10.  The Dockerfile for the .NET Core Pipeline Image FROM microsoft/dotnet:2-sdk RUN curl -fsSL get.docker.com | sh RUN apt-get update && apt-get install -y jq zip COPY stages stages WORKDIR /app The Dockerfile for the Angular tech stack follows the same pattern as the previous two. In this case our base image will be node:9-stretch, which uses a recent version of Debian Linux and provides easy access to Node.js and NPM. We also install two additional dependencies: chromium and chromium-driver. These are used by our application’s test suite. We’ve named this file Dockerfile.node.

80

Chapter 4

A Practical Example

Listing 4-11.  The Dockerfile for the Angular Pipeline Image FROM node:9-stretch RUN curl -fsSL get.docker.com | sh RUN apt-get update && apt-get install -y \     chromium \     chromium-driver \     jq \     zip COPY stages stages WORKDIR /app

R  unning the Pipeline Now that we’ve gone over the sample applications, stage scripts, and Dockerfiles, it’s time to run our pipeline. To begin, we’ll run it locally. Afterwards we’ll show you how to port it to several popular CI tools. Before you move on, we suggest you fork one of our sample projects. These applications have been thoroughly tested, and with a few simple modifications you’ll be able to run them through the pipeline locally and in the cloud. In addition, as part of the sign-up process, both Travis CI and CircleCI will request access to your GitHub account in order to streamline the setup process and start builds when new code is committed to a linked repository. In the end, we think it’ll be easier for you to use one of our applications than build your own. For the rest of the chapter we’ll be using our sample .NET Core project. You can find it at https://github.com/Apress/generic-pipelinesusing-docker. If you’re using GitHub, forking our sample projects is easy! Just navigate to its repository on GitHub and click the “Fork” button in the top right. This will create a copy of the repository in your account. 81

Chapter 4

A Practical Example

Using the Command Line Running the pipeline from the command line is pretty straightforward. First you build the Docker image for the pipeline you want to use by issuing the docker build command. Next you run the image passing along all of the required arguments to mount the Docker socket and set the appropriate environment variables. For example, if you want to build the sample .NET Core app, you’d issue the following commands from the root of the directory containing your application:

Listing 4-12.  Building and Running the Pipeline docker build -t pipeline -f . docker run \   -v /var/run/docker.sock:/var/run/docker.sock \   --env GITHUB_URL=https://github.com/edwardsdl/sample-netcore. git \   --env DOCKER_USERNAME=AzureDiamond \   --env DOCKER_PASSWORD=hunter2 \   pipeline \   /stages/00_run.sh Obviously, the values for GITHUB_URL, DOCKER_USERNAME, and DOCKER_ PASSWORD are placeholders. You need to replace them with the path for your fork and your Docker Hub credentials. Remember to use your Docker repository URL as well! Once you’ve run the docker run command, you’ll see the pipeline go through each of the stages we described earlier. Your output should look like that in Figure 4-4.

82

Chapter 4

A Practical Example

Figure 4-4.  Running the pipeline from the command line In its final stage, the pipeline will “deploy” the sample .NET Core application locally. When you issue the docker ps command as in Figure 4-5, you’ll see you have one container running—the one running the sample application! Using your browser, navigate to http:// localhost:5000/api/values to verify it’s working.

Figure 4-5.  Inspecting the deployed application

Using IntelliJ IDEA CE If you’re more comfortable using an IDE, Intellij IDEA has wonderful support for building Docker images. You can use the “Docker Integration” plugin, which is incredibly helpful. Setup is a little more involved, but once you’re done you have a powerful development environment at your fingertips. 83

Chapter 4

A Practical Example

Tip  If you run into trouble when adding new Docker configuration profiles, check out JetBrains’ help page at www.jetbrains.com/ help/idea/run-debug-configuration-docker.html. To begin, we will create a few new configurations in our IDE. In the top right corner, click the “Select Run/Debug Configuration” drop-down box and click “Edit Configurations…” as in Figure 4-6.

Figure 4-6.  Creating a new configuration On the Run/Debug Configuration window, create a new Docker configuration by clicking on the “Add New Configuration” button in the top left. Under the “Docker” menu item, select “Dockerfile” as shown in Figure 4-7.

Figure 4-7.  Adding a new Dockerfile configuration 84

Chapter 4

A Practical Example

The first configuration we’ll create will allow us to run the .NET Core pipeline. We’ll name this new configuration, “Run .NET Core Pipeline.” In the Dockerfile drop-down box, select “Dockerfile.netcore.” Now, select the checkbox labelled “Run built image” and set the container name to netcore-pipeline. Next, in the executable section, set the command to /stages/00_run.sh. Now we need to mount the host’s Docker socket inside the container. This is what makes it possible for us to execute Docker commands inside the container. To do this, click the button labelled, “…” to the right of the bind mounts textbox. In the “Bind Mounts” window, add a new bind mount setting both the host path and the container path to /var/run/ docker.sock (Figure 4-8).

Figure 4-8.  Mounting the Docker socket

Tip  If you want to learn more about the history of issuing Docker commands inside a container and mounting the Docker socket, Jérôme Petazzoni has written an excellent blog post at https:// jpetazzo.github.io/2015/09/03/do-not-use-docker-in-­ docker-for-ci/. Next, we’ll need to add a few environment variables (Figure 4-9), specifically GITHUB_URL, DOCKER_USERNAME, and DOCKER_PASSWORD. The value of GITHUB_URL will be DOCKER_PASSWORD will be your Docker Hub username and password, respectively. The values we use here are placeholders and won’t work for you.es we use here are placeholders and won’t work for you. 85

Chapter 4

A Practical Example

Figure 4-9.  Adding environment variables

Note  If you have forked the sample GitHub repo, remember to use your GITHUB_URL in these examples! After you’ve added the environment variables, click the OK button to return to the Run/Debug Configurations screen. Confirm that your settings look like those in Figure 4-10, and then click “Apply.”

Figure 4-10.  Adding the Run .NET Core Pipeline configuration 86

Chapter 4

A Practical Example

Next, we’ll repeat this process to create additional configuration profiles for our Java and Node pipelines. Most of the process is identical, but you’ll want to be sure to choose the correct Dockerfile and GITHUB_URL values. Reference Figures 4-11 and 4-12 to ensure your settings are correct.

Figure 4-11.  Adding the Run Java Pipeline configuration

87

Chapter 4

A Practical Example

Figure 4-12.  Adding the Run Node Pipeline configuration Once you’ve added the last configuration profile, click the OK button to close the window. Now the Select Run/Debug Configuration drop-­ down box should contain three items: Run .NET Core Pipeline, Run Java Pipeline, and Run Node Pipeline (Figure 4-13).

Figure 4-13.  Listing the newly created configuration profiles 88

Chapter 4

A Practical Example

M  oving to the Cloud Now that we’ve seen the pipeline work on our local machine, it’s time to get it working using a real continuous integration tool. We’ll start by forking one of our sample projects in GitHub. Next, we’ll show you how to run our pipeline in Travis CI by converting the 00_run.sh script to a .travis.yml file. Finally, we’ll walk you through porting our pipeline from Travis CI to CircleCI.

Moving the Pipeline to Travis CI As mentioned, we’ll be showing you how to use the pipeline with two continuous integration platforms. Up first, we’ll be looking at Travis CI. Travis CI was one of the first—if not the first—CI/CD SaaS offering. It provides an intuitive interface, free accounts for open source projects, good documentation, and a large number of integrations. Because of this, Travis CI is wildly popular, especially amongst open source projects.

Creating a Travis CI Account Before we can move our pipeline to Travis CI, we’ll need to create a new account. If you don’t need help, skip ahead to the next section. Open a browser and navigate to https://travis-ci.org, then click the button labeled, “Sign Up” (Figure 4-14). If prompted, enter your GitHub username and password. If you are already signed in to GitHub, you won’t be asked to do so again.

Caution  Check that top-level domain! You want https:// travis-ci.org not https://travis-ci.com. The latter is for paid projects only!

89

Chapter 4

A Practical Example

Figure 4-14.  The Travis CI homepage Grant Travis CI access to your email address and permission to add new webhooks to your repositories. The service will use this to help you set up new builds, to trigger builds when new code is committed to your repositories, and notify you when things go wrong. After granting Travis CI access, you’ll be dropped on a “Getting Started” page. Take a minute to read through this page. It details the steps required to add a new repository and start building it.

Adding a New Repository Now that your account has been created, you’re ready to add a new repository to Travis CI. This is where it all comes together. Once you’re done setting up the repository, you’ll get to see the generic pipeline in action. Head to your profile page by using the link in the instructions. Alternatively, you can click the “Profile” link in the drop-down menu located in the top right of the navigation bar, as shown in Figure 4-15.

90

Chapter 4

A Practical Example

Figure 4-15.  The “Getting Started” page On your profile page, search for “sample.” This will return a list of any of our sample projects you forked. Click the gray toggle switch to the left of “sample-netcore” to allow Travis CI to integrate with the repository (Figure 4-16).

Figure 4-16.  Adding the .NET Core sample repository to Travis CI Now click on the “Settings” button to the right of the toggle switch to navigate to the settings for this repository. Before we start the first build, we need to tell Travis CI what environment variables to pass to our container (Figure 4-17). These will be the same values you used when running the pipeline locally: DOCKER_USERNAME, DOCKER_PASSWORD, and GITHUB_URL.

91

Chapter 4

A Practical Example

Figure 4-17.  Adding environment variables to the build

Caution Make sure to toggle the “Display value in build log” switch to the OFF position for the DOCKER_PASSWORD environment variable. You don’t want your password showing up in the build log! Now everything is ready for us to kick off our first build! Click the “More options” button on the top right side of the page and select “Trigger build” from the drop-down menu (Figure 4-18).

Figure 4-18.  Triggering the first build You’ll be prompted to select a branch and add a custom commit message and custom configuration. Essentially Travis CI is simulating a commit to your Git repository; nothing is actually pushed. This is actually an incredibly useful feature, especially when you’re getting started! It allows you to quickly test changes to your configuration file without having to go through the change, commit, push loop over and over again. For now, just click the “Trigger custom build” button at the bottom (Figure 4-19).

92

Chapter 4

A Practical Example

Figure 4-19.  Triggering a custom build

Note  For more information about this feature, check out the blog post announcing its release at https://blog.travis-ci. com/2017-08-24-trigger-custom-build. You’ll be redirected to a page showing you a detailed, real-time status of your first build (Figure 4-20). If you scroll through the logs at the bottom, you’ll notice some familiar messages! The application is being cloned, built, tested, and archived just like it was when you ran the pipeline locally!

93

Chapter 4

A Practical Example

Figure 4-20.  Running the pipeline in Travis CI

A Look at the Travis CI Configuration File So how does Travis know how to run our pipeline? It uses the .travis.yml file located in the root of our Git repository. Let’s take a look at what’s inside!

Listing 4-13.  The Travis CI Configuration File services:   - docker before_script:   - |     docker run -it -d \       -v /var/run/docker.sock:/var/run/docker.sock \       -e DOCKER_USERNAME=${DOCKER_USERNAME} \       -e DOCKER_PASSWORD=${DOCKER_PASSWORD} \       -e GITHUB_URL=${GITHUB_URL} \       --name netcore-pipeline \       edwardsdl/netcore-pipeline:latest script:   - docker exec netcore-pipeline /stages/01_clone.sh   - docker exec netcore-pipeline /stages/02_build.sh   - docker exec netcore-pipeline /stages/03_test.sh   - docker exec netcore-pipeline /stages/04_archive.sh 94

Chapter 4

A Practical Example

It turns out the configuration file looks pretty similar to our 00_run.sh script. That’s by design! One of the primary benefits of this architecture is the ease with which you can move from one CI platform to another. The services section describes any custom services—like MongoDB, Memcached, or RabbitMQ—your build requires. Travis CI will include these in your build environment. In our case, we ask that Docker be installed. The before_script section lets us run any last-minute commands before the build really gets started. We’ll use it to pull and run the latest version of the netcore-pipeline Docker image. Just like when we ran it locally, we mount the Docker socket and pass the DOCKER_USERNAME, DOCKER_PASSWORD, and GITHUB_URL environment variables to the container.

Note  You may have noticed our docker run command is preceded by a vertical bar (|). This is called the literal block scalar style and it allows our command to span multiple lines. If you’re fascinated by formal language grammars, check out the YAML specification at http://yaml.org/spec/1.2/spec. html#id2795688. The script section is where we instruct Travis CI to run the stages we’ve included inside the container. Starting at 01_clone.sh, we simply work our way through each script until we’re done. If any stage fails, Travis will stop execution and mark the build as failing.

Note  For more information about configuring your Travis CI build, visit https://docs.travis-ci.com/user/customizing-the-­ build/.

95

Chapter 4

A Practical Example

Running the Pipeline in CircleCI It’s not unusual for organizations to transition from one CI/CD platform to another. Even upgrading from one version to another can be a huge undertaking. In this section, we’ll see what it takes to move our pipeline from Travis CI to CircleCI. CircleCI is one of the world’s most popular CI/CD platforms. Like Travis CI, CircleCI is hosted in the cloud, offers free accounts, and is very easy to get up and running. On top of all that, it’s arguably an even better fit for our pipeline than Travis CI, as it was built with containerized pipelines in mind!

Creating a CircleCI Account Tip  For more information about getting started with CircleCI, visit the 2.0 documentation page at https://circleci.com/ docs/2.0/. This section will guide you through the process of creating an account. The process is fairly straightforward and very similar to that of Travis CI. If you’ve got experience working with CircleCI or are confident you don’t need help, feel free to skip to the next section. To begin, open a browser and navigate to https://circleci.com. In the top right corner, click the button labelled “Sign Up” (Figure 4-21).

Figure 4-21.  The CircleCI home page 96

Chapter 4

A Practical Example

CircleCI will ask you to decide whether you want to sign up using GitHub or BitBucket (Figure 4-22). We’ll be using GitHub; but if you want to use BitBucket, it should be easy to follow along, as the process is almost identical.

Figure 4-22.  Signing up with CircleCI If prompted, enter your GitHub username and password (Figure 4-23). If you are already signed in to GitHub, you won’t be asked to do so again.

Figure 4-23.  Signing up using GitHub 97

Chapter 4

A Practical Example

Next, CircleCI will request access to your email address and read access to your repositories (Figure 4-24). Like Travis CI, this information will be used to help set up your projects and notify you when your builds break.

Figure 4-24.  Authorizing CircleCI to access GitHub repositories

Creating a New CircleCI Project After creating your account, you’ll be sent to your dashboard. This is where you’ll go to view the latest information on all your builds. At the moment, however, you have no projects. Instead of build information, you’ll be presented with a page welcoming you to the platform and directing you to add a new project (Figure 4-25). That sounds like a great idea!

Figure 4-25.  The builds screen 98

Chapter 4

A Practical Example

To begin, click the blue “Add projects” button in the center of the page. This will take you to a list of all the repositories in your GitHub or BitBucket account. In the list of repositories, find the fork you created of the sample .NET Core repository and click the “Set Up Project” button (Figure 4-26).

Figure 4-26.  Adding a new project

Tip Make sure the “Show Forks” checkbox is selected, otherwise your repository won’t show up in the list. On the Add Project screen, select Linux as the operating system and choose “Other” under the Language section (Figure 4-27). Out of the box, CircleCI comes with the ability to intelligently build and test projects written using some popular technologies. While this is a wonderful feature, we won’t be using it for this or any of our other sample applications. All of our projects include a configuration file that tells CircleCI exactly what to commands to execute. We’ll take a closer look at this file a little later.

Figure 4-27.  Setting up the sample .NET Core project 99

Chapter 4

A Practical Example

After configuring the project, click the “Start building” button to create the CircleCI project and start a build. Once you’ve started the build, you’ll be taken to a new page showing you its status (Figure 4-28). The build starts by pulling down our pipeline image, in this case edwardsdl/netcore-pipeline:latest, and then it begins executing the instructions found in the sample project’s config. yml file. Almost immediately however, the build fails! Checking the output of the Clone section makes the problem obvious: stages/01_clone.sh: line 6: GITHUB_URL: parameter null or not set. We never set the GITHUB_URL environment variable!

Figure 4-28.  The first build of the sample .NET Core project To fix this, you’ll need to navigate to the build settings for the project and set a few environment variables (Figure 4-29). Go to the settings page by clicking the button with the gear shaped icon at the top right. Then click the “Environment Variables” link under the “Build Settings” section (Figure 4-30). Next use the “Add Variable” button to add three 100

Chapter 4

A Practical Example

new environment variables: GITHUB_URL, DOCKER_USERNAME, and DOCKER_ PASSWORD. Unlike Travis CI, the values you set here cannot be exposed in plain text.

Figure 4-29.  Projects page in Circle CI

Figure 4-30.  Adding environment variables Now that we’ve set our environment variables, go back to the list of builds by clicking the “Builds” button at the top of the navigation bar on the left side. You should see a single row giving a summary of our failed build (Figure 4-31). Find the “rebuild” link on this row and click it.

Figure 4-31.  Rebuilding the project 101

Chapter 4

A Practical Example

After you click the “rebuild” link, you’ll be taken back to the build details page (Figure 4-32). This time the build should complete successfully! If you examine the build actions list, the output should look familiar. All of our stages are listed, and the output is the same as we saw when running the pipeline locally!

Figure 4-32.  Running the job a second time

A Look at the CircleCI Configuration File So how did CircleCI know how to execute our stages? Take a look at one of our sample projects and you’ll find a hidden directory named .circleci. Inside this directory you’ll find a single file named config.yml. This is the CircleCI configuration file, and it tells CircleCI everything it needs to know. In the sample .NET Core app, this YAML file looks like this:

102

Chapter 4

A Practical Example

Listing 4-14.  The sample-netcore config.yml File version: 2 jobs:   build:     docker:       - image: edwardsdl/netcore-pipeline:latest     steps:       - setup_remote_docker:           docker_layer_caching: true       - run:           name: Clone           command: /stages/01_clone.sh       - run:           name: Build           command: /stages/02_build.sh       - run:           name: Test           command: /stages/03_test.sh       - run:           name: Archive           command: /stages/04_archive.sh The file starts by specifying the Docker image to use when executing the pipeline steps. Those steps are defined in the next section. Most of this should look familiar—the clone stage is run first, followed by the build, test, and archive stages. The setup_remote_docker step is new though. This is what allows us to run Docker commands inside our container.

Note  You can find more information about CircleCI’s config.yml file at https://circleci.com/docs/2.0/configuration-­ reference/. 103

Chapter 4

A Practical Example

Overview In this chapter we took a deep dive into all the components necessary to build a simple, albeit fully functional, generic pipeline. We gave examples of several critical pipeline stages: clone, build, test, archive, and deploy. We then took a look at the Dockerfiles we use to create our build environments. We then put it all together and used our pipeline to build and deploy several sample applications on our local machines. Taking it one step further, we showed you how easy it is to migrate the pipeline from your local machine to Travis CI and then to CircleCI. In the next chapter, we cover more advanced topics and show you how to tackle some of the problems that tend to show up in real-world implementations.

104

CHAPTER 5

Moving Beyond the Basics We’ve covered a lot in this book so far. You’ve seen the transition from monolithic one pipeline per application style CI/CD processes all the way to a fully generic implementation. At this point we’ve created a truly generic and platform-independent pipeline that can deploy any number of languages expanded upon and even look at implementations in some other scripting languages. The goal of this chapter is to expand on the knowledge gained so far, and to further explore and emphasize the power of generic pipelines. By the end of this chapter we’ll have seen what the generic pipeline may look like in a true enterprise implementation. We’ll also look at the role each team plays in its implementation and how a federated DevOps team may come into play. We’ll even explore how you can empower your development teams to take control of the pipeline and run custom scripts.

Additional Stages and Steps So far we’ve built a somewhat basic pipeline with a minimum number of stages. While this is more than enough to get you up to speed on how to build a generic pipeline, we’ve only scratched the surface of what

© Brandon Atkinson, Dallas Edwards 2018 B. Atkinson and D. Edwards, Generic Pipelines Using Docker, https://doi.org/10.1007/978-1-4842-3655-0_5

105

Chapter 5

Moving Beyond the Basics

the pipeline would look like in a production environment. Our pipeline consists of the following stages: •

Clone

•

Build

•

Unit Test

•

Deploy

There are many organizations that would be thrilled to simply have a pipeline with these stages. We talked in Chapter 1 about how there are companies, even today, that still deploy code manually. A simple pipeline like the one in this book would be a godsend for them. However, for companies that truly embrace DevOps and automated CI/CD pipelines, this is simply not enough to meet their needs. A proper enterprise-ready pipeline requires stages that cover much more, including:

106

•

Static Code Analysis: This stage would perform code coverage checks and linting on your application source code.

•

Security Scan: This stage would perform security and vulnerability scanning on the source code. This stage is crucial to ensure the application is secure and not vulnerable to cyberattacks.

•

Licensing Scan: This stage would perform licensing checks on any third-party code incorporated into the application. Some organizations have strict rules around the licensing of code and may restrict what can be deployed.

Chapter 5

Moving Beyond the Basics

•

Artifact Storage: This stage would take the built application code and store it in a repository for later use. This allows for easy promotion of code that has been built and tested and has passed all the necessary checks before being released to Production.

•

End to End Tests: This stage would perform any automated tests against the deployed application. This is where you can initiate test suites to test any functionality that is required to go to Production.

•

Performance Tests: This stage would kick off any performance tests that are needed to exert a load on the application. This could be combined with End to End tests if needed.

That is a total of ten stages including the ones we included in the earlier chapters! As you can see, an enterprise CI/CD pipeline can be quite complex. Each stage may interact with one or more systems, one or more times! It’s quite easy to imagine some of these stages making multiple calls to the same system as well. For instance, you may need to submit source code for a security scan, and then make another call to check the results. We haven’t even talked about pipelines that may need to also provision infrastructure as part of their deployments. Some enterprises use the deployment of new code to also create a new server stack in the process. This is common with cloud providers like AWS. If you have an application that runs on EC2 instances, for example, you may want to provision a new instance with a fresh copy of the underlying OS on each deployment. This gives you an opportunity to get the latest security patches on the OS each time you deploy. The number of stages you have can be limitless. However, the ones listed earlier would be the most commonly used. The main takeaway is that your pipeline may consist of multiple stages and may need to handle

107

Chapter 5

Moving Beyond the Basics

multiple languages. The nice part is, outside of the build and unit test stages, most of the remaining stages are generally not language specific. This makes building and testing those much easier. Again, we must stress that standards are your friends with regard to all your stages. The generic pipeline is built to handle any number of languages passed to it, so your stages must handle those accordingly. If you’re using enterprise-grade tools for scanning, this is probably not going to be an issue. For instance, more static code scanning tools can handle pretty much any code language you can throw at it. The goal in the later stages of the pipeline is to continue to enforce standards and further push the generic pattern.

Other Scripting Languages While shell scripts are very powerful and almost guaranteed to run anywhere, you may find yourself in a situation where you either don’t want to use Shell scripts, or can’t. One of the benefits of the pattern covered in this book is that it can be applied to any scripting language you wish. In fact, it doesn’t even have to be a scripting language! You could accomplish the same goals using any language: for instance, Golang may be a nice option. We talked at length in the book about why we chose Shell over other languages. Scripting languages are interpreted rather than compiled; this provides many benefits. For instance, using Shell allows us to pretty much run in any Linux environment without issues. Bash would be another excellent choice with the introduction of native Bash in Windows environments. This is what makes scripting languages a great choice for this pattern.

108

Chapter 5

Moving Beyond the Basics

Note  Windows 10 is required for native Bash in Windows. The Windows Subsystem for Linux allows you to run Linux environments without the overhead of a virtual machine. You can learn more at the following link: https://docs.microsoft.com/en-us/ windows/wsl/about. Using a compiled language like Golang or C# is not a horrible choice, but it introduces additional hurdles. Aside from needing to recompile your code each time you make a change, it also requires you to install a runtime, deal with versioning problems, and work around platform specific issues. If you needed to jump from a Windows environment to a Linux environment, you would need to compile it separately for each one. If you were using Bash scripts, you would not need to be bothered; simply pick up the code and change operating systems. We’ve explored Shell scripts in detail. Now let’s look at the same examples but in other languages. We’ll basically show each stage we’ve seen already but presented in another language.

P  owerShell Listing 5-1 shows the build stage in PowerShell.

Listing 5-1.  PowerShell Build Stage Script $application_type = application.type switch ( $application_type ) {     java     {         mvn clean package     } 109

Chapter 5

Moving Beyond the Basics

    netcore     {         dotnet restore         dotnet build -c Release     }     node     {         npm install     }     default     {         Write-Host "Unable to build application type $application_type."         exit 1     } } This script assumes that you have parsed your JSON configuration file and the application type is accessible via an object. Listing 5-2 shows the test stage script.

Listing 5-2.  PowerShell Unit Test Stage Script $application_type = application.type $test_path = test.path switch ( $application_type ) {     java     {         mvn test     }

110

Chapter 5

Moving Beyond the Basics

    netcore     {         dotnet test $test_path     }     node     {         npm run test     }     default     {         Write-Host "Unable to test application type $application_type."         exit 1     } } You should be noticing a familiar pattern at this point. Even with PowerShell, the same patterns from the Shell scripts are in place. As with the build stage, your JSON configuration file has been parsed and is available to your script. Listing 5-3 illustrates the archive stage.

Listing 5-3.  PowerShell Archive Stage Script $registry = archive.registry $namespace = archive.namespace $repository = archive.repository $image = "$registry/$namespace/$repository:latest" docker login `     -u $DOCKER_USERNAME `     -p $DOCKER_PASSWORD `     $registry docker build -t $image docker push $image 111

Chapter 5

Moving Beyond the Basics

Finally, listing 5-4 illustrates the deploy stage.

Listing 5-4.  PowerShell Deploy Stage Script $registry = archive.registry $namespace = archive.namespace $repository = archive.repository $image = "$registry/$namespace/$repository:latest" docker run -dP $image What is truly amazing here are the similarities between the Shell script and PowerShell. Even though they are completely different scripting languages, the code looks nearly identical. Again, this is the power of standards coming into play. By keeping your code lean and sticking to the core commands in each language, you will find your scripts are small, lightweight, and ultimately easily transferrable to another language. PowerShell may not be the best example of porting your pipeline code to another language. For this book we’ve used Shell scripts running in Linux environments. A more realistic conversion would be to a language that gives you more power than Shell, but is still very Linux friendly. So now, let’s look at these scripts written in Python.

P  ython Listing 5-5 shows the build stage in Python.

Listing 5-5.  Python Build Stage Script application_type = application.type if application_type == 'java':     subprocess.run(['mvn', 'clean', 'package'], check=True) elif application_type == 'netcore':     subprocess.run(['dotnet', 'restore'], check=True)

112

Chapter 5

Moving Beyond the Basics

    subprocess.run(['dotnet', 'build', '-c', 'Release'], check=True) elif application_type == 'node':     subprocess.run(['npm', 'install'], check=True) else     raise Exception(f'Unable to build application type {application_type}.') Much like the Shell script and the PowerShell script, we simply determine the application type and perform the standard build commands for each language. Listing 5-6 illustrates the unit test stage.

Note This is a slimmed down Python script. It does not include the shebang or any import statements that would be required. We’re focusing solely on the logic in the script. The version of Python shown here is 3.6. Listing 5-6.  Python Unit Test Stage Script application_type = application.type test_path = test.path if application_type == 'java':     subprocess.run(['mvn', 'test'], check=True) elif application_type == 'netcore':     subprocess.run(['dotnet', 'test', test_path], check=True) elif application_type == 'node':     subprocess.run(['npm', 'run', 'test'], check=True) else     raise Exception(f'Unable to test application type {application_type}.')

113

Chapter 5

Moving Beyond the Basics

By now the beauty and simplicity of the pattern should be taking hold. You’ve seen Shell, PowerShell, and now Python. Even if you don’t regularly use these languages, by sticking with standards and keeping things simple, the code is very easy to read. More importantly, it’s very easy to port should that need ever arise. Listing 5-7 shows the archive stage.

Listing 5-7.  Python Archive Stage Script registry = archive.registry namespace = archive.namespace repository = archive.repository image = f'{registry}/{namespace}/{repository}:latest' subprocess.run(['docker', 'login', registry,     '-u', docker_username, '-p', docker_password], check=True) subprocess.run(['docker', 'build', '-t', image], check=True) subprocess.run(['docker', 'push',  image], check=True) Finally, listing 5-8 illustrates the deploy stage.

Listing 5-8.  Python Deploy Stage Script registry = archive.registry namespace = archive.namespace repository = archive.repository image = f'{registry}/{namespace}/{repository}:latest' subprocess.run(['docker', 'run', '-dP', image], check=True) Now imagine your team decides that all this Shell and PowerShell and Python stuff is for the birds; they want to use Golang! Not a problem, you say! At this point it would be a trivial task to research how to issue commands via Golang, and rewrite the pipeline. Of course, this all

114

Chapter 5

Moving Beyond the Basics

depends on how complex your pipeline is. However, by sticking to standards and not having a lot of “one-off” code, this task becomes much easier.

Custom Script Hooks The most important rule of the generic pipeline, without a doubt, is to strictly adhere to standards. Standards are your friends; do not deviate from them. They are the only thing that makes a generic pipeline work, period. Standards keep the playing field level and ensure that all applications are handled in the exact same manner, every time. This is invaluable when it comes to placing new languages on the pipeline, debugging, as well as adding new features. Without a doubt teams will come to you and ask (or even expect) for one-off changes that may apply only to their application. Do not give in to these requests! It may seem benign at first, and you’re happy to oblige. But over time all those tweaks begin to add up. Bugs start to creep in and it becomes increasingly difficult to diagnose and fix issues. The cost and effort required to maintain the overall pipeline skyrockets. And let’s not forget, a fix here may break another application over there. By making all development teams play by the same rules, you ensure everyone gets treated the same and the pipeline remains very stable. You’re going to make some people unhappy, and you may be faced with having to turn some teams away. If a team has an application that requires a lot of custom work, you may have to say “sorry, build your own pipeline.” This is life with a generic pipeline. However, being a DevOps team, you don’t want to be too rigid. After all, the development teams are your customers. And we want to have happy customers! So what options do we have if want to keep standards in place but also give the teams some flexibility in customizing things. You can offer your teams script hooks. 115

Chapter 5

Moving Beyond the Basics

Script hooks are simply an opportunity for teams to inject their own functionality via scripts into the pipeline. This can be a happy medium in keeping the pipeline clean from a lot of custom logic and allowing your development teams to do custom stuff. Let’s look at what script hooks look like and how you can incorporate them into your pipeline.

Pre- and Postpipeline Hooks Common areas for custom hooks are at the beginning and end of the pipeline. This allows a development team to define custom actions that are to be performed before or after their application is built or deployed. Some common actions may include: •

Installing any custom software, plug-ins, packages, etc. that are required by the application

•

Performing any clean-up actions needed by a new version of the application

•

Performing any data migrations or clean-up that may be needed

What is nice about these hooks is while they give development teams more flexibility, they’re also quite easy for the DevOps team to implement. Providing a hook at the beginning and end of the pipeline is accomplished by simply running a Shell script that is provided by the development team. Figure 5-1 illustrates the flow.

116

Chapter 5

Moving Beyond the Basics

Figure 5-1.  Implementing pre- and post scripts in the pipeline As Figure 5-1 shows, we’re still running inside a container specific to our language. Nothing has changed in that regard. However, you now are executing custom scripts at the beginning and end of the pipeline. In this scenario the development teams would provide a directory in their code repository containing the scripts. The naming convention of the directory and scripts would be standardized for the pipeline. Imagine a directory structure like the following: •

Code Repo •

/src

•

/pipeline_scripts ■■ pre.sh ■■ post.sh

•

/assets

•

/scripts

•

index.html 117

Chapter 5

Moving Beyond the Basics

This is obviously a simplistic application, but let’s just focus on the structure. In this scenario there is a “pipeline_scripts” folder that contains any Shell script the developers wish to run during the pipeline execution. "pre.sh" is run at the beginning of the pipeline, and "post.sh" is run at the end. This naming convention allows for any development team to include custom scripts. The pipeline will simply look in the agreed upon directory and, if the scripts are present, execute them. While this option provides a lot of flexibility to your development teams, its power is also its weakness. The DevOps team forfeits a lot of control with this pattern. When development teams first begin to utilize this feature, many things may go wrong. Luckily, you’re running inside a container that will be destroyed at the end of the pipeline execution. This provides a good number of protections and eases some concerns. If a team commits a bad hook script and wrecks the pipeline execution, we can simply throw away the container and try again. This is the beauty of running in a containerized environment. However, there are still a few guidelines that should be followed when implementing pipeline hooks: •

118

Run Custom Scripts with Least Privilege: There will be many stages in the pipeline that require logging into third-party systems or performing actions that require higher access. This is fine if you wrote the code and know what it’s doing. It becomes more challenging when you’re running custom scripts. Always run custom scripts with a lower level account that has fewer permissions. The last thing you want is a stray script doing horrible things with an administrator account.

Chapter 5

Moving Beyond the Basics

•

You Wrote It, You Own It: When providing this feature to teams, it’s extremely important to limit the amount of support your team provides when issues arise from custom scripts. The main reason is you will quickly become a support team for scripts you did not write. You must be firm and inform teams that if they choose to write custom scripts, they must support them. It’s your job to provide proper logging so they know what went wrong.

•

Always, Always Run Inside a Container: These scripts will most likely do things that change the environment, install packages, etc. You want to ensure this happens in a container so it does not affect your platform or other pipeline executions.

•

Trust No One, and Log Everything: Don’t trust that any script is safe. Remember the first point, run with fewer permissions, and log everything the script does!

If that list didn’t scare you off, then it’s time to talk about taking this pattern one step further with stage hooks.

Stage Hooks Pre- and postscript hooks provide a great deal of functionality. In fact, it’s rare to have this level of control over how the pipeline executes. And even when development teams do have a lot of pipeline control, it’s administered via normal work processes. A feature request is raised with the DevOps team who then implements a change. By using stage hooks, this process is turned on its head.

119

Chapter 5

Moving Beyond the Basics

Supporting pre- and post hooks will encourage teams to add more custom functionality at the stage level and less custom functionality in the before and after hooks. Common pre- and posthook actions are similar to before and after hook actions. They typically include: •

Installing additional packages that are needed for build and unit tests

•

Using a different unit test framework than what is currently supported by the DevOps team

•

Running additional tests, then combining test coverage reports

Just like with the pre- and post scripts, the custom stage scripts get picked up and run before each stage, as shown in Figure 5-2.

Figure 5-2.  Build and Unit Test custom scripts being injected

120

Chapter 5

Moving Beyond the Basics

In this scenario, we provide the development teams with a prebuild and preunit test script hook. These would be the most common areas for hooks to be present. For instance, it’s unlikely to need any kind of hook for static code analysis, security scans, or deployments. Those are stages that are sacred and should not be altered. Build and Unit Test, however, are areas in which the teams may have legitimate needs for custom functionality. As with pre- and post scripts, standardization is your friend. The application repo may look like the following: •

Code Repo •

/src

•

/pipeline_scripts ■■ pre.sh ■■ post.sh ■■ pre-build.sh ■■ pre-test.sh

•

/assets

•

/scripts

•

index.html

The script naming here is subtle but very important. By prefixing the scripts with “pre” it’s an indicator to the development teams that this is a prebuild script only. You still intend to execute your pipeline code, and you should. These are not meant to be full overrides of your pipeline code, although you could certainly do that if you wish. However, it’s been our experience that doing that invites a lot of support requests and broken pipeline executions. Custom script hooks are actually meant to provide the DevOps team with a bit of breathing room. It allows you to give your development teams

121

Chapter 5

Moving Beyond the Basics

options for expanding the capabilities of the pipeline to meet their needs, but in a controlled setting. If done properly, you can keep your standards-­ based pipeline running smoothly while giving your development teams the flexibility to tailor parts of the pipeline to their specific needs.

Where to Go from Here You’ve now been introduced to a shared generic pipeline with Docker. You’ve seen how you can extend its functionality and provide your development teams with all the features they need and then some. So where do you go from here? Hopefully we’ve shown you some things that you can practically apply to your own pipelines. Even if you don’t go out and refactor your pipelines to be generic across the board, you may have seen some things that can help in your day to day activities. This book was written from practical knowledge from folks who’ve been down this road. If you decide to embark on the truly generic shared pipeline, remember these tips:

122

•

Standards, Standards, Standards: We were absolutely a broken record with this, but it’s true. The only way a generic pipeline will succeed is with clearly defined standards and only one way to do things (i.e., build, test, etc.). The more customization you place in the pipeline, the harder it will be to maintain and troubleshoot.

•

Say No, and Often: Development teams will always ask for more—new features, customization requests, and special exceptions for their team or project. You have to say no more often than you say yes. A generic pipeline is meant for all teams to use, and if that is the case it must be truly generic. Learn to say no; that word is one of your best friends.

Chapter 5

•

Moving Beyond the Basics

Over Communicate with Teams: As you build the pipeline and maintain it, you’ll need to make changes from time to time. Some of these are big; some go unnoticed. It’s very important to over communicate with your development teams. They are putting a lot of trust in your pipeline and they want to know what’s going on. Even if there is a small change, publicize it. Ensure everyone knows what is happening with upcoming changes long before you do them.

Now you’re ready to tackle the generic pipeline. By using Docker and strictly adhering to standards, you can deliver a CI/CD experience unlike most others. When done properly, you will not only support your development teams but quite possibly the whole organization. Don’t be surprised if teams come knocking at your door to get on board. Embrace the change, and you and your team can deliver great things.

123

Index A Alpine containers, 40 Angular 5 application, 66 configuration file, 69 Dockerfile, 80–81 using IntelliJ IDEA CE, 88 Artifact storage, 107 ASP.NET Core Web APIs build stage, 72–73 command line, 65, 82 configuration file, 68 deploy, 83 Dockerfile, 80 using IntelliJ IDEA CE, 83, 85–86

B Bash, 108–109 Bypass unit tests, 9

C CI/CD pipeline, 10–11 CircleCI account, creation, 96–97 authorizing, 98

project creation build screen, 98–99 configuration file, 102 environment variables, 100–101 .NET Core project, 99–100 rebuild, 101–102 signing up, 81, 97–98 Commercial off-the-shelf (COTS), 6

D DevOps, 14

E, F End to end tests, 11, 107 Enterprise CI/CD pipeline deployments, 107 stages, 106–107 Enterprise CI/CD platform Circle CI, 59 containers, 51–52 Docker builds, 51 install, 50

© Brandon Atkinson, Dallas Edwards 2018 B. Atkinson and D. Edwards, Generic Pipelines Using Docker, https://doi.org/10.1007/978-1-4842-3655-0

125

Index

Enterprise CI/CD platform (cont.) .NET Core 1.0, 52 .NET Core 2.0, 53 Jenkins (see Jenkins) .NET Core, 47–49 shared code, 53–54 shell scripts, 55 Travis CI, 60–61

G, H Generic pipeline pre-and postpipeline hooks, 116–119 script hooks, 115–116 stage hooks, 119–122 stages, 106 tips, 122 Golang, 108–109

I IntelliJ IDEA CE, 83–88

J, K Jenkins configure libraries, 56–57 directory structure, 57 Docker plugin, 57–58 vars folder, 57 jq tool, 73

126

L Language-specific pipelines benefits, 25 build step angular, 38–39 generic pipeline, 37 switch/case statement, 37–38 code reuse, 29–30 configuration file, 43, 45–46 DevOps team, 33 microservices, 34 shell scripts, 40–43 UIs, 26 using shared code, 36 Licensing scanning, 106 Lines of business (LOBs), 43

M, N, O Microservices architecture, 18 CI/CD pipeline steps, 20–21 monolithic application, 21 two-week release cycle, 31–32 with UI, 23–24 Monolithic application bypass unit tests, 9 CI/CD pipeline definiton, 10 processes, 6–7 static code analysis, 11 deployment pipeline, 7–8

Index

deployment process, 5 design, 2 feature development timeline risks, 12–14 MVC frameworks, 3–4 quality pipeline stages, 10–11

P, Q, R Performance tests, 11, 107 Pipeline archive stage, 75–77 build stage, 72–73 clone stage, 71 configuration file, 67–71 deploy stage, 77–78 running, 81 test stage, 74–75 Travis CI (see Travis CI) PowerShell archive stage, 111 build stage, 109–110 deploy stage, 112 unit test stage, 110–111 Python archive stage, 114 build stage, 112–113 deploy stage, 114 unit test stage, 113

S Script hooks, 115–116, 121 Security scanning, 106 Shared pipeline benefits, 26 goal, 28 standards, 27–28 Shell scripts, 108–109 Spring Boot applications configuration file, 67 Dockerfile, 78–80 Hello World message, 64 using IntelliJ IDEA CE, 87 Stage hooks, 119–122 Static code analysis, 106

T, U, V, W, X, Y, Z Travis CI account, creation, 89–90 configuration file, 94–95 environment variables, 91 .NET Core sample repository, 91 run pipeline, 93 sign-up process, 81 trigger building, 92–93

127