Workload Automation Using HWA: With Architecture and Deployment Options 148428884X, 9781484288849

Citation preview

Workload Automation Using HWA With Architecture and Deployment Options — Navin Sabharwal Subramani Kasiviswanathan

Workload Automation Using HWA With Architecture and Deployment Options

Navin Sabharwal Subramani Kasiviswanathan

Workload Automation Using HWA: With Architecture and Deployment Options Navin Sabharwal New Delhi, India

Subramani Kasiviswanathan Chennai, 600127, India

ISBN-13 (pbk): 978-1-4842-8884-9 https://doi.org/10.1007/978-1-4842-8885-6

ISBN-13 (electronic): 978-1-4842-8885-6

Copyright © 2023 by Navin Sabharwal and Subramani Kasiviswanathan This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Aditee Mirashi Development Editor: Laura Berendson Coordinating Editor: Aditee Mirashi Cover designed by eStudioCalamar Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, Suite 4600, New York, NY 10004-1562, USA. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@ springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail [email protected]; for reprint, paperback, or audio rights, please e-mail [email protected]. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book’s product page, located at www.apress.com/978-1-4842-8884-9. For more detailed information, please visit http://www.apress.com/source-code. Printed on acid-free paper

Thank you, goddess Saraswati, for guiding us to the path of knowledge and spirituality.

असतो मा साद गमय, तमसो मा ज्योतिर् गमय, मृत्योर मा अमृतम् गमय (Asato Ma Sad Gamaya, Tamaso Ma Jyotir Gamaya, Mrityor Ma Amritam Gamaya) Lead us from ignorance to truth, lead us from darkness to light, lead us from death to immortality.

Table of Contents About the Authors���������������������������������������������������������������������������������������������������� ix About the Technical Reviewers������������������������������������������������������������������������������� xi Acknowledgments������������������������������������������������������������������������������������������������� xiii Introduction�������������������������������������������������������������������������������������������������������������xv Chapter 1: Introduction to Workload Automation����������������������������������������������������� 1 Workload Automation Concepts���������������������������������������������������������������������������������������������������� 1 Introduction to HCL Workload Automation������������������������������������������������������������������������������� 6 HCL Workload Automation Strengths������������������������������������������������������������������������������������� 11 Summary������������������������������������������������������������������������������������������������������������������������������������ 12

Chapter 2: HCL Workload Automation Architecture������������������������������������������������ 13 HWA Components����������������������������������������������������������������������������������������������������������������������� 13 HCL Workload Automation Components and Communication����������������������������������������������� 16 Architecture Types����������������������������������������������������������������������������������������������������������������� 21 Summary������������������������������������������������������������������������������������������������������������������������������������ 24

Chapter 3: HCL Workload Automation Deployments����������������������������������������������� 25 Planning Deployments���������������������������������������������������������������������������������������������������������������� 25 Stand-Alone Architecture������������������������������������������������������������������������������������������������������ 27 High Availability Architecture������������������������������������������������������������������������������������������������� 32 Disaster Recovery Architecture��������������������������������������������������������������������������������������������� 35 Containerized Deployment���������������������������������������������������������������������������������������������������� 39 HWA Deployment on Cloud���������������������������������������������������������������������������������������������������� 47 Summary������������������������������������������������������������������������������������������������������������������������������������ 48

v

Table of Contents

Chapter 4: Workload Design and Monitoring Using DWC���������������������������������������� 49 Designing and Monitoring Workload Objects Using HWA Dynamic Workload Console��������������� 49 Business Requirement���������������������������������������������������������������������������������������������������������� 55 Summary������������������������������������������������������������������������������������������������������������������������������������ 96

Chapter 5: HWA for Managed File Transfers����������������������������������������������������������� 97 Business Requirement���������������������������������������������������������������������������������������������������������������� 97 Summary���������������������������������������������������������������������������������������������������������������������������������� 109

Chapter 6: HWA Integration with SAP Application������������������������������������������������ 111 Business Requirements������������������������������������������������������������������������������������������������������������ 112 Summary���������������������������������������������������������������������������������������������������������������������������������� 122

Chapter 7: Automate Job Executions on Microsoft SQL Server���������������������������� 123 Business Requirement�������������������������������������������������������������������������������������������������������������� 123 Summary���������������������������������������������������������������������������������������������������������������������������������� 129

Chapter 8: Working with RESTful Web Services��������������������������������������������������� 131 Business Requirement�������������������������������������������������������������������������������������������������������������� 132 Summary���������������������������������������������������������������������������������������������������������������������������������� 138

Chapter 9: Submit, Orchestrate, and Monitor Jobs on a Kubernetes Cluster������� 139 Business Requirement�������������������������������������������������������������������������������������������������������������� 139 Summary���������������������������������������������������������������������������������������������������������������������������������� 144

Chapter 10: HWA Integration with Microsoft Azure���������������������������������������������� 145 Business Requirementax���������������������������������������������������������������������������������������������������������� 145 Summary���������������������������������������������������������������������������������������������������������������������������������� 152

Chapter 11: HWA Integration with Ansible����������������������������������������������������������� 153 Business Requirement�������������������������������������������������������������������������������������������������������������� 153 Summary���������������������������������������������������������������������������������������������������������������������������������� 159

vi

Table of Contents

Chapter 12: HWA Event Rule Management����������������������������������������������������������� 161 HWA Integration with ServiceNow�������������������������������������������������������������������������������������������� 161 Use Case 2: Auto-Remediation of Process Down���������������������������������������������������������������������� 167 Summary���������������������������������������������������������������������������������������������������������������������������������� 172

Chapter 13: Tool Administration and Best Practices�������������������������������������������� 173 Daily Application Health Check������������������������������������������������������������������������������������������������� 173 Housekeeping Procedure to Maintain Application Health��������������������������������������������������������� 176 Database Maintenance Procedure�������������������������������������������������������������������������������������������� 181 Database Backup and Restore Policies������������������������������������������������������������������������������������ 183 Summary���������������������������������������������������������������������������������������������������������������������������������� 183

Chapter 14: Alerting and Troubleshooting Issues������������������������������������������������� 185 Configuring Alerts and Responses�������������������������������������������������������������������������������������������� 185 Restore Services Post Outage��������������������������������������������������������������������������������������������������� 191 Troubleshooting Techniques������������������������������������������������������������������������������������������������������ 198 Summary���������������������������������������������������������������������������������������������������������������������������������� 203

Chapter 15: HWA Reporting���������������������������������������������������������������������������������� 205 Summary���������������������������������������������������������������������������������������������������������������������������������� 210

Chapter 16: HWA Security������������������������������������������������������������������������������������� 211 Traditional Model (File Based)��������������������������������������������������������������������������������������������������� 211 Role-Based Access Control������������������������������������������������������������������������������������������������������� 217 Summary���������������������������������������������������������������������������������������������������������������������������������� 227

Index��������������������������������������������������������������������������������������������������������������������� 229

vii

About the Authors Navin Sabharwal is Chief Architect and Head of Strategy for Autonomics at HCL Software. He is responsible for innovation, engineering, presales, and delivery of award-­ winning autonomics platforms. Navin has eight patent grants and has published numerous books on AIOps, DevSecOps, public cloud, automation, and machine learning. He can be contacted via his LinkedIn profile: www.linkedin.com/in/navinsabharwal/. 

Subramani Kasiviswanathan is a Solution Architect, leading the Practice and Engineering Services for workload automation, AIOps, data analytics, and reporting in HCL DRYiCE, where he is responsible for supporting IP developments and service delivery in these areas. He has overall 19 years of IT experience and 6 years of experience in academics.  

ix

About the Technical Reviewers Ummiti Ramesh is a Subject Matter Expert for workload automation solution and has overall 12 years of IT experience and 11 years of experience in HCL workload automation. Currently working as Technical Lead and Subject Matter Expert for all workload automation products supported under HCL DRYiCE, he is responsible for architecting solutions for greenfield customers and crosstool migrations. 

Franco Mossotto is a Sr. Software Architect responsible for HCL/IBM workload automation product family development. He joined IBM in 1998 as a Tivoli Workload Scheduler for z/OS developer. Franco has worked in design, development, and support of IBM Tivoli scheduling and provisioning products. In the scheduling area, he worked as a developer, chief designer, L3 technical leader for both Tivoli Workload Scheduler and Tivoli Workload Scheduler for z/OS, and as an architect for the development of cloud offerings. Since September 2016, with the IBM and HCL IP partnership, Franco moved to HCL Technologies with the rest of the team to continue his career with the HCL/IBM Workload Automation portfolio.  

xi

Acknowledgments To my family for their love and support as always; without your blessings, nothing is possible. To my coauthors, contributors, and reviewers and the brilliant technical team Subramani Kasiviswanathan, Ummiti Ramesh, Franco Mossotto, and Ajmeer Mohideen – thank you for the hard work and quick turnarounds to deliver this book. It was an enriching experience, and I am looking forward to working with you again soon. Thank you to Aditee and the entire team at Apress for making this happen.

xiii

Introduction Workload automation has been the backbone of business processes. Most business processes involve job scheduling and workload automation to ensure that the systems are integrated and information flows across them in a seamless error-free fashion. In the space of workload automation, there have been so many advancements in terms of integrating your complex workload, workflow, and business processes across automation platforms, ERP systems, and business applications from mainframe to multi-­ cloud. Unlike other systems of automation, workload automation is focused on real-time processing, predefined event-driven triggers, and situational dependencies. Workload automation offers centralized control of the management of multiple tasks, making it possible to schedule enterprise-wide tasks. Further, it supports the timely completion of tasks. WLA increases efficiency, reduces the turnaround time for workflows, and reduces errors in end-to-end processes. The book discusses how HCL workload automation solution has been able to meet automation requirements of the digitally transformed platforms. It specifies detailed architecture and deployment options. The course demonstrates best practices to deliver robust workload automation solution by providing insights into best practices and integration with various systems. The book can be a reference material for beginners as well as advanced users of workload automation. It not only provides information on how to use the tool but also gives ideas on how to choose the right candidates or use cases for workload automation. The course contains numerous use cases and their implementation procedures that can be referred for any workload automation requirements.

xv

CHAPTER 1

Introduction to Workload Automation This chapter introduces workload automation concepts, mandatory features that a workload automation solution should possess, the components of HCL workload automation, network communication between various components, different processes that constitute HCL workload automation service, and the strengths of HCL workload automation.

Note  As a takeaway of reading this chapter, the reader is expected to understand the concepts of workload automation and the need for this solution in IT operations.

Workload Automation Concepts Workload automation is not new. Gartner introduced the concept of workload automation in their report “Hype Cycle for IT Operations Management” during the year 2005. IT operation teams were following the traditional approach of static and manual job scheduling. Before the evolution of workload automation tools, job scheduling and maintenance was a manual and error-prone task. With the arrival of job scheduling tools, this was automated and made easier through usage of standardized tools. Workload automation addresses the requirements of automating the executions of business processes and transactions. It is also known as a job scheduling system. Workload automation solution automates the workflows which can be executed at a particular time, or they can be executed dynamically based on events and triggers. The following Figure 1-1 depicts a typical stand-alone job stream and the various objects that constitute a job stream, which we will go over in the upcoming sections and chapters. 1 © Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6_1

Chapter 1

Introduction to Workload Automation

Figure 1-1.  HCL Workload Automation Stand-Alone Job Stream Any workload automation solution should be able to address the following requirements: •

Centralized job management Every application/technology has a built-in scheduler that is natively provided and supported by the vendor, for example, Informatica offers Datastage, MSSQL offers SQL scheduler, Windows offer Windows scheduler, Unix offers Crontabs, SAP offers SAP scheduler, etc. The native schedulers can automate the business processes and transactions pertaining to their application. These are aware of the dependencies within their application; however, the limitation is that these schedulers are not aware of the status of the processes running for other applications on which they are dependent. In a distributed environment where applications are loosely coupled and dependent on each other, it becomes highly important to visualize the status of all business processes centrally in a unified console to get a holistic view of what is going on with all the business processes running across the application landscape. A business process may cut across multiple transactions which are happening in different applications, and multiple jobs may need to be triggered in a defined sequence across multiple applications for a business process to execute. A job scheduling software helps in automating this by providing a master

2

Chapter 1

Introduction to Workload Automation

scheduler which can control the execution of jobs across different applications and platforms. Thus, from an IT operations perspective, the operators and administrators can get a central console to view the status of these connected or dependent jobs as well as manage the complex scheduling and execution of these jobs in an easy-touse console. Figure 1-2 depicts a typical job stream with external dependencies and the various objects that constitute it.

Figure 1-2.  HCL Workload Automation Interdependent Job Streams •

Time-based and event-driven scheduling The workload automation solution should be able to schedule the business processes to run on the target based on a calendar/date/ time. It should also be able to trigger business processes/workflows based on certain conditions or events that are occurring on the target machines.

•

Application and platform integrations In a traditional approach of handling business processes and their dependencies, a scheduler or the application developer had to write wrapper scripts using any scripting languages like PowerShell, Shell, Perl, Python, VBScript, etc., to provide all inputs for the job to trigger on the target application server like a valid application user and their credentials, application profiles, source path and job to trigger, database details, queries, procedures, and so on. In a transformed workload automation environment, we have the benefits of 3

Chapter 1

Introduction to Workload Automation

application and technology adaptors or plug-ins that can help to seamlessly integrate with the applications or platforms and provide an interface for all the inputs needed for batch processing on target applications and platforms. This eliminates the need for custom development and usage of error-prone scripts for automating job scheduling. A large repository of adaptors and plug-ins provide outof-the-box integration with widely used applications and platforms. •

Ability to handle/store outputs in variables The business processes run on the set targets generate the output or log files that are stored either on the server or client or both. The workload automation solution should be able to scan the output, trace the required messages, and store them into variables. These variables can then be used for further batch processing. This enables the workflow to proceed based on output provided by each step that acts as an input to the next step.

•

Conditional batch processing The related business processes are grouped and defined as one batch. They are executed according to the sequence based on the dependencies set on them. The dependencies can be within the same batch or a different batch. The dependencies within the same batch are called internal dependencies, and the ones from other batches are called external dependencies. The batches or any individual business processes can be set to execute based on certain logical conditions. The values that are stored in variables after an output scan can be compared, and batches can be set to trigger based on the results of the comparison.

•

Handling resources virtual and physical It is very important to ensure that acceptable workload is submitted to execute on the target hosts at any given point in time so that it does not impact the performance of the application and the databases. Any workload automation solution should have the ability to manage the workload by configuring physical or virtual resources to control the load that is exerted on the target.

4

Chapter 1

•

Introduction to Workload Automation

Ability to manage alerts, events, and logs Workload automation solution should be able to configure and generate the alerts for events such as a status change, process running longer than expected, possible impact in meeting the SLA, service/process failure, etc., and it should also be able to log the details of the events or errors that will help schedulers and application users to troubleshoot the issues and figure out the root cause. They should also be able to capture the application logs and store them on the server or client or both.

•

Ability to provide self-service and RBAC It is an expectation from the workload automation solution to provide the users with self-service capabilities where users can view, edit, execute, and manage the business processes pertaining to their application only. The access granted should be granular and based on the roles.

•

Manage releases for process/schedule definitions Just like any other development environment, workload automation jobs are also created in a development environment and moved to testing. The tools should provide capabilities to promote the preproduction environment to production environment.

•

Critical path and batch impact analysis The solution should possess the capability to define critical processes in a batch or workflow and should be able to proactively identify potential delays, likely errors and how they could impact systems and business users. They should also be able to alert such cases to schedulers and application users in a timely fashion so that actions can be taken proactively so as to avoid or minimize disruptions to the critical business processes.

•

Application-level high availability and disaster recovery Workload automation architecture should provide capabilities for application-level high availability, i.e., when primary node of an application is not in service, the architecture should support 5

Chapter 1

Introduction to Workload Automation

automatic failover to a secondary node of the application and restore the service with minimal or no downtime. Similarly, it should also provide the built-in mechanism to restore the services on primary once it is up and running. The application architecture should also address the requirements of disaster recovery at times when the data center itself is down and services need to be restored from a DR location. •

Security The solution must be able to manage security at all levels, i.e., user-­ level security, application security, and data security.

Introduction to HCL Workload Automation HCL workload automation is an enterprise solution to meet hybrid workload management needs of modern enterprises. It provides a platform for centralized job management across different platforms and application environments via its strong integration capabilities. It caters to all KPIs that a workload automation solution should meet as described in the previous section. The solution has three main components in the portfolio: 1. HCL workload automation for Z 2. HCL workload automation for distributed 3. Dynamic Workload Console (DWC) HCL workload automation comprises of the following components: •

HCL workload automation engine This is a scheduling engine which is the heart of the system that governs and processes the tasks pertaining to the scheduler. It is assigned with specific roles during the installation of the software in the HCL workload automation network. We will see more in detail about the roles in upcoming chapters. Figure 1-3 depicts the core components of HCL workload automation.

6

Chapter 1

Introduction to Workload Automation

Figure 1-3.  HCL Workload Automation Components •

Dynamic workload console DWC is a unified and centralized web-based graphical user interface for HCL workload automation. This component provides a userfriendly interface to access all the workload automation functions. The following Figure 1-4 depicts Dynamic Workload Console (DWC).

Figure 1-4.  Dynamic Workload Console

7

Chapter 1

•

Introduction to Workload Automation

HWA agents Agents are HWA software that are installed on the end points where the workload automation needs to be established. These end points can be any application servers, database servers, web servers, etc. This component helps execute jobs on the intended end points and send back the output and status to the master. A typical/basic HCL workload automation network is shown in the following illustration. We will discuss each of these elements in detail.

Figure 1-5.  HCL Workload Automation Network •

Master domain manager This is the central processing unit of a workload automation system and is a management hub. Master domain manager communicates directly with the database to collect information related to HWA objects and generate the production plan that needs to be supplied to all its domain managers and fault tolerant agents. All the other components like the domain managers, agents, etc. receive instructions from the master domain manager as to what is their schedule/tasks for a defined business day.

8

Chapter 1

•

Introduction to Workload Automation

Domain manager It plays an integral part in a widely distributed and large network where the workload is required to be divided and managed locally in smaller groups to ensure balanced workload and smooth execution of batches.

•

Dynamic workload broker This component governs and provides all services related to the dynamic scheduling which consists of dynamic agent, pool, dynamic pool, etc.

•

Dynamic agent This component is installed on the target machines or application servers on which the execution tasks need to happen. Agents are automatically discovered by their domain or master domain managers once they are installed in the network and get registered with them. They then start receiving instructions from their domain or master domain managers as to what to process and when to process.

•

Fault tolerant agent (FTA) FTA is installed on the target machines or application servers on which the tasks need to be executed. As the FTAs receive a copy of their schedule well in advance from the domain manager or master domain manager, they are aware of their dependencies and hence can provide scheduling service even when their domain or master domain manager is not active in the network.

•

Pool Pool is a virtual definition that comprises of one or more dynamic agents. This component performs load balancing and launches the jobs on the targets based on the resource availability.

•

Dynamic pool DP is a virtual definition that comprises of one or more dynamic agents. This component performs load balancing and launches the jobs on the targets based on the resource availability. The user 9

Chapter 1

Introduction to Workload Automation

has the advantage to control the target where the jobs need to be executed. Moreover, the processes can be executed based on the availability of physical resources such as CPU, disk, etc. •

Standard agent (SA) This agent type executes instructions as directed by its manager.

•

Extended agent (XA) This is governed by FTA but also extends additional attributes to execute workload for specific applications like PeopleSoft, Oracle applications, SAP R3, etc.

After having gone through the HWA components and its functions, let us understand some basic concepts and terminologies used in HWA. Production Plan (JnextPlan) JNextPlan is a script that runs every day in HWA during the start of a defined business day and follows a sequence of steps to move the old plan and generate the new schedule/tasks for the current business day, as illustrated in Figure 1-6. The production plan generated for the current day is called the “Symphony file.” The following are the steps involved during the generation of a production plan in HCL workload automation. StartAppServer This job checks the availability of the application server and restarts it in case it’s not running. MakePlan Master domain manager connects to database via the web server (Liberty) and creates the production plan based on the scheduling conditions and dependencies defined in its database. During the plan creation, it locks the database using “Planner process” to avoid any new processing during this time that can obstruct the plan generation process. This process creates a pre-production file known as “SymNew.”

10

Chapter 1

•

Introduction to Workload Automation

SwitchPlan “Planner process” then updates the pre-production plan and “Stageman” merges pre-production file SymNew with the pending process executions and incomplete schedules that need to be carried forward from previous day and consolidates it in a file called “Sinfonia”; a copy of it will then be updated to the new production plan file called Symphony for the current day. The Symphony file for the previous day will get archived in Schedlog folder. The planner then unlocks the database and makes it available for use. The newly created symphony file gets distributed to its agents.

Figure 1-6.  JnextPlan Flowchart

HCL Workload Automation Strengths WA provides extensive capabilities and features for job scheduling and orchestration. Let us go through some of the key capabilities that HWA provides. •

HWA is evolving to become a more general orchestrator tool with extensive scheduling capabilities.

•

HWA provides UI and REST API import/export features to allow the integration of HWA in CI/CD pipeline. You can use this feature in your pipeline for running any scheduled jobs as a part of the testing process.

•

HWA has a development toolkit to build new job types and many out-­ of-­the-box integrations. The new Automation Hub website provides 11

Chapter 1

Introduction to Workload Automation

information and plug-ins to orchestrate multiple automation domains. •

HWA Web Console is highly intuitive and provides effective graphical views and dashboards with alerts on KPIs. The Self-service Catalog provides mobile access for business users.

•

All HWA components (Server, Console, and Agents) can be deployed on containers. The Kubernetes job type allows to manage containerized applications.

•

HWA provides full REST API support (anything that can be done with the Console), and a CLI is available to interact with the product.

•

HWA provides versioning for all the database objects with compare and restore capabilities. All actions are logged and available for auditing.

•

HWA allows SLA management and predictive scheduling through the Workload Service Assurance and the What-If features.

Summary To summarize the content of this chapter, the reader should have understood the following:

12

•

Workload automation concepts

•

Various HCL workload automation terminologies and their definitions

•

HWA JNextDay plan

•

HWA strengths

CHAPTER 2

HCL Workload Automation Architecture In this chapter, we will cover details on deployment architecture of HWA. The objective is to impart, among the readers, a clear understanding of the architecture components and communication between them.

Note  The readers are expected to have read and understood Chapter 1 so that they can relate with and cover Chapter 2.

HWA Components The following are the HCL workload automation components: HCL Workload Automation Master Engine/Scheduler The HWA master is called the scheduler. This engine is the application server component that processes all the instructions that are scheduled to be executed. It plans the schedules day wise and distributes the workload to its agents to process as per definitions, hence governing all the processes that are scheduled to run for a particular business day and updating their status to its database. HWA extends its service in the following areas: •

The schedule plan is generated by the master for the business day; previous plans are captured in the server log location for further reference/troubleshooting.

© Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6_2

13

Chapter 2

HCL Workload Automation Architecture

HWADATA/schedlog •

The master captures audit information for every access/change to scheduling definition or the plan and store them in the database. The same information is also logged in log files on the master and FTA where the command is run. HWADATA/audit

•

The master and the agents capture the logs of all the business processes that are scheduled to run on the remote systems. This shall be used for troubleshooting and audit purposes.

•

The master captures the logs of all the communications that happen between HWA components internally and stores it in the log location for further reference/troubleshooting. HWADATA/stdlist/traces/_TWSMERGE.log

•

•

The master provides options to set up granular role-based access controls (RBAC) to its local or domain users.

•

The architecture supports various databases to which the master can communicate with.

•

The HCL workload automation integrates with various application/technology/platform/server environments to directly interface with the application to execute/monitor/schedule/ manage via plug-ins.

•

The master has Application HA capability to fail over and fail back the services automatically between the master and the backup master to ensure service maximum availability.

Dynamic Workload Console •

14

DWC is the front-end HWA console which brings in the feature of graphical user interface for the users to monitor, design, and manage their workload. DWC also provides the functionality to define and administer HWA and its components.

Chapter 2

•

•

HCL Workload Automation Architecture

•

DWC provides functionality to restrict object-level access to users of various roles like administrator, operator, developer, etc. so that they can see specific views according to their roles.

•

DWC provides functionality to design specific dashboards and insights about the various business processes for an application, system availability, critical jobs, min/max duration, reports based on job status, etc.

Fault Tolerant Agent •

FTA provides functionality to run command jobs or executables on the remote machines where FTA agent is installed.

•

FTA provides functionality to perform agentless job executions on a remote host via the SSH protocols.

•

FTA provides functionality to run business processes on a remote host even when the master that it reports to is not in service as it is well aware of its workload via the plan circulated by its master.

Dynamic Agent •

DA provides the functionality to run jobs for advanced job types like RESTAPI jobs, Database, Cloud Native, etc., via job templates. Unlike FTAs, plan is not delivered to DAs by master; hence, they receive the instructions in real time from their master about their workload that they need to run. They can also provide the same feature like FTA.

•

The remote servers where the business processes are launched can be grouped in a single dynamic pool on which the processes can be submitted when specific conditions satisfy like available CPU, memory, etc.

•

DA provides the functionality of the gateway server which receives connections from various dynamic agents and validates and approves them before sending the communication to the target system component.

15

Chapter 2

•

•

•

HCL Workload Automation Architecture

Dynamic Workload Broker •

DWB enables the dynamic agent to plan and run the workload on specific remote servers or static/dynamic pool after analyzing the job definitions, resources, etc.

•

DWB job dispatcher dispatches and submits the jobs that it receives from the master to the respective dynamic agents to process. Before submitting the resource, advisory analyzes and decides the static/ dynamic pool on which the job has to be submitted.

Liberty •

Liberty is an application server runtime environment that helps in communicating between various components of HWA including the database.

•

On distributed platforms, Liberty provides both a development and an operational environment.

Database •

Database is a data store for HWA objects. HWA architecture supports various databases to which the master can communicate with. Some of the supported databases are HCL OneDB, Informix, Azure SQL database, DB2, MSSQL, and Oracle.

•

DB2 database is bundled with HWA package and does not need any separate licenses. HA and DR functionality is also inclusive.

 CL Workload Automation Components H and Communication HCL workload automation has several internal processes running that help in segregating and manage networking, resolving dependencies, and running the jobs on the target systems. It performs these activities with the help of message queues. There are many such processes that get initiated during the HWA application startup. The diagrams shown in Figures 2-1 and 2-2 depict the processes that constitute the internal communication between the various components of HCL workload automation.

16

Chapter 2

HCL Workload Automation Architecture

Figure 2-1.  HCL Workload Automation Interprocess Communication

Figure 2-2.  HCL Workload Automation Messaging Communication 17

Chapter 2

HCL Workload Automation Architecture

•

Netman process reads NetReq.msg file and receives all different network communications between the FTA agents and domain manager to take respective actions.

•

Mailman process reads the Mailbox.msg file. It processes the messages and instructions from the message file. These messages are coming from all the components that constitute HCL workload automation.

•

Batchman process reads the Intercom.msg file. It processes the messages and instructions from the message file. These messages are coming from local mailman process.

•

Jobman process reads the Courier.msg file. These messages are written by batchman process. Jobman executes the jobs under the direction of batchman process. Jobman process is owned by root user to enable it to execute the job by any user.

•

Batchman process writes the messages to Planbox.msg and Server. msg files. These instructions are read by the HWA scheduling engine for further processing.

This part explains how the various components of HWA communicate with each other to serve the purpose of workload automation. Figure 2-3 shows the detailed communication path of HWA component.

18

Chapter 2

HCL Workload Automation Architecture

Figure 2-3.  HWA Components Communication/Network Path •

RDBMS Tier denotes the database component; from Figure 2-3, it connects to DB2 database, and default DB2 database port is 50000. In case of other databases, the port numbers will change.

•

Application Server Tier talks about the HWA master and Backup master domain managers.

•

Web server tier is the DWC Dynamic Workload Console.

•

Agent Tier is represented by the fault tolerant agent, dynamic agent, extended agent, and standard agent.

Table 2-1 provides a list of default ports that are used for communication among the HWA components.

19

Chapter 2

HCL Workload Automation Architecture

Table 2-1.  List of HWA communication ports S.no Source

Destination

Port

1

Master Domain ManagerBackup Master Domain ManagerWeb server

Backend Database

500005000050000 DB connectivity Bidirectional

2

Master Domain Manager

Backup Master Domain 3111131131 Manager

3

Master Domain Manager

Backup Master Domain 311143111631117 Callback Manager Bidirectional

4

Master Domain ManagerBackup Master Domain Manager

Agent Tier

5

Agent Tier

Master Domain 3113131116 ManagerBackup Master Domain Manager

Unidirectional

6

Agent Tier

Master Domain 31111443 ManagerBackup Master Domain Manager

Bidirectional(HTTPS)

7

Operator Desktop

Master Domain 9443443 ManagerBackup Master Domain Manager

Unidirectional

8

Operator Desktop

Web Server

9443443

Unidirectional

9

Master Domain ManagerBackup Master Domain Manager

Web Server

3111631117

Bidirectional

20

31114

Rule notes

Callback Bidirectional

Unidirectional

Chapter 2

HCL Workload Automation Architecture

Architecture Types This part explains the various HWA deployment architecture options. The following are the types of architectures that can be used based on the environment size and complexity. •

Stand-alone Stand-alone architecture is proposed for small customers with minimum number of jobs. Typically, less than 500 jobs would fall under this category. Along with number of jobs, another factor is the complexity of the environment and jobs. This architecture should be used for deployments where the complexity is minimal. Stand-alone architecture comprises of a single Master Domain Manager, DWC, Liberty, and Database that manage the workload. This architecture choice introduces a downtime when the master services are stopped due to some reason, and this could cause a potential business impact. Figure 2-4 represents a typical standalone deployment architecture.

Figure 2-4.  Stand-Alone HWA Architecture

21

Chapter 2

•

HCL Workload Automation Architecture

High availability High availability architecture is proposed for medium and large customers with more than 500 jobs and higher complexity of jobs. High availability architecture comprises of Primary and Backup Domain Managers with all components highly available. This avoids downtime when the master services are stopped on any of the active master domain managers. It introduces high availability for both HWA and database components. In this architecture, we will have two master domain managers and consoles. We would need to make sure network communication is established between these two servers as well as agents connected to the master servers. The services on primary master will be active while the services on the backup master will be passive. Whenever the primary master server goes down, HWA performs the health check and identifies that the primary master is down and initiates the auto failover option so that the backup master will take the role of primary master server and inform the agents connected to the primary master server to connect with new primary master server (backup master). The advantage is that during the failover, there will not be any issues with job processing and no downtime is required. Once the primary master domain manager is active, HWA has the functionality to perform failback from backup master to primary master. However, this process needs to be initiated manually after required health checks are performed on primary master. This process also does not impact job processing or introduce any downtime.

Note  The HA architecture should be deployed in a scenario where we need the services to be available all the time within the data center. Figure 2-5 represents a typical high availability deployment architecture.

22

Chapter 2

HCL Workload Automation Architecture

Figure 2-5.  High Availability HWA Architecture •

Disaster recovery Disaster recovery architecture is similar to the high availability architecture but is proposed for customers who would like to have their DR set up in a different location to cater to disaster scenarios. This setup ensures high availability of the HWA services within the data center as well as ensures services to failover and be available at a DR data center when the primary data center is down or unavailable. The DR master server takes the role of a backup master but is put to use only when the backup master in the primary data center is also not available. This means that the data is in sync between the primary, backup, and the DR servers.

23

Chapter 2

HCL Workload Automation Architecture

It takes approximately 5–10 minutes for the DR process to be triggered and bring up all the services on the DR site. We will be covering this scenario in the upcoming chapters. Figure 2-6 represents a typical disaster recovery deployment architecture.

Figure 2-6.  Disaster Recovery HWA Architecture

Note  The DR architecture should be deployed in a scenario where we need the services to be available at a DR data center when the primary data center is down or unavailable.

S  ummary To summarize the content of this chapter, the reader should have understood the following:

24

•

Workload automation components and their functions in detail

•

HCL workload automation network communication

•

HWA architecture types and their features

CHAPTER 3

HCL Workload Automation Deployments HCL workload automation provides multiple deployment options to their customers according to the business requirements and the criticality. The following are the various deployment options and recommendations. In this chapter, we will cover these architecture options in detail. This will allow customers to decide on a suitable architecture for their enterprise requirements. •

Stand-alone architecture

•

High availability architecture

•

Disaster recovery architecture

•

HWA deployment on containers

Note  This chapter is an extension from the previous chapter to provide detailed information on the deployment options.

Planning Deployments While planning for any deployment, prerequisites are important and so is the case with HWA. Prerequisites must be completed before the deployment. •

HWA requires minimum of 2 GB of virtual swap memory space. Figure 3-1 provides the memory requirements for each component in MB.

© Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6_3

25

Chapter 3

HCL Workload Automation Deployments

Figure 3-1.  Memory Requirements

26

•

Temporary directory must have a maximum of 1GB free space.

•

The disk space for HCL workload automation is specified based on the data, log files, etc.

•

HCL workload automation should have a dedicated user to install the HWA application. HWA Windows user passwords can include any alphanumeric characters and ()!?=^*/~[]$_+;:.,@`-#. And for Unix user, passwords can include any alphanumeric characters and ()!?=*~_+.-.

•

Database must be installed prior to the HWA implementation. Even if not mandatory, for the rest of the chapter, we have assumed to have two database instances (db2inst1 and db2inst2) created on db server. db2inst1 will be used for HWA database, and db2inst2 will be used for DWC database. Make sure the database user should have full privileges to read/write the information into the database. In this chapter we are considering DB2 as the backend database for HWA.

•

Communication between HWA engine and database has to be enabled. Please work with your network team to enable the required ports. Refer to Chapter 2 for the network communication in Figure 2-2. Test your port connectivity between the server where you are planning to install the HWA engine and the database servers.

•

Once the preceding components are configured, we can proceed with HCL workload automation deployment.

Chapter 3

HCL Workload Automation Deployments

Stand-Alone Architecture This type of architecture is proposed when the customer environment is small and less critical workload. In this architecture, there is no high availability; thus, if the system goes down you will have to use other techniques at the virtualization level to bring up the machine or restore from a backup to bring up the machine. Database must be installed prior to HWA installation. In this deployment, we will install Liberty, MDM (Master Domain Manager), and DWC (Dynamic Workload Console). Figure 3-2 depicts a typical stand-alone HWA setup.

Figure 3-2.  Stand-Alone Architecture The following is the deployment procedure: •

Download the Liberty, HWA, and DWC and db2 package from Flexnet portal. Following is the link to the portal. https://id.hcltechsw.com/login/login.htm?

•

Install the Liberty using the following command: java -jar /image/wlp-base-all-20.0.0.11.jar --acceptLicense

27

Chapter 3

•

HCL Workload Automation Deployments

Create the database using create_database.sql query. Please note the database to be created using database instance user, so switch to db2inst1 user and run create_database.sql using the following command.    db2 -tvf create_database.sql

•

Once the database is created, switch to root user and configure the database using configureDb.sh Script like the following: ./configureDb.sh --wlpdir --rdbmstype DB2 -dbhostname --dbport 50000 --dbname TWS --dbuser --dbadminuser -dbadminuserpw

•

Once database is configured, install the HWA application using serverinst.sh script. ./serverinst.sh --acceptlicense yes --rdbmstype DB2 -dbhostname --dbport 50000 --dbname TWS -dbuser --dbpassword --wauser wauser -wapassword --wlpdir --inst_dir -HCL --thiscpu --brwksname -xaname --displayname --componenttype MDM -hostname --licenseserverid

•

Once HWA is installed, move to install DWC. Create the DWC database using create_database.sql query. Please note database to be created using DWC instance user db2inst2, so switch to db2inst2 user and run create_database.sql as the following to create the database. db2 -tvf create_database.sql

•

28

Once the database is created, switch to root user and configure the database using configureDb.sh script like the following:

Chapter 3

HCL Workload Automation Deployments

./configureDb.sh --wlpdir --rdbmstype DB2 -dbhostname --dbport 50001 --dbname DWC --dbuser --dbadminuser -dbadminuserpw •

Install DWC using dwcinst.sh script. ./dwcinst.sh --acceptlicense yes --rdbmstype DB2 --user wauser --password --dbname DWC --dbuser -dbpassword --dbhostname --dbport 50001 --wlpdir --inst_dir

•

Login to DWC portal and validate the environment. Login to DWC using wauser and connect to DWC portal using this link https://:9443/console/login.jsp.

•

To connect DWC to HWA server, navigate to Administator➤Manage Engine. Refer to Figure 3-3.

Figure 3-3.  Manage Enginetion Page •

This opens the interface to create the new engine. Refer to Figure 3-4.

29

Chapter 3

HCL Workload Automation Deployments

Figure 3-4.  New Engine Creation Page •

This opens the interface where we update HWA credentials and database credentials. Refer to Figure 3-5. Select “Test Connection.”

Figure 3-5.  New Engine Properties 30

Chapter 3

•

HCL Workload Automation Deployments

This opens the interface where it will show the status of the connection to the database. Refer to Figure 3-6.

Figure 3-6.  Test Connection Results •

We can see that connection to the database is successful, and to do further validation, we can now try to open the manage workload definition page. Refer to Figure 3-7.

Figure 3-7.  Landing Page of DWC •

This opens interface where we can confirm that the database connection to the HWA engine is working fine from console. Please refer Figure 3-8.

31

Chapter 3

HCL Workload Automation Deployments

Figure 3-8.  Workload Designer Page

High Availability Architecture •

High availability architecture is proposed when the customer is having >5000 and Manage workload definition. Refer to Figure 13-5.

176

Chapter 13

Tool Administration and Best Practices

Figure 13-5.  DWC Home Page •

This opens the following interface where we can create the job definition; select job definition and select Unix job as follows .Refer to Figure 13-6.

Figure 13-6.  Unix Job Definition

177

Chapter 13

•

Tool Administration and Best Practices

This opens the following interface to create the Unix job. Update job name, server name, and hwa username. The login user has to hwa user since we are removing the hwa logs alone. Refer to Figure 13-7.

Figure 13-7.  HWA Job Properties •

178

Under Task tab, update the rmstdlist path to remove logs older than 30 days. In case if we have to remove 90 days’ logs, we can change the 30 days to 90. Refer to Figure 13-8.

Chapter 13

Tool Administration and Best Practices

Figure 13-8.  HWA Job Task Properties •

Select “Save.” We have successfully created the job and it will remove the following logs: /stdlist/appserver/engineServer/logs /stdlist/logs /stdlist/traces To schedule the job, refer to Chapter 4. To remove the logs under /schedlog path, we have to create a Unix job with command “find /opt/ TWS/twsprd/TWS/schedlog/* -mtime +30 -exec rm {} \;”. This command will remove logs older than 30 days. The ideal way to do this is to run the job on both master and backup master; as a best practice, archive the schedlog files on an archiving system for future reference and proceed with removing the logs. 179

Chapter 13

Tool Administration and Best Practices

Please see the following procedure to create the job. To create the job, navigate to design ➤ Mange Workload definition ➤ Job definition ➤ Unix job. Refer to Figure 13-9.

Figure 13-9.  HWA Unix Job Properties •

180

Under task tab, in the command section, update “find /opt/TWS/ twsprd/TWS/schedlog/* -mtime +30 -exec rm {} \;” to remove logs older than 30 days. Refer to Figure 13-10.

Chapter 13

Tool Administration and Best Practices

Figure 13-10.  Unix Job Task Properties •

Select “Save.” We have successfully created the job. To schedule the job, refer to Chapter 4. In case if the customer wants to remove the logs older than 90 days, we have to change the command. From "find /opt/TWS/twsprd/TWS/schedlog/* -mtime +30 -exec rm  {} \;" To "find /opt/TWS/twsprd/TWS/schedlog/* -mtime +90 -exec rm  {} \;"

Database Maintenance Procedure Please follow the following high-level steps for database maintenance. 1) Database backup has to be taken as agreed with client. For offline/ online backup procedure, refer to the following link: 181

Chapter 13

Tool Administration and Best Practices

www.ibm.com/docs/en/license-­metric-­tool?topic=databasebacking-­­up-­db2 2) Run stats on the tables of database to update the stats time of tables. Refer to the following link: www.ibm.com/docs/en/db2/11.5?topic=commands-­runstats 3) Reorg on tables of database to reorganize the tables. Downtime as per agreed with client. Refer to the following link. www.ibm.com/docs/en/db2/11.5?topic=commands-­reorg-­table 4) Run lower high-water mark on table spaces to reclaim space at mount points. Refer to the following link: www.ibm.com/support/pages/db2-­reducing-­size-­table-­space 5) Auto-maintenance should be set to ON in db cfg in case no downtime for reorg on tables, not in case of big tables. Refer to the following link. www.ibm.com/docs/en/db2/11.5?topic=parameters-­auto-­ maint-automatic-­maintenance-­switches 6) Recreate indexes of tables wherever it is required. Refer to the following link. www.ibm.com/docs/en/db2-­for-­zos/11?topic=indexes-­types 7) If online backup is configured, include or exclude the archive logs in database backup. 8) Archive logs mount point backup to be taken if not included in database backup. 9) Pruning of archive logs which have been backed up. Points 7, 8, and 9 apply only in case where archival logging is set to a path, otherwise, offline backup with downtime in case of circular logging. 10) If db2audit log is enabled, mount point housekeeping is required. www.ibm.com/docs/en/db2/9.7?topic=commands-­db2audit-­ audit-facility-­administrator-­tool 182

Chapter 13

Tool Administration and Best Practices

11) Dialog path needs to be taken care under housekeeping. db2 get dbm cfg |grep -i diag One will get the diagpath and it should be kept below threshold.

Database Backup and Restore Policies For backup and restore of db2 databases, kindly refer to the following links: www.ibm.com/docs/en/license-­metric-­tool?topic=database-­backing-­up-­db2 www.ibm.com/docs/en/license-­metric-­tool?topic=database-­restoring-­db2

Summary In this chapter, we covered the following capabilities of HWA: •

HWA application health check and housekeeping

•

Database backup and restore policies

•

Database maintenance

183

CHAPTER 14

Alerting and Troubleshooting Issues In this chapter, we are going to discuss HCL workload automation alerts and troubleshooting issues. HWA has a wide variety of alerting mechanism to notify end users to work on their issues on priority. The following three topics describe the alerting and troubleshooting issues in details: 1. Configuring alerts and responses 2. Restoring services post outage 3. Troubleshooting techniques

Configuring Alerts and Responses HWA alerts are configured using HWA events. Event rules are used to detect event condition and take response actions that need to be performed. An event rule is created in the database. We have explained few use cases in Chapter 12 as well. Let’s create a use case for job failure email notification to users. Job failure email notification to users: Let’s create the event rule for job failure email notification; to create the event, navigate to design ➤ create event rule. Refer to Figure 14-1.

© Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6_14

185

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-1.  Create Event Rule •

This opens the following interface where we have to select event rule. Refer to Figure 14-2.

Figure 14-2.  Event Rule Home

186

Chapter 14

•

Alerting and Troubleshooting Issues

This opens the following interface where we have to update the event rule name and disable the DRAFT option; if the event rule is created with draft, it will not be activated. Refer to Figure 14-3.

Figure 14-3.  Event Rule General Tab •

Select Add Events and start creating the event. Refer to Figure 14-4.

Figure 14-4.  Event Rule Condition Page

187

Chapter 14

Alerting and Troubleshooting Issues

Events are divided into the following major categories: HCL workload automation object-related events – all the modifications in jobs, job streams, and workstations can be created using these events. Refer to Figure 14-5.

Figure 14-5.  Event Rule Options File monitoring event – events relating to changes to files and logs. Events for files monitoring that are predefined are Log message written, File created, File deleted, and Modification completed. Application monitoring event – events relating to HWA processes, file system, and message box. Events for Application monitoring that are predefined are Message queues filling, HCL Workload Automation file system filling, and HCL Workload Automation process not running. SAP-related event – these events are available only if you have installed HWA for applications. Using these events, we can detect if a specified string is matched in the log file or when the specified IDoc is created. •

In our case, select job status changed and select “+” symbol; refer to Figure 14-6.

Figure 14-6.  Job Status Change Condition 188

Chapter 14

•

Alerting and Troubleshooting Issues

This opens the following interface where we need to update the following details. Refer to Figure 14-7. Job Job Job Job

stream workstation – job stream workstation name stream name – job stream name name – name of the job to be monitored workstation – workstation name of the job

Figure 14-7.  Event Rule Properties •

We are creating the event for all jobs, so we are updating the value as “*.” Please update the status of the job to ERROR status and internal status to ABEND/FAIL to monitor job failure details. Refer to Figure 14-8.

189

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-8.  Event Rule Properties •

Once event is added, select the Add actions “+” and select the actions based on our requirement. Since we are creating job failure email notification, we have to select mail sender plug-in and send mail to send emails for job failures. Refer to Figure 14-9.

Figure 14-9.  Event Rule Action Properties •

190

We have to update recipient address, subject, and body of the mail as in Figure 14-9. And select “SAVE.” In HWA, there are multiple options

Chapter 14

Alerting and Troubleshooting Issues

like including HWA variables that contain information about the HWA objects in subject and body of the mail as shown in Figure 14-9. •

We have successfully created email notification for job failure event, and please see the following figure for sample email notification for job failure. Refer to Figure 14-10.

Figure 14-10.  Sample Email Notification Follow the preceding steps to create similar event rule for other workload scheduler objects.

Restore Services Post Outage Outage is nothing but services that are not available. There are different types of outages. 1. Planned outage 2. Unplanned outage 3. Emergency outage During planned outages, we can stop the applications and inform the application status to respective team to proceed further with the outage. To stop the HWA application during planned outages, log in to HWA master server and execute the following commands from the following paths to stop the Liberty process. Refer to Figure 14-11.

191

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-11.  Liberty Process Stop Command •

Check the process is stopped by executing “ps -ef | grep java.” Refer to Figure 14-12.

Figure 14-12.  Liberty Process Status •

192

We can see that Liberty process stopped; we have to stop the other HWA process. To stop other HWA process, execute the following commands. Refer to Figure 14-13.

Chapter 14

Alerting and Troubleshooting Issues

conman stop conman shut;wait conman stopmon ShutDownLwa

Figure 14-13.  HWA Process Stop Commands •

To check the process are stopped, please execute “ps -ef | grep TWS.” Refer to Figure 14-14.

Figure 14-14.  HWA Process Status Command 193

Chapter 14

•

Alerting and Troubleshooting Issues

We can see that all the HWA processes are stopped successfully. In case of any issues while starting the application, check the log files for reference.

Whereas in unplanned or emergency outages we will not be having enough notification to stop the application, all of a sudden our application will stop without intimation. During this time, we should check with our respective platform team (OS team) about the details. Once the outage is completed, we should start the application from scratch. To start the HWA application from scratch, follow the following steps. •

Log in to the Master server and execute “StartUp” command. Refer to Figure 14-15.

Figure 14-15.  HWA Process Start Command •

194

Once StartUp Command is issued, check the processes that are started by using “ps -ef | grep TWS.” Refer to Figure 14-16.

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-16.  HWA Process Status •

Once the process is started using StartUp, we should check database connectivity using “optman ls” command; in case of any issues, check log files; the output of the command should be like in Figure 14-17.

Figure 14-17.  Db Connectivity Check •

We can see that database is accessible from HWA; we can start the remaining process. Please follow the following steps to start other process like batchman, monman, mailman, etc. Execute the following commands; refer to Figure 14-18.

195

Chapter 14

Alerting and Troubleshooting Issues

conman start conman startmon StartUpLwa

Figure 14-18.  HWA Start Commands •

Once the commands are executed, check the process status by executing “ps -ef | grep TWS.” In case of any issues, check log files. Refer to Figure 14-19.

Figure 14-19.  HWA Process Status •

196

We can see that all the HWA processes are started; now change directory to and execute startAppServer.sh script to start the DWC Liberty process. Refer to Figure 14-20.

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-20.  DWC Process Start Command •

Once the process is started, check the status of process by executing “ps -ef | grep DWC.” In case of any issues, check log files. Refer to Figure 14-21.

Figure 14-21.  DWC Liberty Process Status •

We can see that DWC Liberty process is started. Log in to the DWC console to validate DWC is working fine. Refer to Figure 14-22.

197

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-22.  DWC Home Page

Troubleshooting Techniques In this part, we are going to discuss about HWA application troubleshooting steps and its fixes. Please see the following use cases: •

HWA agent process down.

•

Workstation unlinked status.

•

Dynamic workload console is not reachable.

HWA agent process down: Let us consider SVALIAPWLAP002 agent process is down. To troubleshoot the issue, first log in to Dynamic Workload Console and navigate to Monitoring and Reporting ➤ Monitor workload ➤ select workstation and run. Refer to Figure 14-23.

198

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-23.  Monitor Workload Page •

This opens the following interface where we can check the status of the agents. Refer to Figure 14-24.

Figure 14-24.  Agent Status Page •

We can see that agent process is down for SVALIAPWLAP002 server. To start the agent, select the agent SVALIAPWLAP002 check box and select start option to start the process. Refer to Figure 14-25.

199

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-25.  Agent Process Start Action •

Once the start action is issued, the agent is started without any issues. Refer to Figure 14-26.

Figure 14-26.  Agent Status •

200

In case of any issues, log in to the agent server and check the logs under .

Chapter 14

Alerting and Troubleshooting Issues

Workstation unlinked status: Let us consider SVALIAPWLAP002 is unlinked, and we got the alert for the same. To troubleshoot the issue, first log in to Dynamic Workload Console and navigate to Monitoring and Reporting ➤ Monitor workload ➤ select workstation and run. Refer to Figure 14-27.

Figure 14-27.  Monitor Workload •

This opens the following interface where we can check the status of the workstation; refer to Figure 14-28.

Figure 14-28.  Agent Status •

We can see that the agent status is unlinked; select the agent check box and select link option to link the agent server. Refer to Figure 14-29. 201

Chapter 14

Alerting and Troubleshooting Issues

Figure 14-29.  Agent Is Linked •

This will link the agent to Master server; even after providing the link if the agent is still unlinked, check the agent connectivity to Master server by running telnet command as the following. Refer to Figure 14-30.

Figure 14-30.  telnet Connection

Note  svcas0246 is the master server. •

If the issue is still existing, check the agent and master log files. Path for both the files are .

Dynamic Workload Console is not reachable: 202

Chapter 14

Alerting and Troubleshooting Issues

Let us consider DWC is not reachable and we got the mail from monitoring team. First check DWC access by logging into DWC link (https://host/ ip:9443/console); if dwc is not accessible, then there may be an issue with DWC Liberty process. Log in to Master server and switch to as the following and execute startAppServer.sh to start the DWC Liberty process. Refer to Figure 14-31 to start the DWC Liberty process.

Figure 14-31.  DWC Liberty Start •

Validate whether DWC is accessible by logging into the DWC console. If the liberty process is up and still it is not accessible, check the logs under .

Summary In this chapter, we covered the following capabilities of HWA: •

HWA alerts and configuration.

•

HWA troubleshooting steps for few use cases.

•

HWA stop and start steps in detail.

203

CHAPTER 15

HWA Reporting In this chapter, we will focus on HWA reporting features. HWA has a built-in reporting feature to generate, validate, and compare the job runtime, status, executions, and statistics. There are two types of reporting available in HWA: 1. Predefined reports 2. Personalized reports Predefined reports have a defined category in which we can select our desired data and provide the required information and run the report. In predefined report, we can generate different types of reports like the following •

Job Run Statistics – to check successful jobs %, errors %, job late start time, long running job statistics, etc.

•

Job Run History – to calculate how many times a particular application job has run over a period.

•

Workstation Workload Summary – we can check how much workload a job is utilizing on that server.

•

Workstation Workload Runtimes – workstation runtime details.

•

Actual Production Details – information to be displayed in the actual production details report output.

•

Planned Production Details – you can specify time intervals and have only the jobs that ran in these time frames included in the report output.

•

Custom SQL Report – we can run a database query by giving the required table details.

© Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6_15

205

Chapter 15

HWA Reporting

Let’s generate the predefined report for makeplan job in DWC; to generate the report, log in to DWC and navigate to Monitoring and Reporting ➤ Manage Predefined reports. Refer to Figure 15-1

Figure 15-1.  Reporting Page •

Select “Job Run History Report” and select Report Header tab.

•

Update the required description of the report. Refer to Figure 15-2

Figure 15-2.  Report Header 206

Chapter 15

•

HWA Reporting

Select Filter Criteria tab and update workstation name, job stream, and job details, and specify from and to dates. Refer to Figure 15-3

Figure 15-3.  Filter Criteria •

Select the output report format; in HWA, we have HTML, PDF, and CSV formats available. Refer to Figure 15-4. In our case, we are generating PDF format.

Figure 15-4.  Output Format 207

Chapter 15

•

HWA Reporting

Once all the details are provided and saved, we can see the report name as the following. Refer to Figure 15-5. Now we can run the report using Run option as shown in Figure 15-5.

Figure 15-5.  Report Name Page •

Once the report is executed, we will get the output in PDF format. Refer to Figure 15-6.

Figure 15-6.  Output Report Format In personalized reports, the administrator can manage and import personalized reports created with BIRT (Business Intelligence and Reporting Tool) and execute it. To generate the personalized reports, navigate to Monitoring and Reporting ➤ Manage Personalized reports. Refer to Figure 15-7. In personalized and manage personalized reports, we have to select the database type where our HWA Master is connected. In our case, database type is db2 and we are generating report for all jobs.

Figure 15-7.  Managed Personalized Report Page

208

Chapter 15

•

HWA Reporting

This opens the following interface where we need to give the job name, job stream, and workstation. Since we are generating report for all jobs, select “*.” Refer to Figure 15-8.

Figure 15-8.  Managed Personalized Report Parameters •

Select “run” to generate the report. Refer to Figure 15-9.

Figure 15-9.  Report Output •

We can see that report has been generated successfully; perform the same steps for personalized reports to get the required outputs.

Note In Hwa, we have some custom BIRT report sample for HWA available on https://github.com/WorkloadAutomation/ReportSamples for reference.

209

Chapter 15

HWA Reporting

Summary In this chapter, we covered the following capabilities of HWA:

210

•

HWA reporting capabilities and its features

•

How to create HWA reports and download them for reference

CHAPTER 16

HWA Security In this chapter, we will discuss HWA security. HWA security is managed through the security file. The purpose of the security file is to control access on one or more HWA objects created in database and plan. The security file information consists of user and groups with permissions to perform certain tasks. In HWA, we can give access to the users using two models: 1. Traditional model (file based) 2. Role-based access control

Traditional Model (File Based) This option is disabled after the master domain manager installation by default. To enable the traditional model, set “enRoleBasedSecurityFileCreation” to “Yes.” In this model, we can access security file using “dumpsec” and “makesec” commands from the command-line interface. Dumpsec command is used to decrypt the current security file into an editable configuration, whereas makesec command is used to encrypt the security file and apply the modifications. We should add the users in HWA Master server as well as DWC to give respective roles and access for users. The HWA Dynamic Workload Console has the following roles to give access to users: Administrator – the administrator can access the security, the workload definitions, workload submission, forecast, SAP, and the section monitoring and reporting. User management – the admin can customize the access on the environment.

© Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6_16

211

Chapter 16

HWA Security

Operator – the operator can manage the available plans and create a trial plan and a forecast plan. In addition, it can access workload and event monitoring. Developer – the developer can access the workload definitions and can view the preproduction plan. Analyst – the analyst can only manage the workload reports. Reporting – the operator can generate the reports from the preproduction plan. View access – only display and list access for objects. Please follow the following procedure to create scheduling and operator users in DWC. The username details are as follows: Scheduling user : HWA_Sched Operator user : HWA_Operator To create the new users, navigate to Administration tab ➤ Manage Roles.

Figure 16-1.  Manage Roles •

212

This opens the interface shown in Figure 16-2, and as per our requirement, add the “HWA_Sched” user under Developer role.

Chapter 16

HWA Security

Figure 16-2.  Selection of Roles •

To add the user, select Entities option as shown in Figure 16-2.

•

This opens the interface as in Figure 16-3; select “Add.”

Figure 16-3.  Selection of Users •

This opens the interface as in Figure 16-4 and add the user HWA_ Sched and select “Add entity” to add the user under Developer role and click “save.”

213

Chapter 16

HWA Security

Figure 16-4.  Addition of Users into Developer Role •

We can see that user HWA_Sched has been successfully added to the Developer role as shown in Figure 16-5.

Figure 16-5.  User Added to Developer Role View

214

Chapter 16

•

HWA Security

Perform the preceding same steps as to HWA_Opertor but add the user under “Operator” role as shown in Figure 16-6.

Figure 16-6.  User Added to Developer Role View Once we have added the roles to the user (local user), we must add the users in authentication_config.xml files under HWA and DWC paths, path for the files as follows:    /usr/servers/engineServer/configDropins/overrides/ authentication_config.xml /usr/servers/dwcServer/configDropins/overrides/ authentication_config.xml •

Please refer to Figure 16-7 to update the users HWA_Sched and HWA_Operator on authentication_config.xml file.

Figure 16-7.  Authentication File •

Once the users are added in xml files, let’s export the existing security file information; to export the existing information, execute dumpsec command as the following. Refer to Figure 16-8.

215

Chapter 16

HWA Security

Figure 16-8.  dumpsec Command •

Modify access.txt file and add the users “HWA_Sched and HWA_ operator” like the following. Refer to Figure 16-9 and save the file.

Figure 16-9.  Modifying the File •

Once the file is saved, execute makesec command as the following. Refer to Figure 16-10 to give access to the user.

Figure 16-10.  dumpsec Command 216

Chapter 16

•

HWA Security

We have successfully given access to the users “HWA_Sched and HWA_operator” using traditional model.

Note Usually in production, LDAP or OpenID Connect is used for user authentication; in that case, there is no need to add the users to the xml files.

Role-Based Access Control In HWA, we can give role-based security access as well. We have to define the role-based security model in the master domain manager database by using the Manage Workload Security interface from Dynamic Workload Console or the composer command-line utility. The default configuration to use role-based value “enRoleBasedSecurityFileCreation” is set to “yes,” which means that the role-based security model is enabled after the master domain manager installation. In order to check if the role-based access control is enabled, run the command as shown in Figure 16-11.

Figure 16-11.  enRoleBasedSecurityFileCreation Option Once role-based access is enabled, create the users using role-based access control. Please see the following requirements: 1. Create cust1,cust2 users and add them to group “customer.” cust1 and cust2 users should have operator, analyst, and developer roles, and they should manage HWA objects created on “C1” folder only. 2. Create tenant1, tenant2 users and add them to “Tenant” group. tenant1 and tenan2 users should have operator, analyst, and developer roles, and they should manage HWA objects created on “C2” folder only. A folder is used to organize the workload objects into different categories. 217

Chapter 16

•

HWA Security

We are using keycloak for access management, so we should add cust1&2 and tenant1&2 users on their respective groups. Log in to keycloak and add the users into respective groups. Refer to Figure 16-12 and 16-13.

Figure 16-12.  Groups Added into Keycloak

Figure 16-13.  Users Added into Groups

218

Chapter 16

•

HWA Security

Once the users are added into keycloak, add the groups into Dynamic Workload Console. To add the groups in DWC, navigate to Administration ➤ Manage role ➤ operator. Refer to Figure 16-14.

Figure 16-14.  Groups Added into Roles •

Perform the same steps to add groups on analyst and developer roles, and create the security role, domain, and access control list using composer utility command as the following. Refer to Figures 16-15 to 16-20.

219

Chapter 16

HWA Security

Figure 16-15.  Folder Creation

220

Chapter 16

HWA Security

Figure 16-16.  Security Domain Creation

221

Chapter 16

HWA Security

Figure 16-17.  Security Role Creation

222

Chapter 16

HWA Security

Figure 16-18.  Security Role Creation

223

Chapter 16

HWA Security

Figure 16-19.  Security Role Creation •

224

Perform the preceding same steps for tenant group. Once the RBAC is created on master domain manager for both the groups, we can log in to cust 1 and tenant 1 users to check the access. Log in to Dynamic Workload Console using cust1 and validate if cust1 user is able to see the objects under “C1” folder. Refer to Figure 16-21.

Chapter 16

HWA Security

Figure 16-20.  Access Control Creation

225

Chapter 16

HWA Security

Figure 16-21.  cust1 User Console View •

We can see that cust1 user is able to see only the objects under “C1” folder. Log in to tenant 1 and validate if the user can see the objects under “C2” folder only. Refer to Figure 16-22.

Figure 16-22.  tenant1 User Console View

226

Chapter 16

•

We can see that tenant1 user is able to see only the objects under “C2” folder.

•

We have successfully given access to cust1&2 and tenant1&2 users using role-based access control.

HWA Security

Summary In this chapter, we covered the following capabilities of HWA: •

Access creation for users in traditional security model

•

Access creation for users/groups using role-based access control

227

Index A

D

Alerting mechanism email notifications, 191 end users, 185 event rule, 186–190 job failure email notification, 185 job status, 188 restore outages, 191 Ansible definition, 153 inventory file, 153 requirements action tab, 157 configure job, 153 DWC home page, 154 job properties, 156 plug-in view, 155 schedule job, 159 submit job, 158 yml file, 153 Application monitoring event, 188

Daily health check, 173 Disaster Recovery (DR), 5, 6, 23–24, 35–39 dumpsec command, 211, 215, 216 Dynamic agent (DA), 9, 15–16, 19, 97, 111, 126, 133, 141, 147, 156 Dynamic pool (DP), 9–10, 15, 16 Dynamic Workload Broker (DWB), 16 Dynamic Workload Console (DWC), 14, 27 ad hoc jobs, 53 business requirements, 55 actions view, 95 Ad Hoc job, 89–91 dependency options, 71, 72 dependency selection, 78–81 design job, 56 HWA agent, 89 job log, 87 job properties, 58–60 jobs, 56 job selection, 67, 68 job status, 92, 94, 95 job stream, 61–63, 69, 70, 73, 74, 77, 78, 81, 87, 88 monitoring job, 92 object-type selection, 86 rerun options, 93, 94 run cycle, 64–66, 75, 76 submit job, 82–85

B Business Intelligence and Reporting Tool (BIRT), 55, 208

C Command-line interface (CLI), 12, 49, 211

© Navin Sabharwal and Subramani Kasiviswanathan 2023 N. Sabharwal and S. Kasiviswanathan, Workload Automation Using HWA, https://doi.org/10.1007/978-1-4842-8885-6

229

INDEX

Dynamic Workload Console (DWC) (cont.) CLI, 49 HWA login page, 50 landing page, 52 predefined job, 53 tabs administration, 51 design, 51, 52 forecast plan, 52 monitoring/reporting, 53–55 planning, 52 preproduction plan, 53 trial plan, 52 workload submission, 53 web browser, 49

FTP plug-in, 100 job created, 108 job definition, 100, 101 job submission, 108 log information, 109 source file information, 104 SSH, 98 test connection results, 103 workflow, 97, 98 destination server, 97 DWC home page, 99 security protocols, 97

G GitHub, 48

E

H, I

“exit 1” command, 166 Explicit File Transfer Protocol (FTPES), 97 Extended agent (XA), 10, 19

HCL workload automation (HWA), 1, 49 advantages, 11, 12 application HA capability, 14 architecture types, 21–24 audit, 14 communication ports, 20 components, 6, 7, 9, 10, 13–16 console, 7 DA, 15 database, 16 definition, 6 DWB, 16 FTA, 15 HWA master, 13 interprocess communication, 17 JnextPlan, 11 KPIs, 6 liberty, 16

F Fault tolerant agent (FTA), 8, 9, 14, 15, 17, 19, 37, 38, 97, 111, 133, 156 File monitoring event, 188 File Transfer Protocol (FTP), 97, 98, 100, 107 File Transfer Protocol Secure (FTPS), 97 File transfers business processes, 98 connection properties, job, 102 destination file information, 105, 106 destination server files, 110 FTP options, 107 230

INDEX

messaging communication, 17–19 multiple deployment options (see Planning deployments) network, 8 Process Status Command, 193 reporting personalized, 205 predefined, 205 scheduler, 13 software, 8 terminologies, 10 types, 205 HWA event rule management auto-remediate job failures agent running, 171 batchman process, 167–169, 171, 172 event rule properties, 169 event status, 170 HWA process, 167 server log file, 171 ServiceNow, 172 actions, 165, 166 add event options, 163 create event rule, 162 definition, 161 event parameters, 164 job failure ticket, 162 job status, 164, 166 xml format, 161 Hype cycle for IT operations management, 1

J JNextPlan, 10, 11, 173, 174 Job scheduling software, 2

K Kubernetes cluster HWA capabilities, 144 batch jobs, 141 cluster config file information, 143 connection parameters, 142 job definition properies, 142 job plug-in, 141 mysql-deployment, 139 process information, 143 submit job, 144 HWA dynamic agent, 139

L Liberty, 10, 16, 21, 27, 32, 33, 35, 36, 191, 192, 196, 197, 203 Liberty Process Stop command, 192

M, N makesec command, 211, 216 Master domain manager (MDM), 8–10, 19–22, 27, 38, 56, 167, 173, 217, 224 MDM_DA agent server, 98 Microsoft Azure definition, 145 VM, 145 client/tenant, 148 connection parameters, 149 create new VM, 151 DWC home page, 145 exiting VM options, 150 job definition, 147 plug-in, 147 submit job, 151, 152

231

INDEX

Microsoft SQL server definition, 123 HWA integration, 129 use cases, 123 database selection, 127 DWC home page, 124 HWA job properties, 126 job selection, 128 KPI, 123 plug-in, 125 submit job options, 129

O “optman ls” command, 195 Outages db connectivity check, 195 DWC liberty status, 197, 198 DWC process start commands, 197 HWA process status command, 193 HWA process stop commands, 193 HWA start commands, 196 liberty process, 192 planned, 191 startup command, 194 types, 191 unplanned/emergency, 194

P, Q Personalized reports BIRT, 208 HWA Master, 208 output, 209 report parameters, 209 Planning deployments containerized deployment, 39–47 disaster recovery, 35–39 232

high availability architecture, 32–35 HWA deployment, 47, 48 memory requirements, 26 prerequisites, 25 stand-alone architecture, 27–32 Pool, 9, 15, 16 Predefined reports definition, 205 filter, 207 header, 206 name page, 208 output format, 207, 208 reporting page, 206 types, 205

R Relational database management system (RDBMS), 19, 123 REST API definition, 131 HTTP methods, 131 HWA, 131 use case authentication, 135 DWC home page, 132 GET, 132 HWA rest api job, 136, 137 job properties selection, 134 methods, 135 plug-in page, 133 service URL, 138 submit job, 137 web URL, 132 “rmstdlist” script, 176, 178 Role-based access control (RBAC), 14 access role creation, 225

INDEX

add groups, 219, 220 enRoleBasedSecurityFileCreation, 217 keycloak, 218, 219 requirements, 217 security domain creation, 221 security role creation, 222–224 tenenat1 user control view, 226 user control view, 226

S SAP R3batch access method, 111 SAP-related event, 188 Scheduler, 2, 3, 5, 6, 13, 51, 111, 191 script/command, 53, 153 Secure Shell (SSH), 15, 97, 98, 101, 156 Security file role-based access control, 211 traditional model (file based), 211 ServiceNow, 161–167 Sinfonia, 11 Stageman, 11 Stand-alone architecture, 21, 27–33, 35, 36, 39, 47 Standard agent (SA), 10, 19 StartAppServer, 10, 196, 203 StartUp Command, 194, 195 SymNew, 10, 11 Symphony, 10, 11, 37 Systems Applications and Product (SAP) ABAP jobs, 111 business processes, 111 design page, 116 job status, 122 job submission, 120 library files, 113, 114 monitoring workload, 121 plug-in, 117 properties, 118, 119

R3batch options, 114, 115 workflow, 112 enterprise scheduler, 111 plug-in, 122 products, 111

T Time-based and event-driven scheduling, 3 Tool administration daily health check, 173–175 database backups/restore, 183 database maintenance, 181–183 housekeeping maintains application, 176, 181 DWC home page, 177 HWA job properties, 178, 179 HWA Unix job, 180, 181 log files, 176 Unix job, 177 Traditional model (file based) authentication file, 215 developer roles, 214, 215 DWC, 211, 212 enRoleBasedSecurityFileCreation, 211 managed roles, 212, 213 modifying file, 216 Troubleshooting techniques agent status, 199–201 DWC, 203 linked agent, 202 monitor workload page, 199 telnet connection, 202 use cases, 198

U, V Unplanned/emergency outages, 194 233

INDEX

W, X, Y, Z Workload automation application-level high availability, 5 application platform/integration, 3 architecture, 5 batch processing, 4 batch/workflow, 5 disaster recovery, 6

234

events/errors, 5 features, 1 handle/store outputs, 4 job stream, 1–3 requirements, 1, 2 security, 6 self-service capabilities, 5 virtual/physical resources, 4