BizTalk: Azure Applications 9781501505652, 9781501514760

Citation preview

Suren Machiraju and Suraj Gaurav BizTalk

Suren Machiraju and Suraj Gaurav

BizTalk

Azure Applications

ISBN 978-1-5015-1476-0 e-ISBN (PDF) 978-1-5015-0565-2 e-ISBN (EPUB) 978-1-5015-0564-5 Library of Congress Control Number: 2018941010 Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available on the Internet at http://dnb.dnb.de. © 2018 Suren Machiraju and Suraj Gaurav Published by Walter de Gruyter Inc., Boston/Berlin Printing and binding: CPI books GmbH, Leck Typesetting: MacPS, LLC, Carmel www.degruyter.com

Life is a wonderful journey and I am blessed to share it with my amazing wife, Aparna. Thank you. Suren Machiraju I want to dedicate this book to my loving and generous father Shri Surendra Kumar Sinha. Suraj Gaurav

DOI 10.1515/9781501505652-201

About De|G PRESS Five Stars as a Rule De|G PRESS, the startup born out of one of the world’s most venerable publishers, De Gruyter, promises to bring you an unbiased, valuable, and meticulously edited work on important topics in the fields of business, information technology, computing, engineering, and mathematics. By selecting the finest authors to present, without bias, information necessary for their chosen topic for professionals, in the depth you would hope for, we wish to satisfy your needs and earn our five-star ranking. In keeping with these principles, the books you read from De|G PRESS will be practical, efficient and, if we have done our job right, yield many returns on their price. We invite businesses to order our books in bulk in print or electronic form as a best solution to meeting the learning needs of your organization, or parts of your organization, in a most cost-effective manner. There is no better way to learn about a subject in depth than from a book that is efficient, clear, well organized, and information rich. A great book can provide life-changing knowledge. We hope that with De|G PRESS books you will find that to be the case.

DOI 10.1515/9781501505652-202

Acknowledgments BizTalk has been nearly a two-decade long journey for us and we want to take this opportunity to acknowledge the contribution of Microsoft engineers, field staff, partners, and most importantly the developer community. Every interaction with you all has led to information which eventually culminated in this publication. In the context of this book, we wish to acknowledge the contribution of three friends: Brajesh Sinha, Rahul Jain, and Jennifer Curiak. Brajesh is a rising star in the BizTalk developer world and provides great insights into BizTalk deployments. Rahul is an amazing technical editor, and last but not least, Jennifer does an amazing job of technical proofreading and cleaning the book up. Thank you. Suren Machiraju Suraj Gaurav

DOI 10.1515/9781501505652-203

About the Authors

Suren Machiraju developed an innovative supply chain solution integrating online stores with market makers and aggregators, founding Commercia Corporation in the late 1990s. Within one year, Microsoft acquired Commercia Corp, providing Suren the opportunity to lead the business to business (B2B) interoperability team within the BizTalk business unit. Over the next six years, Machiraju’s team delivered five releases of the BizTalk Server (2000-2006R2). Subsequently, Machiraju led the BizTalk Rangers-Customer Advisory Group, and in two years, lit up over twenty of the largest middleware deployments on the .NET stack. In 2011, Suren collaborated to create the Azure Customer Advisory Team at Microsoft. For five years, Machiraju led efforts in engaging enterprise customers, start-ups, and partners for architectural reviews and deployments of cloud/ hybrid cloud .NET and OSS applications on the Azure platform. The team pioneered solutions for the most challenging cloud projects producing dozens of successful deployments. Most recently, in 2014, Suren accepted an appointment as a Technology Business Partner at the Bill & Melinda Gates Foundation where he collaborates with leading nongovernmental organizations and nonprofit partners in devising technical solutions for some of the world’s most challenging social issues. Machiraju holds a master’s degree in Mechanical Engineering from the Birla Institute of Technology and Science in Pilani, India. He is a listed author of over twenty patents in business software areas of B2B and Data Interchange Standards and has published books and authored dozens of MSDN articles/technical blogs on Azure and .NET. When he is not publishing blogs or presenting works to the larger technical community, he is enjoying time with his family in the beautiful Pacific Northwest and cheering on the Seahawks each Sunday. “Please contact me if I can be of assistance in architecting your cloud-based solution, collaborating in this space is one of my greatest passions”—Suren. https://about.me/surenmachiraju DOI 10.1515/9781501505652-204

xii 

 About the Authors

Suraj Gaurav started his career in 2000, at the height of the dot-com era. He worked in a start-up, Asera, that was developing a revolutionary platform for building business to business applications. In 2002 he moved to Seattle to work for Microsoft. He spent almost ten years there and worked on various products including BizTalk Server, the Commerce platform, and Office 365. He has in-depth experience building enterprise-scale systems like BizTalk, to internet scale services like Office 365. He also built the consumption-based billing platform serving as the commerce engine for Azure. Gaurav holds a bachelor’s degree in Computer Science from the Indian Institute of Technology, Kanpur, India. He is listed as an inventor in over twenty-five patents. When he is not working, he can be found spending time with family and enjoying the beautiful outdoor life of the Pacific Northwest.

About the Technical Reviewer Jennifer Curiak specializes in Dynamics 365 implementations, agile coaching, project management, business analysis, quality assurance, and technical writing. She works to help teams in a variety of industries become more productive, communicate more effectively, and generally get stuff done. A writer at heart, Curiak started her career as a technical writer for a software company in 2000 and has evolved into designing solutions, managing QA processes and resources, coaching large and small teams in agile development practices, acting as scrum master, and working on Dynamics 365 customizations and implementations. She was the technical reviewer on the book Administering, Configuring, and Maintaining Microsoft Dynamics 365 in the Cloud in 2018, continues to write in-house technical documentation and end-user documentation, and contributes to other professional publications. Jennifer and her husband Mike live in Western Colorado and spend most of their free time exploring empty and desolate areas of the West by mountain bike and packraft. She can be contacted directly at [email protected]. Jennifer Curiak C) 720-933-5587

DOI 10.1515/9781501505652-205

Foreword

It’s a privilege to write the foreword for this book, BizTalk: Azure Applications, by my colleagues Suren and Suraj. My company, GCommerce has been at the forefront of adopting cutting-edge technologies—BizTalk in 2004 and six years later, Azure in 2010. When Suren reached out to me for the foreword, I found it very apt to share my thoughts on BizTalk: Azure Applications. BizTalk is the backbone of our operations, allowing us to solve complex problems in distribution supply chain markets, while transacting millions of dollars every month and while it has its issues, including the licensing model, it does its job well. With the advent of cloud and prevalence of open source technologies, I believe we are at a crossroad here. It’s my belief that for the 2020 decade we have the opportunity to leverage Azure Cloud Services and build the next generation of integration technologies that tie in EDI transaction processing with Blockchain, AI, and IoT. I look forward to leading this initiative and invite the community to join me. The book, BizTalk: Azure Applications, is very relevant for fast scaling companies and those aspiring to drive change in the next decade. After setting context to BizTalk 2016, this book extensively covers how we can develop the next generation of integrated solutions using Azure Cloud Services and open source technologies. I especially like how various technology choices are contrasted. I thank Suren and Suraj for authoring this very timely book and wish them success in this and future endeavors. Jason Popillion CTO, GCommerce Inc. https://www.linkedin.com/in/jason-popillion-863a464/

DOI 10.1515/9781501505652-206

Contents Chapter 1: Evolution of BizTalk  1 Component Object Model (COM)  1 Programming Languages for COM Components  2 Microsoft Transaction Server (MTS)  2 MTS Explorer  3 Distributed Component Object Model (DCOM)  4 Language Independent  5 Protocol Independent  5 Garbage Collection  5 Reusability  5 Microsoft Site Server 1.0 6 Microsoft Merchant Server  7 Microsoft Site Server 2.0  7 Microsoft Commerce Server  8 Electronic Data Interchange (EDI)  8 First EDI Message  9 Transportation Data Coordinating Committee (TDCC)  9 First EDI Standard, SWIFT, and EDIA  10 EDIFACT  10 EDI Today  11 BizTalk Server  11 BizTalk Server 2000  11 BizTalk Server 2002  12 BizTalk Server 2004  12 Covast  13 BizTalk Server 2006  17 BizTalk Server 2006 R2  17 BizTalk Server 2009  18 BizTalk Server 2010  18 BizTalk Server 2013  19 BizTalk Server 2013 R2  20 BizTalk Server 2016  20 BizTalk Server 2016 Feature Pack 1  21 BizTalk Server 2016 Feature Pack 2  22 Windows Azure BizTalk Services (WABS)  23 Azure BizTalk Services  23 Summary  24 DOI 10.1515/9781501505652-207

xviii 

 Contents

Chapter 2: BizTalk Integration Patterns  25 Enterprise Application Integration (EAI)  25 Orchestration  29 Exploring Orchestration Design Surface  31 Business to Business (EDI, HIPAA, and XML)  32 Electronic Data Interchange (EDI)  32 Health Insurance Portability and Accountability Act (HIPAA)  33 XML  34 B2B and EAI with BizTalk  34 Implementing the Process through BizTalk  37 Summary  49 Chapter 3: BizTalk Server 2016 Features  51 New and Enhanced Features of BizTalk Server 2016  51 Exploring the Features Related to EAI Integration  52 Artifacts Analytics Tracking  52 Azure Logic Apps Adapter  61 Support for SAP .NET Connector (NCo)  77 Connect to SQL Server Always Encrypted Column from BizTalk Server  77 Exploring the Features Related to Business Process Control Integration  84 Map Compilation  85 Ordered Delivery on Dynamic Send Ports  87 Advanced Location Scheduling  88 Exploring the Features Related to the Business to Business Integration  90 Connecting to Azure File Share from BizTalk File Adapter  90 Refined B2B Import/Export Process  98 Improved FTP and SFTP Adapters  102 Exploring the Improvements to Miscellaneous/UI Features  104 Simultaneous Configuration of Multiple Hosts/Host Instances  105 Searching Artifacts  106 Save Multiple Suspended Messages Simultaneously  106 Connect to Management REST APIs  107 Operational Data Feed for Power BI  108 Resizable Schema Window  112 BizTalk Server 2016 Azure VMs in Production  113 Summary  114

Contents 

Chapter 4: BizTalk in Microsoft Azure Services  115 BizTalk Infrastructure as a Service  115 BizTalk Integration Platform as a Service  116 Azure Logic Apps  116 Integration Account  118 Hybrid Connections  138 EAI Implementation Using BizTalk Azure Services  138 Building Required Artifacts  139 Implementing Workflow in Azure Logic Apps using Integration Account  143 Testing the Workflow  144 Extending the Scenario using Hybrid Connection  146 Summary  154 Chapter 5: Web 3.0, Custom BizTalk Azure Application  155 Introduction  155 BizTalk Alternatives  155 Pipeline Processing Alternatives  155 Transformation Alternatives  158 Orchestration/Workflow Alternatives  159 A Simple Stateless End-to-End Process Implementation Using Alternatives  161 Windows Service  162 The FileSystemWatcher Component  163 Configuration File  164 Moving to Business Process  165 Transform/Mapping  166 Code Implementation  171 Installing Order Processing Service  173 Testing the Order Processing Service  175 Extending the Scenario (EDI Processing and Persistence)  178 EDI Processing  179 Persistence Scenario  180 Implementation  181 Testing the Extended Scenario  196 Extending a Bit More: Generate ACK  199 Summary  202

 xix

xx 

 Contents

Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration  203 Introduction  203 Key Comparison Pivots  203 Application Integration Capabilities  204 Long-Running Transactions  204 Dynamic Message Processing and Routing  207 Business Rules Execution  208 Workflow Support  210 End-to-End Transactional Support  211 Tracking System  212 EDI Business Processes  215 Message Protocol Support  215 Protocol and Line of Business Connectors  216 Developer Experience  217 Scalability  218 Maintainability  220 Ease of Implementation  220 Cost of Ownership  221 Summary  224 Chapter 7: Partner Solutions for BizTalk Azure Applications  227 Introduction  227 BizTalk360  227 Operations  231 Monitoring  246 Analytics  249 WPC (WPC-EDI)  252 HIPAA Database Toolkit  252 Installing Database Toolkit  253 Enabling the CLR in the SQL Server  255 Generating EDI (Version 5010) Transaction DDL Scripts  255 Using DDL Script to Configure EDI Transaction Database  256 Using the Database Toolkit (DBToolKit) Adapter  259 Neudesic  264 What is Enterprise Service Bus (ESB)?  265 BizTalk ESB Toolkit  265 BizTalk ESB Toolkit Components  265 Installation  266 Configuring ESB Toolkit  268

Contents 

Configuring ESB BizTalk Applications  270 ESB Management Portal  272 BizTalk ESB Implementation  277 Implementing Dynamic Routing Based on Message Type  277 Creating an ESB Itinerary Application  283 Defining the Sequence of Execution (Using Connectors)  291 Validating the Itinerary  292 Deploying the Itinerary  293 Testing ESB Implementation  295 Summary  297 Chapter 8: Summary—Three Valuable BizTalk Azure Application Options  299 Introduction  299 History of BizTalk  299 BizTalk Integration Patterns   300 Features of BizTalk Server 2016  301 BizTalk in Microsoft Azure Services  303 Web 3.0 as a BizTalk Alternate  303 Compare BizTalk IaaS, BizTalk PaaS, and Custom Integration  305 Microsoft Partners in BizTalk Domain  307 Conclusion  308 Index  309

 xxi

Preface We had the rare privilege of being associated with BizTalk from its version v1— BizTalk Server 2000, to its current avatar as Azure Services. While many features have come and gone, the core of BizTalk—its robust runtime engine, support for electronic data interchange, and integration with Visual Studio have ensured relevancy even after two decades. BizTalk was born in the days when automated server-to-server communication software would cost upwards of half a million dollars. BizTalk democratized and disrupted the market by pricing it at about 1% of the cost of competing technologies. Sure BizTalk has its challenges with one of its biggest advantages being its drawback too. BizTalk Platform is often compared to a Swiss Army knife—the one tool that does many things. BizTalk offers solutions for many varied domains and use cases; yet does not provide any holistic end-to-end solution and therefore requires the enterprise developer to round it off. In the two decades since v1 release, the technology landscape has changed dramatically. Today, Web 3.0 enables smart applications to search, auto-discover, and communicate with each other using microservices and best-of-breed open source technologies. A good architect-developer is able to draw on these microservices and open source technologies to put together a lightweight integration solution. The bottom line is that you now have the choice to compose a right-sized and right-featured solution that is perfect for you. We as practitioners of BizTalk, produced this book from real-world implementations—the code samples and scenario descriptions are real and can be applied directly to your implementations. The book also gives you an insight into some very useful partner solutions—all these partner solutions fill the feature gaps left behind by BizTalk. Again these are used and verified by us. The book is organized into eight chapters—we attempted to make each chapter stand-alone, to provide you all the relevant information in one read without cross-referencing. Chapter 1 was a walk down memory lane—we traced the birth of BizTalk, important to know so you can better gauge the future. Chapter 2 ensures we have a common understanding of the integration patterns. Chapters 3 and 4 walk you through the best new features in BizTalk 2016 and BizTalk Azure Services. Chapter 5 is huge—it provides you patterns and know-how to architect your custom integration solution using open source and Azure Microservices as building blocks. Chapter 6 contrasts available options and assists you with decision making. As with any technology offering, there are bound to be gaps and in Chapter 7 you will learn of three partner offerings that will provide huge value-add to your solution—cheaper to buy them versus DOI 10.1515/9781501505652-208

xxiv 

 Preface

building them out. Chapter 8 summarizes what you have learned and closes out the book. We thank you for showing interest in this book and investing in it. Let’s keep BizTalking! Best wishes, Suren Machiraju Suraj Gaurav Issaquah, WA March 2018

Chapter 1 Evolution of BizTalk BizTalk is synonymous with integration. It is an integral part of tech jargon and is often used as a verb! BizTalk Server was first released in 2000, nearly two decades ago, and is still around. So why do businesses continue to use Microsoft’s BizTalk Server as the backbone to integrate line-of-business applications with their trading partners, and how do recent changes make it even more effective? The quick answer to the question is versatility—BizTalk is akin to a Swiss Army knife—it has the breadth and depth to provide varied integration solutions. With the advent of Azure, BizTalk functionality is further enhanced while at the same time the cost of operations and maintenance is reduced. In this chapter, we will familiarize you with BizTalk’s evolution—how it all came together, and equally as important, how it has evolved over the past two decades. BizTalk Server was introduced in a decade when electronic digital data transfer was looking more and more plausible. The concept of BizTalk Server came from Component Object Model (COM) technology, which allowed software components to communicate with other components. Later, other technologies including COM+, DCOM, Windows Runtime, ActiveX, and OLE, etc., were introduced. These technologies were all based on the COM technology. The first version of BizTalk Server was launched in 2000 and was named BizTalk Server 2000. The motivation for releasing BizTalk Server was to provide integration services between software entities sans coding and make it an ITPro play. BizTalk Server has come a long way since it was first introduced nearly two decades ago. From the server world, technology has gone serverless via BizTalk PaaS Azure Logic Apps. Let’s follow this journey together.

Component Object Model (COM) COM is a standard introduced by Microsoft in 1993. It provides a system through which software components can be created. These components can be used to communicate with other components in a system. It is the standard that defines programming requirements and the object model to allow COM components (or COM objects) to communicate with other objects. A COM component is not limited to communication with components within the same process. It can com-

DOI 10.1515/9781501505652-001

2 

 Chapter 1: Evolution of BizTalk

municate with components in other processes running on the same computer or different computers. COM defines a binary object standard, which states that all binary modules (DLLs and EXEs) of a COM component should match a specific structure after compilation. These binary modules should be language independent so that they can be used and accessed by any programming language that meets requirements including platform compatibility. COM not only defines a basic binary object standard, but it also defines a few basic interfaces. These interfaces contain common methods required by all COMbased technologies. COM defines a standard for COM components to communicate with other components in a network. It also adds several security features to provide integrity to systems in a network.

Programming Languages for COM Components Like other object-oriented standards, a COM component is a set of data and methods. In a COM object, data can be accessed through a set of related functions, called an interface. As mentioned earlier, a COM component should be language independent. Therefore, functions (methods) of an interface should be accessible through a pointer to the interface. As interface methods should be accessible through a pointer to the interface, we can create a COM component in any programming language that allows us to create a structure with the pointer either explicitly or implicitly. Earlier, most of the COM components were created using C++ as pointers and, it supported structural features. These features simplify the process of creating and implementing a COM object. However, today, several other languages including C#, VB, and JAVA allow you to create and use COM objects.

Microsoft Transaction Server (MTS) The next step in COM evolution was to effectively manage COM components that were required to complete a solution. To handle COM components effectively, Microsoft introduced Microsoft Transaction Server (MTS), which is a component-based transaction processing system. A transaction processing system enables a developer to build and administer network applications.

Microsoft Transaction Server (MTS) 

 3

MTS provides a development framework that allows us to create components that can be used in transactions easily. MTS is software that acts as a client to manage applications and databases across networks. Launched by Microsoft, MTS was launched to allow users to create applications and data in distributed environments easily. MTS allows users to drag and drop components to create a transaction model. This model was initially created for a single user. Eventually, the same transaction model was usable by multiple users. MTS provides several features. Some of them are shown in Figure 1.1. It provides a runtime environment for COM components It supports the DCOM programming model to develop distributed applications It provides a runtime environment to deploy distributed applications It provides an option to deploy ActiveX components It allows users to access Component Caching and Database Connection Pooling It provides the ACID properties for components in transaction(s) Figure 1.1: Features of MTS

MTS Explorer MTS Explorer was a Microsoft Management Console (MMC) snap-in. This snap-in helped developers administer and use MTS applications. A collection of components inside a Transaction Server was called a package. These components worked collectively to create an application. You could save a package as a redistributable component called a prebuilt package. This package contained the relevant information required to install the package on another system. MTS Explorer displayed information about each component of a package including its methods and interfaces. It was also used to set properties of a selected component.

4 

 Chapter 1: Evolution of BizTalk

Note MTS Explorer was a part of Windows NT 4.0 Option Pack. However, it is no longer available. You can find more information about MTS Explorer at the following link: https://msdn.microsoft.com/en-us/library/aa480416.aspx

Distributed Component Object Model (DCOM) Via COM, businesses could connect components in a monolithic environment or deployment. The next stage in evolution was connecting components in a distributed deployment, which lead to Distributed Component Object Model (DCOM). DCOM is a Microsoft technology that provides standards for allowing components to interact with other components in a distributed environment. DCOM was available with the initial release of Windows NT 4.0 in 1995. It was an extension of the COM technology. Like COM, DCOM provides an interface between objects or components on different computers/networks. Did You Know? In the context of providing distributed services, DCOM is equivalent to Common Object Request Broker Architecture (CORBA). DCOM is a Microsoft technology whereas CORBA is sponsored by the rest of the IT industry under the Object Management Group.

DCOM uses the Remote Procedure Call (RPC) mechanism to send and receive information between COM components across the network. There are several features of DCOM. Some of them are shown in Figure 1.2. Features of DCOM

Language independent Protocol independent Garbage collection Reusability

Figure 1.2: Features of DCOM

Let’s discuss each feature in some detail.

Distributed Component Object Model (DCOM) 

 5

Language Independent As discussed earlier, DCOM is an extension of COM. As COM is language independent, DCOM is also considered language independent due to its binary specification. Different programming languages including C++, VB, C#, and Java, etc., can be used to code DCOM components.

Protocol Independent DCOM provides an interface to allow communication between components on one computer with components on another computer located on the network. Communication with components on another computer may require different protocols. DCOM supports different types of protocols. Therefore, we do not need to worry about the protocol to be used for the communication between DCOM components. Also, we do not need to recompile a DCOM component even if a protocol is changed during communication.

Garbage Collection DCOM components are called by a remote client. After receiving the required output from the connected DCOM components, the client releases the connection to the DCOM components. However, the connections remain active on the remote machine but are not useful as they are used only when connected to the client. Here, they consume or block available memory on the remote machine. This problem can be solved by using the automatic garbage collection feature of DCOM. DCOM uses a pinging protocol to check whether the client is active or not. At the same time, the client machine pings a message to the DCOM Server. If the DCOM Server does not receive the message, DCOM assumes that the connection with the client is not active and the DCOM component is removed from memory.

Reusability One of the most important features of DCOM is its reusability, which is defined as its ability to reuse existing components within a distributed environment. Therefore, a COM component can be used as a DCOM component without changing the

6 

 Chapter 1: Evolution of BizTalk

client-side or component code base with the help of DLL surrogates and location transparency. DLL surrogates and location transparency are described as follows: –– DLL Surrogates: Most assemblies (DLLs) run as the in-process DLLs. As we know, an assembly is not a self-executable piece of software, which means that it must be called from an executable process. Then, the DLL is called to let it run in the same process in which the executable process is running. A surrogate process was first introduced with Windows NT 4.0 Service Pack (SP). A surrogate provides the ability to host/run the DLLs outside of the process that has requested it. COM allows you to build DLL servers. Once the DLL servers are built, they can be inserted into a surrogate executable process. –– Location Transparency: You can change the configuration of a COM component by making modifications to the registry. After modifying the registry, the component is registered on the remote machine to make it a DCOM component. In actuality, the location of a component is hidden by DCOM. You can provide a new location for the created DCOM component by changing the registry settings. A component can be either DLL type or EXE type. Based on the type of component, the update required to registry settings may be different. Each DLL or EXE has a unique identifier in the form of a GUID value. Therefore, use the GUID to search for each component. Once the component is found, the following keys and required values need to be added to that component. [HKEY_CLASSES_ROOT\APPID\{}] "RemoteServerName"= and [HKEY_CLASSES_ROOT\CLSID\{}] "AppId"=""

Microsoft Site Server 1.0 Another trend prevalent in the late nineties was e-commerce. E-commerce websites integrated the content and the commerce. Commerce also required integrating the Order to Cash business process—from the moment a customer places an order on a website; transmitting the order to the warehouse; informing the buyer of the shipment; charging the credit card and potentially dealing with returns and credits. Microsoft solved the e-commerce solution via Site Server.

Microsoft Merchant Server 

 7

Site Server is an Intranet Server that launched specifically for Windows NT Server with IIS. It was first released in 1996. It has several useful features, some of them mentioned below: –– Content management and analysis –– Search and full-text indexing –– User management –– Authentication and authorization

Microsoft Merchant Server Microsoft Merchant Server was launched in October 1996. It was introduced to provide e-commerce solutions. For this, Microsoft acquired a company named eShop, which was a product-based company and created a product using Python. Microsoft used this product and added modules to plug it into IIS and run the primary server code as an NT service. This was the first and last version of Microsoft Merchant Server. Later, most associated technologies were migrated to Microsoft Site Server 2.0 (MSSC).

Microsoft Site Server 2.0 After launching Microsoft Merchant Server, Microsoft also started shipping Active Server Pages in December 1996. Microsoft Merchant Server was unable to use ASP technology. Therefore, Merchant Server technology was converted into COM objects, which were used by Active Server Pages (ASP). These changes and ASP pages were incorporated into Microsoft Site Server 2.0. Microsoft Site Server 2.0 was also known as Microsoft Site Server Commerce (MSSC). It provided simplified e-commerce solutions. It inherited most of the functionalities and technologies of Microsoft Merchant Server. Distinct features of Microsoft Site Server 2.0 were as follows: –– It provided a comprehensive website environment for managing rich intranet and internet websites. –– It used advanced web server technology and helped customers and solution providers in resolving business-critical functions. –– It provided indexing and search features. –– It provided content management and product management. –– It provided order processing and site personalization.

8 

 Chapter 1: Evolution of BizTalk

After the introduction of Microsoft Site Server 3.0 (published in 1998), the e-commerce functionality was moved to Microsoft Commerce Server while its content management and document management capabilities were moved to Microsoft Content Server.

Microsoft Commerce Server In 2000, Microsoft introduced Microsoft Commerce Server as a replacement for Microsoft Site Server. It added more functionalities and tools for web developers to effectively manage e-commerce applications. Microsoft Commerce Server allowed users to create high-performance e-commerce solutions. It provided several tools that simplified setup, management, and administrative tasks. Common features of Microsoft Commerce Server were as follows: –– A default site with 30 web parts and controls –– Self-service portals using catalogs, profiles, or content –– Multichannel functionality –– Management of ads and set rules for ads –– Profile management –– Service-oriented architecture –– Catalog, order, and inventory management –– Comprehensive solution for business to business (B2B) and business to consumer (B2C) scenarios –– Data integration with third-party systems –– 64-bit support Microsoft provided support to Microsoft Commerce Server from 2000 to 2009. Within that period, Microsoft released several versions of Commerce Server. Service Packs (SPs) were also released for most of the versions of Commerce Server to provide solutions to the issues raised by developers. The last version of Microsoft Commerce Server was Microsoft Commerce Server 2009 R2. Subsequently, Sitecore had taken over the development, shipping, and support responsibilities of Microsoft Commerce Server.

Electronic Data Interchange (EDI) There is one other technology we must understand before we get into BizTalk— Electronic Data Interchange (EDI). In earlier pages, we mentioned Order to Cash. The warehouses, credit card processors, shipping companies, etc., used

Electronic Data Interchange (EDI) 

 9

an automated machine-to-machine lightweight protocol to communicate their transactions. It was important to integrate this protocol with the e-commerce application in order to scale the Order to Cash business process. So, let’s understand EDI better. EDI is a communication method that allows transmission of business documents electronically between the trading partners. Note Trading partners are the business entities who are involved in the process of exchanging documents electronically.

First EDI Message This first EDI message was developed by Ed Guilbert, who is also known as the father of EDI. During the Berlin airlift, Ed Guilbert worked with the U.S. Army officers to develop a message format for sending cargo information. The first EDI message was a Telex message. It was exchanged in 1965 for the first time between shipping companies. Holland-American steamship line sent it to Trans-Atlantic shipping. At that time, a full-page of information could be sent in approximately two minutes without using any file transfer protocol. These messages were loaded into a magnetic tape and then loaded into the computers.

Transportation Data Coordinating Committee (TDCC) In 1968, a group of transportation industries formed a committee called the Transportation Data Coordinating Committee (TDCC) to develop and manage EDI standards, formats, and protocols. The committee contained several hundred people who were separated into different teams to develop industryspecific standards. The basic rules followed by TDCC while developing standards were: –– The EDI interface should not be sensitive to the internal architecture of computer equipment. –– The EDI interface should respond as per the requirements of end users. After successful setup, these standards were termed TDCC EDI standards. These standards were primarily used for rail lines. Nowadays, they are used for exchanging electronic messages.

10 

 Chapter 1: Evolution of BizTalk

First EDI Standard, SWIFT, and EDIA TDCC released its first EDI standard in 1975. This EDI standard became the base mode of data transmission in 1977. In the same year, the Society for Worldwide Interbank Financial Telecommunications (SWIFT) established the SWIFT messaging system, which was based on the EDI system. SWIFT defined standard messages used for interbanking communication. In 1978, TDCC was renamed the Electronic Data Interchange Association (EDIA). In the same year, it was adopted by the American National Standards Institute (ANSI) and was renamed the ANSI-X12 committee. The primary responsibility of this committee was publishing EDI standards. In 1981, the ANSI-X12 standards were published for different industries including banking, food, transportation, drug, and warehouses. In 1982, reputed automotive and retail industries started adopting EDI standards for data transmission and made it compulsory for their suppliers.

EDIFACT In 1985, the United Nations created the EDIFACT EDI standard to support EDI technology globally. While EDIFACT is adopted globally, U.S. businesses still use EDIFACT-X12. The bottom line is that both standards are in play and global solutions need to support both. In 1996, the communication of EDI data over the internet became standardized through EDI over the Internet (EDIINT), which was introduced by the Uniform Code Council (UCC). This standard provides the following components: –– A set of syntax rules –– Interactive exchange protocol –– A set of standard messages to allow interaction between trading partners AS2 In 2000, the Uniform Code Council (UCC) published the AS2 communication standard. This standard allows users to transmit data over the internet in encrypted form using the HTTP protocol. In 2004, this standard was accepted by large retail chains, such as Walmart, Target, and Lowe’s to communicate with their suppliers.

BizTalk Server 

 11

EDI Today As EDI standards have been developed for all types of industries, almost all industries are adopting EDI for data transmission irrespective of their type. Now, 90% of Fortune 500 companies and over ten million businesses in the United States are using an EDI solution to communicate with their customers, suppliers, and other stakeholders. All major industries using EDI follow Compliance EDI, which is a set of standards that govern the transmission of electronic documents between companies, and streamline the supply chain. Each industry publishes its own set of guidelines on how to implement these standards. Different EDI standards are used in different regions, for example, X12 is used in the United States, while EDIFACT is used in Europe and Asia.

BizTalk Server The birth of BizTalk came from the words “Business Talk,” which were words commonly used to describe the need to automate communications between internal and external business applications.

BizTalk Server 2000 The first version of BizTalk Server was launched in 2000 and was named BizTalk Server 2000. This version of BizTalk Server included several features, such as XML editor, BizTalk Orchestration Engine, and Mapper. It provided the BizTalk Orchestration Engine feature for business process management, the Mapper feature to allow data to be read by different systems, and the XML editor to facilitate cross-platform communication. BizTalk Server 2000 provided the following features: –– XML Tools –– Messaging –– XLang –– BizTalk Orchestration Engine –– Document tracking

12 

 Chapter 1: Evolution of BizTalk

Two additional SPs for BizTalk Server 2000 were launched in 2001 and 2002 with version numbers 3.0.1764.0 and 3.0.1764.1, respectively. These SPs were introduced to fix bugs found in BizTalk Server 2000.

BizTalk Server 2002 BizTalk Server 2002 was introduced to provide enhancements to existing BizTalk Server 2000 services. In BizTalk Server 2002, Mappers were updated to provide better XML support. Orchestrations and adapters were updated to provide more flexibility. In BizTalk Server 2002, additional development and deployment tools were added to meet demands of developers. BizTalk Server 2002 provided the following features: 1. Deployment tools 2. XSDs 3. EAI (Partners) 4. EAI (Adapters) 5. Vertical B2B

BizTalk Server 2004 BizTalk Server 2004 was introduced in 2004 with several enhanced features. One enhancement included the routing mechanism, that is, implementing content-based routing. It supported Visual Studio 2003. A list of other features provided by BizTalk Server 2004 is as follows: –– WSE, SQL, SOAP, and EDI Adapter –– Delivery notifications (acknowledgments and negative acknowledgments) –– Business Activity Services (BAS) –– Human Workflow Services (HWS) –– BizTalk Explorer –– Health and Activity Tracking (HAT) –– Visual Studio .NET Human Workflow Services (HWS) Human Workflow Services (HWS) was added to BizTalk Server 2004 to provide better workflow infrastructure. These services streamlined interactions that required human input, for example, review and approve, and were commonly known as “workflows.”

BizTalk Server 

 13

As discussed earlier, HWS provided a workflow infrastructure that helped to automate workflow. To access the HWS infrastructure, web services were required. The access to HWS infrastructure was required so that any client application could use it. For instance, applications within the Microsoft Office suite were the most important clients that used HWS. These applications provided an environment (user interface) to allow human interaction.

Covast Covast is a company based in Rotterdam, Netherlands. It was founded in 1996 and was introduced to Microsoft by the Gartner Group in 2001. They agreed that the Covast EDI technology could improve the functions of BizTalk Server. The Covast EDI Accelerator for BizTalk Server 2000 was born out of these meetings between Microsoft and Covast during 2001. Covast was brought in to address the gaps in EDI technology in BizTalk Server. The first version of the integrated product was released in early 2002. Covast Solutions for BizTalk Server 2004 and Later Versions After BizTalk Server 2004 was launched, most organizations were looking for a solution to consolidate their application integration to a single platform. They were also facing some issues related to mapping conversion and EDI adapters. To overcome these issues, Covast introduced a suite of tools to provide a fast and cost-effective B2B integration solution for different integration platforms, such as BizTalk Server and IBM WebSphere. For BizTalk Server, Covast introduced the following tools: 1. Covast Map Accelerator for BizTalk Server 2. Covast EDI Accelerator for BizTalk Server 3. Covast AS2 Adapter for BizTalk Server Covast Map Accelerator for BizTalk Server Covast Map Accelerator for BizTalk Server was developed for migrating maps and document specifications. It provided a conversion/migration rate of 80%, which means that 80% of the information of a map was converted automatically while the remaining map information could be taken manually from the original map. The accelerators and adapters helped the BizTalk developers who were required to complete map migration manually. They did not require access to the legacy EDI system for the map migration because the original map information was available in the BizTalk Mapper environment.

14 

 Chapter 1: Evolution of BizTalk

The Covast Map Accelerator supported the following EDI formats for migration: –– Positional flat file –– Delimited flat file –– XML –– X12 (U.S.) –– VICS (U.S. general merchandise retail) –– WINS (U.S. warehousing retail) –– UCS (U.S. grocery retail) –– EDIFACT (worldwide) –– EANCOM (European retail) –– Tradacoms (U.K.) –– VDA (German automotive) –– Odette (automotive) –– Cargo-IMP (airfreight) –– Gencod (France) Covast EDI Accelerator BizTalk Server 2004 provided basic EDI functionality, but it was unable to handle larger-scale EDI implementations. However, it remained a good option for implementing a limited number of EDI files where EDI communication was not required. To overcome these challenges, Covast introduced the EDI Accelerator in coordination with Microsoft. In other words, Microsoft and Covast collectively developed the Covast EDI Accelerator, which allowed BizTalk developers to implement complex EDI requirements into an XML framework of BizTalk Server. It also allowed organizations to use a single platform for handling their B2B activities. Quotes “We felt the Covast EDI Accelerator would provide the most integrated, robust and modern architecture to make BizTalk a complete platform from the B2B perspective.” —Matt Mulligan, Senior Product Manager, Microsoft Corporation

The Covast EDI Accelerator can be considered better than the BizTalk Server 2004 Base EDI adapter when considered within the following parameters: –– Communication –– Batching –– Validation –– Acknowledgments –– Control numbering –– Auditing

BizTalk Server 

 15

These parameters are discussed in some detail in the following subsections. Communication Covast EDI Accelerator provided common communication setup for different EDI protocols, such as FTP, HTTP, X.400, and AS2, etc. Covast EDI Accelerator also introduced several connectors for the most common VANs, which were not available in the base EDI adapter, as it did not have an EDI specific communication support. Batching The Base EDI adapter did not possess the capability of batching multiple transactions into one database transaction. This was a significant issue with the solution. However, Covast EDI Accelerator provided a standard batching facility, which is required for each EDI transaction. Validation The Base EDI adapters were using XDR schemas from BizTalk Server 2002 for validation purposes. However, these schemas were missing several validation rules, such as complex hierarchical structures and composite checks, etc. Covast EDI accelerator added a true EDI validation engine that added all the missing rules in BizTalk 2002 XDR schemas. Acknowledgments The Base EDI adapter was capable of controlling acknowledgments on the document level only, as it did not have the outbound batching facility. Also, only segment level errors were produced in the negative acknowledgment. On the other hand, the Covast EDI Accelerator supported acknowledgments on different levels including interchange, group, and document levels. Control Numbering The Base EDI adapter was capable of supporting control numbers on different levels including document, group, and interchange level. However, it was unable to check duplicate EDI transactions. On the other hand, Covast EDI Accelerator could support all these control numbers without any limitation. Auditing The Base EDI adapter was integrated with the HAT tool to log all the messages, that is, inbound and outbound messages. However, it provided limited error details. On the other hand, Covast EDI Accelerator allowed people to view errors with explanations. It also helped users in analyzing the actual point of error in the document as it highlighted the specific point where the error occurred.

16 

 Chapter 1: Evolution of BizTalk

Covast AS2 Adapter for BizTalk Server Earlier, EDI documents were exchanged through Value Added Networks (VANs) and other bilateral connections, such as leased lines and dial-up connections. These types of connections were used for the following reasons: –– They provided guaranteed delivery of documents. –– They provided on-time delivery of documents. –– They provided the non-repudiation feature. –– They provided the authentication feature. However, one of the major disadvantages of using these types of connections was their high cost per transaction due to the expensive setup and maintenance cost of the bilateral connections. Later, standard internet protocols were implemented to exchange business documents. However, these efforts failed for several reasons, including no guaranteed delivery, easy interception of messages, and tentative authentication. Finally, the Internet Engineering Task Force (IETF) approved the Applicability Statement 2 (AS2) standard for reliably exchanging documents using the HTTP protocol. It guaranteed all features provided by the traditional VANs and was very cost-effective. Covast, in collaboration with Microsoft, provided support for the AS2 adapter in BizTalk Server. This adapter was available independently and could be installed on top of the Covast EDI Accelerator. The Covast AS2 adapter managed the following types of AS2 traffic for Microsoft BizTalk Server: –– Outbound AS2 traffic: Management involved assigning the AS2 adapter on any Send Port that was described in Visual Studio. Next steps involved signing, encrypting, compressing, and sending outbound messages to the AS2 recipient. –– Inbound AS2 traffic: Managed with the help of a Windows 2000 or Windows 2003 service that could manage multiple AS2 connections simultaneously. The service holds the following responsibilities: ○○ Verifying the digital signatures of the senders ○○ Decrypting the received messages ○○ Decompressing the received messages ○○ Submitting the received messages to the BizTalk Server Message Box database with the help of the Receive Port/Receive Pipeline for further processing The Covast AS2 adapter also handled the Message Disposition Notifications (MDNs) that had to be sent to the originator of the messages.

BizTalk Server 

 17

BizTalk Server 2006 The popularity of BizTalk Server 2004 encouraged developers to get trained. The increased number of trained personnel demanded new features in the next versions of BizTalk Server, which led to the development of BizTalk Server 2006. BizTalk Server 2006 was launched in March 2006 with better monitoring tools, such as BAM (Business Activity Monitoring) and HAT. BizTalk Server 2006 included advanced management capabilities, real-time alert functionalities, advanced deployment capabilities, and a BAM portal to help organizations analyze their business processes and optimize their investment for maximum efficiency. BizTalk Server 2006 provided the following features: –– BAMTracking Profile Editor (TPE) –– BAS –– BizTalk Orchestration Designer for business analysts –– Visual Studio 2005 and .NET 2.0 integration –– POP3 and Windows SharePoint Services adapter –– Enhanced SMTP, FILE, HTTP, SOAP, and MSMQ adapters –– Automated solution-level redeployment –– BizTalk Flat File Schema Wizard for creating schemas –– BAM SQL views and operations API support BizTalk Server 2006 R2 BizTalk Server 2006 R2 was launched in October 2007. Several adapters were added to this version. One of the prominent features of this version was native EDI processing capability and RFID (Radio-Frequency Identification). It also contained several new and enhanced features. Some of them are as follows: –– New native EDI processing capabilities which deprecated Covast –– Improved batching functionality –– Support of EDIINT AS2 –– RFID technology –– New adapters for WCF communications –– Line of Business (LOB) adapters –– BizTalk WCF Service Publishing Wizard and BizTalk WCF Service Consuming Wizard –– BAM interceptor –– Microsoft update support –– Enterprise Single Sign-0n 4.0

18 

 Chapter 1: Evolution of BizTalk

Microsoft released one Service Pack in 2010 and four cumulative update packs (CUPs) for BizTalk Server 2006 R2, the last in 2012.

BizTalk Server 2009 BizTalk Server 2009 was launched in April 2010. It was essentially BizTalk Server 2006 R3. However, it was renamed BizTalk Server 2009 due to the new features included in it. It was built on the core architecture of BizTalk Server 2006. It had a presence in all dimensions of application-to-application and B2B process automation. BizTalk Server 2009 had the following features: –– Improved failover clustering –– New Hyper-V virtualization support –– Enhanced BAM –– Enhanced EDI and AS2 support –– New application life-cycle management support –– Enhanced WCF adapter –– Support for Visual Studio 2008 and SQL Server 2008 –– Support for .NET Framework 3.5 –– Enhanced support for SWIFT –– Better connectivity with intelligent RFID devices –– WCF LOB Adapter Pack 2.0 Microsoft released seven CUPs for BizTalk Server 2009, the last in 2014.

BizTalk Server 2010 BizTalk Server 2010 was launched in October 2010. One of the most exciting features in BizTalk 2010 was its integration with Visual Studio. This release also offered support for Windows Server AppFabric, which allowed users to control Windows Workflow Foundation (WWF) and Windows Communication Foundation (WCF). Its simple installation and improved BAM tools forced BizTalk developers to implement BizTalk Server 2010. The following features were provided by BizTalk Server 2010: –– Improved Monitoring Management Pack –– Remodeled BizTalk Mapper –– Settings dashboard –– Enhanced FTP adapter –– Updated platform support

BizTalk Server 

–– –– –– –– –– –– –– –– ––

 19

Remodeled trading partner management Enhanced support for HIPAA documents Support for .NET Framework 4.0 Easy monitoring of known issues in BizTalk Server Customer Experience Improvement Program (CEIP) SQL Server Backup Compression SQL Server Transparent Data Encryption (TDE) PowerShell provider Monitor BizTalk Server SQL Agent Jobs

Microsoft released nine CUPs for BizTalk Server 2010, the last in 2016.

BizTalk Server 2013 BizTalk Server 2013 was launched in March 2013 and was an important achievement in terms of cloud integration. BizTalk Server 2013 Virtual Machines (VMs) were available on the Azure portal. It provided new adapters for integrating BizTalk Server with Azure. It also updated SharePoint adapters. In addition, the ESB Toolkit was integrated into BizTalk 2013 Server installation setup. In a nutshell, BizTalk Server 2013 offered the following features: –– BizTalk Server 2013 Azure VMs (nonproduction) –– Adapters for Azure connectivity –– Tracking dependencies between artifacts –– Support for Visual Studio 2012 –– .NET Framework 4.5.x –– Integrated ESB Toolkit –– Updated EDI Schemas –– XslCompiledTransform for map compilation –– Ordered delivery –– Configurable dynamic Send Port –– BAM alerts –– PowerShell provider –– REST (Representational State Transfer) support –– Support tools like BizTalk Assembly Checker, MsgBoxViewer Microsoft released five CUPs for BizTalk Server 2013, the last in 2016.

20 

 Chapter 1: Evolution of BizTalk

BizTalk Server 2013 R2 BizTalk Server 2013 R2 was a revised version of BizTalk Server 2013, which was launched in June 2014. This version included several new and advanced features, as follows: –– BizTalk Health Monitor plug-in –– Proxy support in SFTP (Secure File Transfer Protocol) adapter –– SAS support in the Service Bus adapter –– JSON support –– Windows Server 2012 and Windows Server 2012 R2 support –– SharePoint 2013 SP1 support –– 64-bit MLLP adapter –– Visual Studio 2013 –– .NET Framework 4.6.x –– Free Text in HL7 accelerator Microsoft released five CUs for BizTalk Server 2013 R2, the last in 2016.

BizTalk Server 2016 BizTalk Server 2016 was launched in October 2016. The focus of this version was providing integration services, such as on-premise to cloud integration, cloud to cloud integration, hybrid integration, and Azure Logic Apps integration. Now, most organizations are running their applications on the cloud. They may need to connect to on-premise resources as well as cloud resources. BizTalk Server 2016 allows users on the on-premise BizTalk Server to connect to cloud applications with the help of Azure Logic Apps. It also allows users running the on-premise BizTalk Server to connect with SAAS apps to enable enterprise cloud messaging between trading partners. BizTalk Server 2016 provides several new and advanced features. Some of these features are as follows: –– SQL Server 2016 Always-On Availability groups –– Operational data feed for Power BI with BizTalk Server –– Management REST APIs –– Application Insights for BizTalk artifacts –– BizTalk Azure VMs in production –– Azure Logic Apps adapter –– Ordered delivery on dynamic Send Ports –– Connect to Azure file share from BizTalk File adapter

BizTalk Server 

–– –– –– –– –– –– –– –– ––

 21

Selection of multiple messages in the BizTalk Server Administration Console Option to choose among XslTransform and XslCompiledTransform Refined B2B import/export process Connect to SQL Server Always Encrypted columns through BizTalk Server Improved SFTP, MLLP, and the Service Bus adapters SAP NCO (.NET Connector 3.0) support Full SHA-2 (Secure Hash Algorithm) support and more SHA-2 hash functions Shared Access Signature (SAS) authentication for Azure Service Bus HA/DR (High Availability/Disaster Recovery) in Azure

Note All of these features will be discussed in detail in the succeeding chapters.

Microsoft released three CUs for BizTalk Server 2016, the last in 2017.

BizTalk Server 2016 Feature Pack 1 The primary focus of BizTalk Server 2016 launch was integration services, especially hybrid integration. To add more functionality to BizTalk Server 2016, Microsoft introduced BizTalk Server 2016 Feature Pack 1 in September 2017, with version number 3.13.177.2. This release provided enhanced features in deployment, analytics, and runtime, as shown in Table 1.1. Table 1.1: Features of BizTalk Server 2016 Feature Pack 1 Feature Category

Feature Description

Deployment

Allows users to automatically deploy and update applications using Visual Studio Team Services (VSTS). Provides new Management REST APIs with the complete Swagger support, which allows users to manage BizTalk environment remotely.

Analytics

Provides the Application Insights feature, which allows users to track valuable application performance data and helps to understand workflow. Allows users to utilize Power BI for viewing the operational data from anywhere and through any device.

Runtime

Provides enhanced support for the Always Encrypted columns, which allows them to connect to SQL Server securely. Provides the Advanced Scheduling feature.

22 

 Chapter 1: Evolution of BizTalk

BizTalk Server 2016 Feature Pack 2 Microsoft has released another feature pack, Microsoft BizTalk Server 2016 Feature Pack 2 in November 2017. It provides several advanced features related to deployment, administration, runtime, and analytics and reporting. These features are outlined in Table 1.2. Note Azure Event Hubs is a hyper-scale telemetry ingestion service. It is responsible for gathering, transforming, and saving events.

Efforts are ongoing to release future versions of BizTalk Server, especially as Azure Services. Table 1.2: Features of BizTalk Server 2016 Feature Pack 2 Feature Category

Feature Description

Deployment

Allows users to define the multiserver deployment of BizTalk Server 2016 and manage deployed systems throughout the application lifecycle with the help of VSTS.

Administration

Allows administrators to back up the BizTalk Server database to Azure Blob storage at the time of deploying BizTalk Server to Azure VMs.

Runtime

Provides the Azure Service Bus Premium adapter, which can be used for enterprise-scale workloads while using the Service Bus adapter. Provides the Transport Level Security (TLS) 1.2 authentication and encryption feature, which allows you to deploy BizTalk Server securely. Provides BizTalk Orchestration Engine endpoints that can be published through Azure API Management. It also allows organizations to publish APIs to external and internal developers, unlocking the potential of their data and services. Provides a new Azure Event Hubs adapter, which is used by BizTalk Server to send and receive messages to/from Azure Event Hubs. BizTalk Server can work as an event publisher and a subscriber (a part of a new Azure cloud-based event-driven application).

Analytics and Reporting Allows users to transmit BizTalk Server tracking data to Azure Event Hubs. Provides the Azure Sign-in dialog, which is used at the time of preparing BizTalk Server to send tracking data to Application Insights, thus simplifying the configuration of named instances of SQL Server.

Windows Azure BizTalk Services (WABS) 

 23

Windows Azure BizTalk Services (WABS) Microsoft introduced Windows Azure BizTalk Services (WABS) in November 2013. WABS provides powerful business scenarios, such as supply chain and cloudbased EDI, and Enterprise Application Integration (EAI). It provides built-in support for managing EDI relationships between trading partners and setting up EAI bridges with on-premise systems, such as SAP, SQL Server, Oracle, and Siebel systems. It also provides built-in support for integrating with on-premise systems and provides options for integrating WABS with on-premise BizTalk Server deployments, thereby enabling powerful hybrid enterprise solutions. Note WABS has been deprecated since May 31, 2017. Support for existing subscriptions ended May 31, 2018. All existing subscriptions have been transitioned to Azure Logic Apps and Azure App Service Hybrid Connections.

Azure BizTalk Services As discussed, WABS has been replaced with Azure Logic Apps. Microsoft is providing several options, such as cloud or Azure services in the form of BizTalk services. Azure BizTalk Services can be categorized in the following two forms: –– BizTalk Infrastructure as a Service (IaaS) –– BizTalk Platform as a Service (PaaS) In the form of IaaS services, Microsoft provides a complete VM with BizTalk Server functioning on it. BizTalk PaaS services allow you to: –– Set up independent integration solutions using Azure Logic Apps –– Connect to on-premise resources from Azure –– Create and store artifacts on Azure Although BizTalk PaaS services are new, they do not possess all capabilities of BizTalk Server as a product—it does include many core functionalities, and product investments continue to bring it to parity. Note We will learn about IaaS and PaaS services in detail in succeeding chapters.

24 

 Chapter 1: Evolution of BizTalk

Summary From COM to PaaS, BizTalk has evolved and is fulfilling its mandate of integrating in-house applications with external applications. While investments in the Server edition of BizTalk may have slowed down, they are rapidly growing in Azure, which is in accordance with the current cloud adoption trends. Rest assured, BizTalk Server will continue to live, thrive, and adapt in keeping with evolving technology trends.

Chapter 2 BizTalk Integration Patterns Microsoft’s BizTalk Server, more than just an integration solution, has capabilities to provide an end-to-end business process integration and management platform. This platform helps businesses develop different kinds of solutions, including Business Process Management (BPM), Enterprise Application Integration (EAI), Enterprise Service Bus (ESB), Service Oriented Architecture (SOA), and High Availability and Scalability (HAS). Most commonly, BizTalk Server is designed to integrate different types of applications running in an enterprise. It provides well documented methods to integrate almost all kinds of small or complex IT systems. In this section, we will discuss the following integration patterns: –– Enterprise Application Integration –– Workflow or orchestration for process management –– Business to business (B2B)

Enterprise Application Integration (EAI) Enterprise Application Integration (EAI) is a set of principles that integrates applications into automated business processes within an enterprise. In other words, EAI allows communication between data, applications, and business processes. There are several advantages of using EAI. Some of them are as follows: 1. It helps in reducing the initial financial cost, time, and maintenance cost for storing a large amount of data with the use of the valuable EAI toolset. 2. It helps in reducing administrative costs, as many of the business processes have become automated. 3. It helps in reducing operational costs through the use of effective value-chain processes. 4. It provides better customer satisfaction and thus establishes customer loyalty with the help of additional services provided by Microsoft. 5. It helps in developing decision-making skills as business information can be collated and available in real time. There are several products provided by Microsoft for implementing EAI solutions where each product provides solutions to a unique part of the EAI landscape. For a complete EAI solution, we need to integrate all those products. Microsoft BizTalk Server is one of the valuable products introduced by Microsoft that provides a wide DOI 10.1515/9781501505652-002

26 

 Chapter 2: BizTalk Integration Patterns

range of services in the EAI platform. The following products or services can be used with Microsoft BizTalk Server to fulfill additional service requirements: –– Microsoft Host Integration Server –– Microsoft SQL Server –– Windows core services: ○○ Microsoft XML Web services ○○ Microsoft Data Access Components (MDAC) data connectivity services ○○ Component Object Model (COM+) application services Definitions —— Microsoft XML Web Service: Refers to a service provided by the operating systems that were introduced after Windows 2000. This service enables Windows components to create and receive XML messages. —— MDAC Service: Refers to a service that provides a high-level programming interface, named ActiveX Data Object (ADO). —— ADO: Refers to a programming interface that works in both types of architectures whether client/server or web-centric and helps in creating front-end database clients as well as business objects. —— COM+ Service: Refers to a service that provides security limitations for the business logic components. These components help in the manipulation of data because this service flows across data resources of an organization. —— Microsoft Host Integration Server: Refers to an integration server that enables integration between the host environment and the Windows platform. It includes services, adapters, and transactional gateways.

Applications within an enterprise may or may not be running on different platforms. However, BizTalk Server resolves such concerns with an easy-to-use messaging solution. Two common areas of EAI are as follows: –– Process automation –– Data transfer For automating the business process, BizTalk Server provides messaging support for data transfer and workflow (also known as orchestration). To provide these functionalities, BizTalk Server utilizes several well-known and commonly used protocols, including the following: –– File Transfer Protocol (FTP) –– Secure FTP (SFTP) –– Microsoft Message Queuing (MSMQ) –– Post Office Protocol (POP) –– Simple Object Access Protocol (SOAP)

Enterprise Application Integration (EAI) 

 27

The data transfer process takes some logical steps, as shown in Figure 2.1. Extracting raw data from the database Authenticating the extracted data against a business model Converting business model data into business data Authenticating the business data against the business model Inserting the business data into the database Figure 2.1: Steps involved in the data transfer process

Once BizTalk Server is configured properly, all processes run automatically. For example, as a sales representative, you may need to periodically check for sent orders, received orders, and completed orders. You may also need to make a call to top level management, stakeholders, or trading partners to provide the status of orders. You may also require that a confirmation email is sent upon order completion. Each of these processes is time consuming and requires user interaction. With the help of BizTalk Server, these processes can be completed automatically in the specified time frame. BizTalk Messaging BizTalk Server transmits or receives data in the form of messages in different formats including: –– Electronic Data Interchange (EDI) –– eXtensible Markup Language (XML) –– Electronic Data Interchange for Administration, Commerce and Transport (EDIFACT) –– Flat file BizTalk Server processes the document containing data after getting instructions from BizTalk Messaging Engine, which is responsible for controlling and managing document specifications and processing instructions. The management is done with the help of protocols such as FTP and SFTP, and shared locations. In actuality, BizTalk messaging deals with the act of transmitting data through different communication modes. As stated earlier, BizTalk Server supports several

28 

 Chapter 2: BizTalk Integration Patterns

file formats of messaging. However, the extensibility feature of BizTalk Server can be configured to use a different format that is not present in the default list of BizTalk Server. You can create custom pipelines or adapters to create those files and transact them through different systems as applicable. BizTalk Server helps users to communicate with different services like WCF, HTTP, Web Services, etc. There are a few terms associated with BizTalk messaging that are important to understand clearly: –– Message: A data file –– Multipart message: A message with multiple parts where each part contains some data –– Messaging: The act of transmitting a message from a Receive Port to a Send Port with the help of adapters –– Translation: The act of altering the message format –– Transformation: The act of developing an agreement between the source and the planned target –– Routing: The act of transmitting messages with the help of subscriptions or filtering to subscribers –– Content based routing: The act of transmitting messages on the basis of the configuration properties of the Receive Port BizTalk Message Each BizTalk message is considered a multipart message that contains two parts: –– Message Data/Body Part: The part of a message that contains its business data or the data received by BizTalk Server for further processing. A BizTalk message cannot be changed once it is ready. This means that once the message is recorded in the BizTalk MessageBox database, the message construction process stops. –– Message Context/Message properties: The part of a message that acts as a container for the properties used by BizTalk Server for document processing. Each context (or property) is associated with three variables: name, namespace, and value. Message properties can have one of two values: written or  promoted. The written properties cannot be used as criteria in message routing, while promoted properties can be used. The MessageBox Database BizTalk Server provides reliable and robust messaging services with the help of the MessageBox database that tracks each message through a unique ID. It is referred to as the heart of the publish/subscribe engine in BizTalk Server. It contains the following two components:

Orchestration 

 29

–– Microsoft SQL Server database: Acts as a maintenance warehouse for messages, its parts or properties, data tracking, subscriptions, etc. It also provides information about transmitting messages and satisfying subscriptions. –– Messaging Agent: Acts as an interface between BizTalk Server and the MessageBox database. It is a Component Object Model (COM) component that helps BizTalk Server in publishing, subscribing, or retrieving messages. BizTalk Server receives messages through an adapter and passes them through a pipeline. When a message passes through a pipeline, it is serialized and written (or published) into the MessageBox database. The intended recipient of the published message is decided by BizTalk Server on the basis of the message context properties. Finally, BizTalk Server routes the message to the intended recipient component, be it orchestration or Send Port, on the basis of the subscriptions or filters. BizTalk Server also enables you to hold or halt a sequence of events to be performed in the workflow (orchestration). All the data related to the workflow is stored in the MessageBox database when orchestration instances are delayed or suspended (to be resumed or terminated later). The data stored in the database contains the messages on which orchestration is working, their current state, and context properties.

Orchestration Orchestration is a workflow that defines a sequence of events to be performed or executed on the data received through the Receive Pipeline. It can be defined as a tool that takes advantage of the XLANG/s language for expressing an executable business process. Note XLANG/s is a messaging language. Some of its expressions are the same as available in C#. It supports several standards including XML, XSD, and Web Services Description Language (WSDL). It is used with .NET objects and messages.

The major elements of an orchestration are as follows: –– The message –– The operating actions on the message including send and receive –– The transmission ports

30 

 Chapter 2: BizTalk Integration Patterns

An orchestration is compiled, organized and sent to a BizTalk host. The orchestration can be treated as a class for those instances that are deployed. When the MessageBox database is hit by a message, a check is made for the related subscribers or orchestrations. On the basis of the subscription criteria specified in the orchestration, the subscribing orchestration is invoked. The orchestration in BizTalk Server is shown in Figure 2.2.

Figure 2.2: Orchestration in BizTalk Server workflow

Before getting detailed information about an orchestration, you must be familiar with the following tools and terms: –– BizTalk Orchestration Designer: A graphical tool used for representing the business processes visually. The major functionality of this tool is creating XLANG/s files with the .odx extension. These files are executable files and contain different types of information in different parts of the message. This means that visualization information is stored in the header, while the body

Orchestration 

 31

contains attribute information. It contains a large number of orchestration shapes where each shape is associated with a unique action. –– BizTalk Orchestration Engine: A developer tool that is responsible for executing XLANG/s files generated by BizTalk Orchestration Designer. It handles the following tasks: ○○ Defining the sequence of steps ○○ The routing of messages ○○ The act of subscribing messages by ports ○○ The act of preserving the current state of the message ○○ The act of utilizing the message contexts for executing events

Exploring Orchestration Design Surface The Orchestration Design Surface is the main visualization area that is located in the center of the BizTalk Orchestration Designer and is used for creating orchestrations. You can drag shapes here and configure them. The Orchestration Design Surface is shown in Figure 2.3.

Figure 2.3: Orchestration Design Surface

32 

 Chapter 2: BizTalk Integration Patterns

As shown in Figure 2.3, the Orchestration Design Surface is categorized into three parts: the process area, and two port surfaces: –– Process Area: The central area containing shapes where the orchestration’s actual process flow is depicted. The send and receive shapes are present in the process area that need interaction with the intended ports. Role link shapes are located in the port surfaces part of the Orchestration Design Surface. –– Port Surfaces: Each port surface resides in either side of the process area and works identically. As there are two port surfaces, it becomes easy to create orchestrations on new ports. You can either click on the double arrow icon, or double-click on Port Surface to collapse or expand it.

Business to Business (EDI, HIPAA, and XML) Electronic Data Interchange (EDI) Electronic Data Interchange (EDI) is a mode of transmitting and receiving data in a standardized electronic format. The business entities making transactions in EDI mode are called trading partners. They must follow an agreement containing rules for exchanging information. In EDI, transactions are batched sets, which means a transaction would transmit multiple messages concurrently. The standards supported by EDI are as follows: –– X12, which is standardized by American National Standards Institute (ANSI) –– EDIFACT, which is standardized by the United Nations EDI is especially used for exchanging purchase orders and health-related documents. EDI has reduced the time required for processing purchase orders. The processing of purchase orders includes order processing by a supplier, manual communication with the trading partners, creating purchase orders, printing them, and finally sending them to the supplier. In traditional approaches, the entire process took three to four days to complete. However, with the help of EDI, the entire process can be done in a few hours. Benefits of Using EDI to Exchange Documents There are several benefits of using EDI to exchange documents. Some of them are listed as follows: –– The documents can be exchanged swiftly –– Data entry can be made automatically –– Human interaction is reduced, which leads to minimizing the labor costs

Business to Business (EDI, HIPAA, and XML) 

–– –– –– –– ––

 33

Transaction costs are reduced to a large extent Data quality is increased with the reduction in transaction errors The relationship between trading partners is enhanced Receipts can be verified easily Administrative and maintenance costs are reduced to a large extent

Health Insurance Portability and Accountability Act (HIPAA) One of the most prominent aspects of BizTalk Server is to assist and integrate the HIPAA (Health Insurance Portability and Accountability Act). HIPAA legislation was enacted in 1996 in the United States. It sets standards to protect sensitive data related to patients. As per the HIPAA guidelines, the business entities dealing in exchange of health-related documents must ensure that all the required security measures including physical, network, and process are appropriate. Note The business entities under HIPAA include clearing houses and health care providers.

A few major objectives of HIPAA are listed as follows: –– Allowing workers to receive the benefits of continuous health insurance coverage irrespective of their current job status –– Enabling administrative and financial transactions to be processed electronically, diminishing the administrative burden and cost of healthcare –– Preventing fraud and waste in health insurance Keeping in mind the vulnerability of sensitive data, most of the data is exchanged in the form of EDI. In actuality, health-related data requires the exchange of different types of documents including claim enrollment, approval, authorization, response, and acknowledgment. BizTalk Server provides these types of EDI schemas built-in, which helps organizations create automated business processes. BizTalk Server also helps to create partner-specific settings and agreements to connect with different trading partners. The business entities under HIPAA can use BizTalk Server to develop and integrate solutions related to the transaction. Note HIPAA versions 5010 and 4010 are supported by BizTalk Server.

34 

 Chapter 2: BizTalk Integration Patterns

XML XML Schema Definition (XSD) language helps to define the message structure for the data processed by Microsoft BizTalk Server. The definition of message structure is termed as schema. Different formats, including XML and flat files, are used by the systems that create the structured messages. BizTalk Server works with XML natively, which means that all manipulation is done in XML because it is used as an official representation for business documents by BizTalk Server. It validates EDI and partner-specific properties, including the following: –– EDI schema validation –– Cross-field validation for X12-encoded messages (if configured) –– EDI structural validation –– Extended schema validation (if a node having a non-EDI data type is used to modify a schema) Based on the type of EDI document (834, 820, 837, 999, etc.) or partner-specific property (extracted from interchange), BizTalk Server decides the type of schema required to validate the document. Once the schema is selected, it validates the message data and generates XML from the incoming EDI message data.

B2B and EAI with BizTalk Most of the transactions related to health documents are done in EDI format. Consider the following scenario: a medical service provider and an insured person who gets his/her treatment from the medical service provider. The following is a list of steps (transactions) that would take place: 1. The insured person asks for the claim. 2. The medical service provider prepares the bill and sends it to the clearing house for further processing. 3. The clearing house transmits the bill to the payer/insurance provider after converting it into EDI format. 4. The payer/insurance provider receives the bill and takes it through a validation process called authorization. 5. Once the authorization is done, there could be different authorization results including the following: –– Accepted: Bills are accepted when all the procedures are available in the insurance terms. –– Rejected: Bills are rejected due to incorrect or insufficient data of the insured person. –– Denied: Bills are denied if the requested procedure was not covered in the insurance terms.

B2B and EAI with BizTalk 

 35

–– Partially accepted: Bills are partially accepted if only some of the requested procedures are part of the insurance terms. 6. The authorization result is sent back to the respective clearing house, which is then sent back to the biller. 7. Once the authorization is accepted, the claim is given by the insurance provider. Note The authorization result contains all details including the authorization result and amount covered in the insurance, in case the authorization result is either denied or rejected.

It cannot be ignored that there are different insurance providers who are associated with different clearing houses. As a result, and to avoid ambiguity, there must be unique identifiers in place for documents sent for authorization, and thus received as a response. Clearing houses also require a data warehouse (either of their own or of a third party) for storing all the data. In terms of BizTalk EAI, there could be four to five enterprises for the above scenario, as follows: 1. Medical houses/Biller 2. Clearing houses 3. Payer/Insurance provider 4. Authorization entity or a third-party application (AuthNet) 5. Data warehouse (DWH) These enterprises are connected, as shown in Figure 2.4.

Figure 2.4: Depicting connections between different enterprises in terms of EAI

36 

 Chapter 2: BizTalk Integration Patterns

In the Figure 2.4 example, in EDI terms, we have given a unique identity to a claim doc named 278. The authorization request is named 278-auth, while the authorization response is named 278-AAA. The entire process would contain the following steps: 1. Medical House sends 278 to Clearing House. 2. Clearing House responds back with 999. 3. Based on the type of document and information inside 278, Clearing House saves the initial request data and transforms it in 278-AAA format. Note The transformation process may or may not include a crosswalk of data related to different medical houses and insurance providers, etc.

4. Clearing House sends 278-Auth to Insurance Provider. 5. Insurance Provider sends the received doc (278-Auth) to AuthNet for authorization. 6. AuthNet responds with an acknowledgment id 999. 7. AuthNet authorizes the received doc (278-Auth) and gives the authorization response in the form of 278-AAA to Insurance Provider. It also sends the authorization response in detail in the form of XML to Insurance Provider. 8. Insurance Provider sends the authorization response (278-AAA) that it received from AuthNet to Clearing House. 9. Clearing House processes 278-AAA and transforms it into the respective Medical House format (EDI or non-EDI), which means that Clearing House transforms 278-AAA into the format in which it received the authorization from Medical House. 10. Clearing House generates an additional doc (in XML) and sends it to Data Warehouse for storing the authorization response data. As shown in the above process, there are a large number of steps, and a separate job is required for each step including receiving the file, saving data, transforming the received doc into required 278-Auth format, and sending it to the insurance provider. It makes the entire process cumbersome and thus difficult to monitor. Also, it becomes necessary to investigate each job separately to identify the root cause of an error, if any. The implementation of BizTalk Server solves all these issues. It places all jobs into a single process or workflow, called an orchestration. Therefore, it becomes easy to identify the root cause if an error occurred there. It also becomes easy to debug that error.

B2B and EAI with BizTalk 

 37

In the scenario discussed earlier, BizTalk Server has to communicate or integrate only with three different applications, including the Medical House to receive and send data, one application at the Insurance Provider, and one application to store data in the Data Warehouse.

Implementing the Process through BizTalk As discussed earlier, BizTalk Server helps in automating all processes mentioned in the earlier scenario and provides communication between applications running in different enterprises. The implementation includes the following major steps: Step 1: Receive the authorization doc from the Medical House and send it to the Insurance Provider for authorization To receive the authorization doc from the Medical House and respond back with an acknowledgment 999, we need the following: 1. Create a Receive Port (Receive278FromMH) with the EDIReceive pipeline to receive the claim doc (278) from the Medical House, as shown in Figure 2.5.

Figure 2.5: Creating a Receive Port

38 

 Chapter 2: BizTalk Integration Patterns

2. Configure the trading partner settings to automatically generate 999 (ACK), as shown in Figure 2.6.

Figure 2.6: Configuring the trading partner settings to generate an acknowledgment

3. Create a Send Port for sending ACK (999) to the Medical House and subscribe it to the MH 278 Receive Port, as shown in Figure 2.7. Once the above configuration is done, we can ensure that the Medical House receives an acknowledgement 999 automatically when BizTalk Server receives a claim doc (278) from the Medical House. It is an automatic process in BizTalk Server. We do not need to create and transform 999 (ACK) manually. Similarly, we need to create a Send Port with URI pointing to the Medical House location, where BizTalk Server can send the acknowledgment.

B2B and EAI with BizTalk 

 39

Figure 2.7: Creating a Send Port and subscribe it to the MH 278 Receive Port

Step 2: The process receives the 278 doc, saves the required data, transforms the 278 doc into the desired format and forwards the transformed doc to the Insurance Provider. In this step, orchestration receives the 278 doc from the Medical House, saves the required data into the database, transforms it into the desired format, and finally sends the transformed 278-Auth doc to the AuthNet application used by the Insurance Provider for authentication purposes. For this, we need to perform the following tasks: 1. Create the .ODX file and deploy it, as shown in Figure 2.8.

40 

 Chapter 2: BizTalk Integration Patterns

Figure 2.8: Creating the .ODX file and deploying it

2. Use a map to transform the 278 doc to 278-Auth, as shown in Figure 2.9.

B2B and EAI with BizTalk 

 41

Figure 2.9: Using a map to transform the 278 doc to 278-Auth

Did You Know? While transforming the 278 doc, the map requires communication with the database server for crosswalk data, while ODX requires it for saving the auth-data. For communicating with the database server, BizTalk Server requires a connection string and some .NET assemblies for database operations. BizTalk Server does not require more changes in the code of the database server if the map and ODX are deployed or migrated on some other environment. This can be done easily by changing the name of the database server in the BTS configuration file in the migrated environment. This configuration file contains configuration information, as shown in the following figure.

BTS configuration file

42 

 Chapter 2: BizTalk Integration Patterns

3. Configure the .ODX file to work on the data received from the Medical House that can be done by performing the following tasks: a. Create a Send Port (Send278AuthToAuthNet) for sending the processed 278-Auth to the AuthNet application run by the Insurance Provider, as shown in Figure 2.10.

Figure 2.10: Creating the Send Port for sending the processed 278-Auth to the AuthNet application

Note As orchestration executes all the tasks in a sequence, its primary task is to receive data from the Receive Port configured for receiving 278 docs from the Medical House. It requires configuring the ODX binding after deployment with the help of BizTalk Admin Console.

b. Configure the ODX binding to receive data from the Medical House and send it to the AuthNet application after processing, as shown in Figure 2.11.

B2B and EAI with BizTalk 

 43

Figure 2.11: Configuring ODX binding

Step 3: Receive the Acknowledgment (999), Auth-response (AAA), and AuthXML files from the Insurance Provider (AuthNet) In this step, we receive an acknowledgment 999 from AuthNet and archive it to the local folder. After that, we receive AAA from AuthNet and send it to the Medical House. To receive an acknowledgment 999 from the AuthNet application and archive it to the local folder, the following tasks need to be performed: 1. Create a Receive Port for receiving 999 from the AuthNet application, as shown in Figure 2.12.

44 

 Chapter 2: BizTalk Integration Patterns

Figure 2.12: Creating a Receive Port for receiving 999 from the AuthNet application

2. Create a Send Port (Archive999FromAuthNet) to archive 999 received through the Receive Port, as shown in Figure 2.13.

B2B and EAI with BizTalk 

 45

Figure 2.13: Creating a Send Port

Now, the process of receiving AAA from the AuthNet application and sending it to the Medical House requires the following tasks: 1. Create a Receive Port to receive AAA from the AuthNet application, as shown in Figure 2.14.

46 

 Chapter 2: BizTalk Integration Patterns

Figure 2.14: Creating a Receive Port

2. Transform the received data to crosswalk and apply the same format in which the Medical House wants to receive it, as shown in Figure 2.15.

B2B and EAI with BizTalk 

 47

Figure 2.15:  Transforming the received data to crosswalk

Note You do not need to write any extra piece of code as BizTalk Server can transform the document in the pipeline itself.

3. Send the transformed AAA to Medical House by completing the following tasks: a. Create a Send Port pointing to the medical provider network share, as shown in Figure 2.16.

48 

 Chapter 2: BizTalk Integration Patterns

Figure 2.16: Creating a Send Port

b. Define a filter to the Send Port and subscribe it to the message received through the AAA Receive Port, as shown in Figure 2.17.

Summary 

 49

Figure 2.17: Defining filters

Summary BizTalk Server allows communication between different applications running in different enterprises. With its messaging and workflow system, it can define a sequence of events to be performed and can preserve orchestration states. BizTalk’s ability to setup separate configurations for each trading partner enables enterprises to handle multiple partners precisely and with ease. BizTalk Server also transforms inbound and outbound messages according to the requirements of different enterprises. Finally, the ability of BizTalk Server to communicate with third party systems including CRM and SAP, etc. makes it a perfect choice for enterprises to deploy it. With such depth in functionality, BizTalk Server has become a widely accepted solution by many Fortune 1000 businesses.

Chapter 3 BizTalk Server 2016 Features BizTalk Server 2016 is the latest and grandest version of BizTalk Server released by Microsoft in December 2016. This latest version supports most of the new platforms introduced by Microsoft including SQL Server 2016, Windows Server 2016, Visual Studio 2015, and Office 2016. It also includes a host of new features, and more importantly enhances every popular feature including map compilation; ordered delivery on dynamic Send Ports; simultaneous configuration of multiple hosts/host instances; enabling search of artifacts; ability to save multiple suspended messages simultaneously; and our personal favorite, resizable schema window. In BizTalk 2016, you can also use the Always Encrypted feature of SQL Server to protect confidential data stored in Azure SQL database or SQL Server databases. Very exciting indeed! In this chapter, you will learn about the new and enhanced features of BizTalk Server 2016.

New and Enhanced Features of BizTalk Server 2016 Several new and enhanced features have been introduced by Microsoft in BizTalk Server 2016. These features have added a greater depth of functionality to BizTalk Server. They can be classified based on the integration pattern to which they belong. Table 3.1 lists the categorization of features. Table 3.1: Categorization of Features Category

New/Enhanced Feature

Enterprise Application Integration (EAI)

Support for SAP .NET Connector (NCo) Artifacts analytics tracking Azure Logic Apps adapter Connect to SQL Server Always Encrypted column from BizTalk Server

Business Process Control

Map compilation Ordered delivery on dynamic Send Ports Advanced location scheduling

Business to Business

Connecting to Azure file share from BizTalk file adapter Refined B2B import/export process Improved FTP and SFTP adapters

DOI 10.1515/9781501505652-003

52 

 Chapter 3: BizTalk Server 2016 Features

Table 3.1  (continued) Category

New/Enhanced Feature

Miscellaneous/User Interface Improvements

Simultaneous configuration of multiple hosts/host instances Searching artifacts Save multiple suspended messages simultaneously Connect to management REST Application Programming Interfaces Operational data feed for Power Business Intelligence Resizable schema window BizTalk Server 2016 Azure Virtual Machines in production

Exploring the Features Related to EAI Integration In this section, you will learn about the following features related to EAI integration: –– Artifacts analytics tracking –– Azure Logic Apps adapter –– Support for SAP .NET Connector (NCo) –– Connect to SQL Server Always Encrypted column from BizTalk Server

Artifacts Analytics Tracking Almost each version of BizTalk Server has a tracking mechanism, that is, BizTalkDTADb, which keeps track of the progress of each artifact. However, it is cumbersome to fetch information about one particular instance from the database where the tracking information is stored. To make things simple, Microsoft has announced a cloud level analytics feature called Application Insights. As the Application Insights feature is a cloud feature, the information stored in it is available anytime. BizTalk Server 2016 is integrated with Application Insights, which means that the tracking data can be sent to Application Insights. The tracking data can also be used as a feed for Power BI, which is one of the excellent business analytics services hosted on Azure. One of the interesting features

Exploring the Features Related to EAI Integration 

 53

of analytics is that it can be enabled on a single artifact as well as on a BizTalk group. This feature requires the following prerequisites: 1. BizTalk 2016 feature pack 1 installation, which can be downloaded from the following link: https://www.microsoft.com/en-us/download/details.aspx?id=55100 2. An Application Insights instance on the Azure portal. 3. Internet connectivity on the BizTalk Server host machine to send tracking data to Application Insights. Note In case of nonavailability of Internet connection, tracking data will be stored in the BizTalk local tracking database that is BizTalkDTADb. Once the connection resumes, the tracking data is sent to the Application Insights instance.

Transmit Tracking Data to Azure Application Insights BizTalk Server 2016 allows you to transmit tracking data to Application Insights. The illustrations received from ports (Send and Receive) and orchestrations can be tracked with the help of Application Insights. To use this feature, you need to create a new instance of Application Insights. After creating an instance of Application Insights, you need to perform the following tasks: –– Enable analytics in BizTalk Server –– View analytics on Artifacts Perform the following steps to create a new instance of Application Insights: 1. Log into the Azure portal by visiting the link, https://portal.azure.com/. The Microsoft Azure dashboard opens. 2. Click the New button in the left pane of the dashboard to create a new instance of Application Insights, as shown in Figure 3.1.

54 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.1: Clicking the New button

3. Type the keywords “application insights” in the Search text box. A list of related results appears. 4. Click the Application Insights option in the search result. A preview of the Application Insights window appears in the right pane. 5. Click the Create button, as shown in Figure 3.2.

Figure 3.2: Creating an Application Insights instance

A new Application Insights window appears. 6. Enter the required details in the respective fields.

Exploring the Features Related to EAI Integration 

 55

7. Click the Create button, as shown in Figure 3.3.

Figure 3.3: Entering details in the Application Insights window

An Application Insights instance is created, and an Instrumentation Key is generated, as shown in Figure 3.4.

Figure 3.4: Displaying the Application Insights instance

56 

 Chapter 3: BizTalk Server 2016 Features

Enabling Analytics in BizTalk Perform the following steps to enable group-level analytics in BizTalk: 1. Right-click the BizTalk Group option. A context menu appears. 2. Select the Settings option from the context menu. The BizTalk Settings Dashboard window appears. 3. Select the Enable group-level analytics checkbox under the Analytics section. 4. Select the Application Insight option from the Target type drop-down list. 5. Enter the Application Insights instrumentation key under the Connection parameters text box. 6. Click the OK button to apply the changes, as shown in Figure 3.5.

Figure 3.5: Enabling group-level analytics

After enabling analytics for the BizTalk group, you can enable analytics for a particular artifact, whether orchestration or port. To enable analytics for an artifact, you need to perform the following steps: 1. Right-click the desired artifact. A context menu appears.

Exploring the Features Related to EAI Integration 

 57

2. Select the Properties option from the context menu. The Properties dialog box appears differently depending on the type of artifact selected. 3. Select the Enable Analytics checkbox under the Analytics section. 4. Click the OK button to apply changes, as shown in Figure 3.6.

Figure 3.6: Enabling analytics for an artifact

Viewing Analytics on Artifacts The use of the analytics feature of BizTalk Server 2016 can be better understood with the help of an illustration. For example, a TestODX.odx instance is created, which receives a message and gets suspended. The instance is deployed and its binding is configured, as shown in Figure 3.7.

58 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.7: Deploying an instance and configuring its binding

The analytics feature is enabled for the selected ODX, as shown in Figure 3.8.

Figure 3.8: Enabling analytics for the selected ODX

At this point, a test file is created, which is dropped in the receive location about thirteen times. It means that there would be the same number of suspended

Exploring the Features Related to EAI Integration 

 59

instances in the BizTalk console. Figure 3.9 shows some of the suspended instances.

Figure 3.9: Displaying suspended instances

Now, you are required to visit the Application Insights instance on the Azure portal and select the Analytics tab, as shown in Figure 3.10.

Figure 3.10: Selecting the Analytics tab

A new page opens. You need to locate the Visualizations tab and click the Run button under the Render bar chart option, as shown in Figure 3.11.

60 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.11: Clicking the Run button

As you click the Run button, a new page opens that shows the deployed ODX and ports data in a bar chart. You need to hover the mouse over the artifact’s bar and it will display the artifact’s name and counts, as shown in Figure 3.12.

Figure 3.12: Displaying the artifact’s name and counts

From this scenario, it can be concluded that analytics in BizTalk Server 2016 aids in visualizing the reports of artifacts by configuring ports and artifacts and visiting the Application Insights instance on the Azure portal.

Exploring the Features Related to EAI Integration 

 61

Azure Logic Apps Adapter Azure Logic Apps adapter is used by BizTalk Server to communicate with the Azure Logic Apps. Azure Logic App helps in implementing workflows in the cloud. It starts with a trigger and executes a sequence of events. Definition A workflow is a visual designer tool that automates the process as a sequence of steps.

BizTalk Server 2016 has its own connector, named BizTalk Connector, which is used to connect to a receive location created on BizTalk Server. This connector helps BizTalk Server in receiving messages from Azure Logic App. Before implementing Azure Logic Apps, you should consider the following cases: –– Case 1: BizTalk Server is on-premises: In this case, you first need to install and configure the on-premises data gateway on BizTalk Server. You also need to create the data gateway resource on the Azure portal to connect it to your BizTalk Server. –– Case 2: BizTalk Server is installed on an Azure Virtual Machine (VM) and the VM is not displayed as an HTTP endpoint: In this case, you need to install and configure the on-premises data gateway for Azure Logic Apps. You also need to create a data gateway resource on the Azure portal to connect to BizTalk Server. –– Case 3: BizTalk Server is installed on an Azure VM and the VM is displayed as an HTTP endpoint: In this case, there is no need for any data gateway. Note In our case, BizTalk Server is on-premises.

Installing and Configuring an On-premises Data Gateway Perform the following steps to install and configure a data gateway on a machine hosting BizTalk Server: 1. Clicking this link http://go.microsoft.com/fwlink/?LinkID=820931&clcid=0x409 downloads the gateway installer. The downloading process has started. 2. Run the downloaded file when the download is complete. The Terms of use and privacy statement dialog box appears.

62 

 Chapter 3: BizTalk Server 2016 Features

3. Review the terms and click the Accept button to accept the terms of use and privacy statement. 4. Specify the destination folder where you want to install the package. 5. Click the OK button to proceed. Once the installation is done, a Sign in page appears wherein you are required to provide your Azure credentials. Note The Azure credentials should have a subscription on the Azure portal.

6. Enter the email address that has a subscription on the Azure portal. 7. Click the Sign in button, as shown in Figure 3.13.

Figure 3.13: Configuring an on-premises data gateway

Note For this scenario, we have created a user with the name BTS2016TestUser in the active directory and added the user to the existing subscription.

8. Enter the relevant password and click the Sign in button. The On-premises data gateway window opens a new page to register the gateway. 9. Select the Register a new gateway on this computer checkbox to register a new gateway. Then, click the Next button, as shown in Figure 3.14.

Exploring the Features Related to EAI Integration 

 63

Figure 3.14: Registering a new gateway

The next page of the On-premises data gateway window appears wherein you need to provide the data gateway details. 10. Enter the name of the data gateway in the new on-premise data gateway name text box. In this example, we have entered BTS2016TestDataGateway2. 11. Enter the desired recovery key in the Recovery key text box. 12. Confirm recovery key: Enter the same recovery key. 13. Click the Configure button, as shown in Figure 3.15.

Figure 3.15: Specifying details of the gateway

64 

 Chapter 3: BizTalk Server 2016 Features

Once the configuration is complete, the On-premises data gateway window displays the status that the gateway is online and ready to be used, as shown in Figure 3.16.

Figure 3.16: On-premises data gateway window with completion status

With this, we are done with the setting up and configuration of a new data gateway. Creating Data Gateway Resource on Azure Perform the following steps to create a data gateway resource on Azure: 1. Sign in to the Azure portal using the credentials used for configuring the on-premises data gateway. 2. Search for “on-premises data gateway.” A list of results appears. 3. Click the On-premises data gateway option from the results. The Create connection gateway window appears. 4. Type the desired name in the Resource Name text box. 5. Select the desired subscription option from the Subscription drop-down list.

Exploring the Features Related to EAI Integration 

 65

6. Click the Create button to create a data gateway resource, as shown in Figure 3.17.

Figure 3.17: Creating a data gateway resource

After clicking the Create button, a new data gateway resource is created on Azure, which points to the on-premises data gateway hosted on BizTalk Server. This process can take a few minutes. Setting up BizTalk Server for Communicating with Azure Logic Apps The process of setting up BizTalk Server for communicating with Azure Logic Apps includes the following tasks: –– Installing the Azure Logic Apps adapter –– Setting up Management and ReceiveService applications on IIS –– Creating Azure Logic App

66 

 Chapter 3: BizTalk Server 2016 Features

Installing Azure Logic Apps Adapter As discussed earlier, BizTalk Server has its own adapter for communicating with Azure Logic Apps. However, the Azure Logic Apps adapter does not come bundled with BizTalk Server 2016. You need to download and install it separately. Perform the following steps to install Azure Logic Apps adapter: 1. Visit https://www.microsoft.com/en-us/download/details.aspx?id=54287. The Microsoft BizTalk Server Adapter for Logic Apps window appears. 2. Click the Download button to download the LogicAppAdapter.iso file. 3. Open the LogicAppAdapter.iso file. 4. Run the LogicAppAdapter.msi file to install Azure Logic Apps adapter. 5. Accept the license agreement. 6. Click the Install button. Once the installation is complete, you need to restart the BizTalkServerApplication and BizTalkServerIsolatedHost host instances. The LogicApp Adapter folder (C:\Program Files (x86)\Microsoft BizTalk Server 2016\LogicApp Adapter) contains the following two services: –– Management: Refers to a service that enables communication between BizTalk Connector in Azure Logic App and BizTalk Server. It works on the receive-side of BizTalk and enables BizTalk Server to receive messages from Azure Logic App with the help of data gateway. –– ReceiveService: Refers to a service that is used by BizTalk Connector in Azure Logic App when the receive location is entered. Like the management service, this service is used only on the receive-side of BizTalk Server. It is used for sending messages from Azure Logic App. Setting up Management and ReceiveService Applications on IIS BizTalk Connector uses the URL of the Management IIS application to access the data gateway on BizTalk Server. To create the Management IIS application, you need to perform the following steps: 1. Open the Web browser window and navigate to Internet Information Services (IIS) Manager. 2. Right-click the Default Web Site option. A context menu appears. 3. Select the Add Application option from the context menu. 4. Enter the desired alias (name) for your application in the Alias text box. In our case, we have entered IISLogicApp. 5. Select the desired application pool. In our case, we are using the default application pool.

Exploring the Features Related to EAI Integration 

 67

Note If you want to select an application pool that is not listed in the Application pool box, you need to click the Select option that opens the Select Application Pool dialog box. You need to select the application pool from the list of application pools and then click the OK button.

6. Click the Browse button to navigate to the folder that you want to set for the Physical path text box. In our case, we have set the folder location to C:\Program Files (x86)\Microsoft BizTalk Server 2016\LogicApp Adapter\Management. 7. Click the Test Settings button to verify the application pool identity and pass the authentication and authorization tests. 8. Click the OK button to save the changes. The navigation folder for the Management service is shown in Figure 3.18.

Figure 3.18: Management service

As discussed earlier, BizTalk Connector uses the URL of the ReceiveService application when the receive location is selected. Perform the following steps to create the ReceiveService application: 1. Open the Web browser window and navigate to Internet Information Services (IIS) Manager. 2. Right-click the Default Web Site option. A context menu appears. 3. Select the Add Application option from the context menu. 4. Enter the desired alias (name) for your application in the Alias text box. In our case, we have entered ReceiveWCFService. 5. Select the application pool that you selected in the Management IIS application. 6. Click the Browse button to navigate to the folder that you want to set for the Physical path text box. In our case, we have set the folder location to C:\Program Files (x86)\Microsoft BizTalk Server 2016\LogicApp Adapter\ReceiveService. 7. Click the Test Settings button to verify the application pool identity and pass the authentication and authorization tests. 8. Click the OK button to save the changes.

68 

 Chapter 3: BizTalk Server 2016 Features

The navigation folder for the ReceiveService service is shown in Figure 3.19.

Figure 3.19: ReceiveService service folder

Creating Azure Logic App Perform the following steps to create Azure Logic App: 1. Log in to the Azure portal. 2. Type the desired keywords for searching logic apps in the Search text box. 3. Click the Logic App option. The Create logic app window appears. 4. Enter the relevant details in all the required fields. 5. Click the Create button for creating a new logic app, as shown in Figure 3.20.

Figure 3.20: Creating a logic app window

Exploring the Features Related to EAI Integration 

 69

As you click the Create button, the new Azure Logic App starts deploying. Once done, the created Azure Logic App is displayed, as shown in Figure 3.21.

Figure 3.21: Displaying Azure Logic App

As discussed earlier, Azure Logic App starts with a trigger and executes a sequence of events on execution. Therefore, the need arises for adding a workflow to the created Azure Logic App. To design and create a workflow, we need a tool, called Logic App Designer. This tool becomes available under the Development Tools category. Perform the following steps to create a workflow: 1. Click the Logic App Designer button under the DEVELOPMENT TOOLS section, as shown in Figure 3.22.

70 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.22: Clicking the Logic App Designer button

The Logic App Designer window appears. 2. Select the trigger named, “When a HTTP request is received.” 3. Select the action named, “BizTalk Server - Prepare message from XML” to the trigger. The information about the gateway fills in automatically. 4. Click the Create button, as shown in Figure 3.23.

Exploring the Features Related to EAI Integration 

 71

Figure 3.23: The Logic App Designer window

After you click the Create button, you are prompted to provide some action configuration. 5. Select the HTTP body output for the Input Message text box. 6. Select the schema to be used as a receive message type from the Schema drop-down list, as shown in Figure 3.24.

72 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.24: Setting the action configuration

7. Click the Save button. After you click the Save button, the URL besides the HTTP POST URL option is displayed, as shown in Figure 3.25.

Figure 3.25: Displaying the link beside the HTTP POST URL option

8. Click the Copy icon. The URL is copied. 9. Save the copied URL to an appropriate location. In a similar fashion, you need to add an action to prepare a message from the requested data. To do this, you need to add an action named “Prepare message

Exploring the Features Related to EAI Integration 

 73

from JSON.” For this, you first need to configure the data gateway for on-premises BizTalk Server and then configure the action, as listed below: 2. Set the data gateway properties by performing the following steps: a. Specify the URL of the Logic App management service application in the BizTalk Server URL text box. b. Select the Windows option from the Authentication type drop-down list. c. Specify the user name of the Management service application pool in the User Name text box. d. Specify the password of the Management service application pool in the Password text box. e. Select the desired data gateway from the Gateway drop-down list. f. Click the Create button. 3. Configure the action by performing the following steps: a. Select the HTTP trigger body output for the Input Message text box. b. Select the desired schema from the Schema drop-down list that contains all the schemas deployed on the local on-premises BizTalk Server. The workflow is created, as shown in Figure 3.26.

Figure 3.26: Displaying the workflow

74 

 Chapter 3: BizTalk Server 2016 Features

Sending Transformed/Prepared Message Data from Azure Logic App to BizTalk Server Perform the following steps to send transformed message data from Azure Logic App to BizTalk Server: 1. Add an action on the workflow as “BizTalk Server – Send Message.” 2. Specify the complete URL of the Azure Logic App receive service in the Receive Location text box. 3. Set the previous action’s output body as the value for the Input Message text box. 4. Click the Save button to save the configuration. Note If you have receive locations in BizTalk Server with transport type set to “Logic App,” a list of all those receive locations will be displayed. Otherwise, a complete URL of the Azure Logic App receive service is required to be entered as follows: http://administrator/ReceiveWCFService/service1.svc

Once the configuration is saved, a workflow is developed, as shown in Figure 3.27.

Figure 3.27: Displaying the workflow

Exploring the Features Related to EAI Integration 

 75

Receiving Data into BizTalk Server from Azure Logic App For receiving data into BizTalk Server from Azure Logic App, you need to perform the following steps: 1. Create a Receive Port. 2. Add a receive location to the created Receive Port. The Receive Location Properties dialog box appears. 3. Specify a name for the receive location. 4. Select the Logic App option from the Transport type drop-down list. 5. Click the Configure button. The Logic App Transport Properties dialog box appears. 6. Enter: “/ReceiveWCFService/service1.svc” in the Address (URI) field, because the Azure Logic App receive service was deployed through an application named “ReceiveWCFService.” 7. Enter: “http://administrator/ReceiveWCFService/service1.svc” in the Public Address field. 8. Click the OK button, as shown in Figure 3.28.

Figure 3.28: Specifying transport properties related to Azure Logic App

76 

 Chapter 3: BizTalk Server 2016 Features

Send Messages to Azure Logic App as HTTP Request For sending messages to Azure Logic App as HTTP request, you primarily need one of the API integration testing tools including Postman, Fiddler, and Google advanced REST client. Note We are using the Google advanced REST client API integration testing tool.

Perform the following steps to send messages to Azure Logic App as HTTP request: 1. Open the API integration testing tool. 2. Specify the Request trigger URL that is received when the trigger is saved in Azure Logic App workflow. In our case, the URL is: https://prod-05.eastus2.logic.azure.com:443/ workflows/6cbd11f5562e481c85aefc11ca88380c/triggers/manual/paths/ invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0 &sig=UCPsjEgOHqpSh9gIbM6K0tpOteVXS9hhZLSDaD5D1dY 3. Specify the POST method as http-verb. 4. Specify the value of Content-type as application/json. 5. Select the PO schema that sends a sample data related to the schema. 6. Click the Send button, as shown in Figure 3.29.

Figure 3.29: Sending messages to Azure Logic App

Exploring the Features Related to EAI Integration 

 77

Since this is a one-way call to Azure Logic App and then to BizTalk Server, the result shows HTTP-202 Accepted. Therefore, we can say that Azure Logic App first receives a trigger, prepares a message from requested data and uses that message to send data to BizTalk Server through the Azure Logic App receive location.

Support for SAP .NET Connector (NCo) BizTalk Server 2016 supports both classic RFC SDK and NCo (.NET Connector). The support for classic SAP RFC SDK was depreciated when an update was made to the SAP adapter as WCF-SAP adapter. Earlier SAP systems were used to allow other systems to connect to SAP application servers through RFC SDK. However, the update as WCF-SAP adapter stopped supporting RFC SDK. It resulted in affecting the integration between Microsoft BizTalk Server and SAP systems. To overcome the issue, SAP NetWeaver Library was introduced. The support for classic RFC SDK is back in Microsoft BizTalk Server 2016. The updated BizTalk SAP adapter supports both. However, their usage is optional. You can select the desired option with the help of the ConnectorType attribute in the SAP binding while using SAP adapter. Note To use SAP NCo, you need to install SAP NCo service available at the SAP service marketplace. You also need to install BizTalk adapter cumulative update.

Connect to SQL Server Always Encrypted Column from BizTalk Server Always Encrypted is a feature of SQL Server that helps in protecting confidential data stored in Azure SQL database or SQL Server databases. It encrypts the confidential data provided in client applications. The generated encryption keys are not disclosed to the database engine. To allow BizTalk to connect to SQL Server Always Encrypted column, you need to create an Always Encrypted column in SQL Server and then enable the Always Encrypted option in BizTalk. Creating Always Encrypted Column in SQL Server As discussed earlier, BizTalk 2016 allows you to connect to the Always Encrypted column in SQL Server. Data can be fetched in either encrypted or hashed format. To create an Always Encrypted column in SQL Server, you should have a database containing a table and several columns.

78 

 Chapter 3: BizTalk Server 2016 Features

Note In our case, we have created a database named “TestDB.” The database contains a table named “Employee.” There are three columns in the table: EmpNo, EmpName, and SSN.

Perform the following steps to create an Always Encrypted column in SQL server: 1. Open the SQL Server 2016 application. 2. Right-click the desired database in which you want to create an Always Encrypted column. In this case, we have right-clicked TestDB. A context menu appears. 3. Select the Tasks option from the context menu. A submenu appears. 4. Select the Encrypt Columns option, as shown in Figure 3.30.

Figure 3.30: Selecting the Encrypt Columns option

The Always Encrypted wizard appears with the Column Selection page.

Exploring the Features Related to EAI Integration 

 79

5. Select the desired column that you want to set as always encrypted. In our case, we have selected SSN. 6. Select the desired encryption type for the selected column from the Encryption Type drop-down list. In our case, we have selected Deterministic. 7. Select an encryption key for the selected column. 8. Click the Next button, as shown in Figure 3.31.

Figure 3.31: Specifying the Column Selection settings

The Master Key Configuration page of the Always Encrypted wizard appears. 9. Select the desired option from the Select column master key drop-down list. In our case, we have selected Auto generate column master key. 10. Select the desired key store provider in the Select the key store provider section. In our case, we have selected Windows certificate store. 11. Select the desired master key source from the Select a master key source drop-down list. In our case, we have selected Current User.

80 

 Chapter 3: BizTalk Server 2016 Features

12. Click the Next button, as shown in Figure 3.32.

Figure 3.32: Configuring the master key

The Run Settings page of the Always Encrypted wizard appears. 13. Select the Proceed to finish now radio button. 14. Click the Next button, as shown in Figure 3.33.

Exploring the Features Related to EAI Integration 

Figure 3.33: Displaying the Run Settings page

The Summary page of the Always Encrypted wizard appears. 15. Review the summary. 16. Click the Finish button to finish the processing, as shown in Figure 3.34.

 81

82 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.34: Displaying the Summary page

Once the processing is complete, the Results page of the Always Encrypted wizard appears. 17. Click the Close button to close the wizard, as shown in Figure 3.35.

Exploring the Features Related to EAI Integration 

 83

Figure 3.35: Displaying the Results page

Enabling Always Encrypted Setting in BizTalk Server To enable the Always Encrypted setting in BizTalk Server, perform the following steps: 1. Open the BizTalk Admin console. 2. Create a Receive Port. 3. Specify the receive location of the Receive Port as WCF-SQL. 4. Click the Configure button. The WCF-SQL Transport Properties dialog box appears. 5. Select the Binding tab. 6. Set the ColumnEncryptionSetting property to Enabled, as shown in Figure 3.36.

84 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.36: Enabling Always Encrypted setting in BizTalk Server

Note When you select the Enabled option, the encrypted data is fetched from the Always Encrypted database. On the other hand, the Disabled option allows the port to query the Always Encrypted database that returns hashed data.

Exploring the Features Related to Business Process Control Integration In this section, you will learn about following features: –– Map compilation –– Ordered delivery on dynamic Send Ports –– Advanced location scheduling

Exploring the Features Related to Business Process Control Integration 

 85

Map Compilation Map compilation was always an issue with earlier versions of BizTalk Server. One of the issues was related to the use of APIs used for map compilation including XslTransform and XslCompiledTransform. In earlier versions of BizTalk Server (before the introduction of BizTalk Server 2013), the XslTransform API was used for the compilation of maps. However, with the introduction of BizTalk 2013, maps started using XslCompiledTransform API. This API provided a faster compilation rate in comparison to the older versions. Also, its ability to test maps in Visual Studio itself made it more useful. However, some issues were still present with this API, including no support for inline scripting in maps, no support for returning “null,” and inability to create private methods, etc. These issues have been fixed in the newer version of BizTalk Server, named BizTalk Server 2016. Now, there is an option available for developers to choose between XslTransform and XslCompiledTransform. Note You can find the issues related to BizTalk Server 2013 at the following link: https://support.microsoft.com/en-gb/help/2954101/known-issues-in-biztalk-server-2013

You can define one of the values for the Use XSL Transform property in the Properties dialog box for map compilation, as shown in Figure 3.37.

86 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.37: Setting map compilation properties

In Figure 3.37, we can see that there are three options for map compilation, including Undefined, True, and False. A brief description of these options is as follows: –– Undefined: This uses different registry flag depending on the host instance for the XslTransform setting, as follows: ○○ For 64-bit host instances, it uses the registry flag as: HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0\Configuration ○○ For 32-bit host instances and Visual Studio’s Test Map functionality, it uses the registry flag as: ○○ HKLM\SOFTWARE\Wow6432Node\Microsoft\BizTalk Server\3.0\Configuration –– True: This allows you to set the map compilation property to the XslTransform class. –– False: This allows you to set the map compilation property to the XslCompiledTransform class.

Exploring the Features Related to Business Process Control Integration 

 87

Ordered Delivery on Dynamic Send Ports This is one of the new features of BizTalk Server 2016. This feature allows you to dynamically specify the transport types for dynamic ports. It allows ordered delivery of the messages on dynamic Send Ports. The ordered delivery of messages signifies that messages are delivered to each subscriber in the same sequence followed by messages that are published to the MessageBox database. To enable this feature, you need to select the checkbox next to the Ordered delivery option in the Transport Advanced Options section of the Send Port Properties dialog box, as shown in Figure 3.38.

Figure 3.38: Enabling Ordered Delivery on Dynamic Send Ports

One of the biggest advantages of enabling ordered delivery on dynamic Send Ports is that there is no outbound message failure occurring in the event of any type of failure. This is due to the fact that the Send Port stops automatically when there is an error in the outbound transport. This means that all the messages

88 

 Chapter 3: BizTalk Server 2016 Features

that have been subscribed by that particular port are suspended until the error is fixed, and thus start transforming, in the same sequence once there is no error. In a nutshell, BizTalk Server ensures that the outbound port receives the next message when the previous message has been sent successfully.

Advanced Location Scheduling BizTalk Server 2016 allows you to set advanced scheduling properties in receive locations. The advanced scheduling properties include setting the time zone and the recurrence schedule on the receive location. Time Zone The Time Zone property allows you to set the desired time zone in the receive location that may be different from the time zone specified in your local computer. Depending on the selected time zone, dates and time are specified in the Schedule properties accordingly. It also allows you to either select or deselect Daylight Saving Time (DST). Figure 3.39 shows different options available for the Time zone property under the Schedule properties.

Figure 3.39: Different options available for the Time zone property

Exploring the Features Related to Business Process Control Integration 

 89

Recurrence BizTalk Server 2016 allows you to run the Receive Port on a daily, weekly or monthly basis as per your selection. Figure 3.40 shows daily recurrence configuration.

Figure 3.40: Daily recurrence configuration

Figure 3.41 shows weekly recurrence configuration.

Figure 3.41: Weekly recurrence configuration

Figure 3.42 shows monthly recurrence configuration.

90 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.42: Monthly recurrence configuration

Exploring the Features Related to the Business to Business Integration In this section, you learn about the following features: –– Connecting to Azure file share from BizTalk file adapter –– Refined B2B import/export process –– Improved FTP and SFTP adapters

Connecting to Azure File Share from BizTalk File Adapter BizTalk Server 2016 allows you to connect to Azure file share with the help of a file adapter. To use this feature, you need to perform the following tasks: –– Create an Azure file storage account –– Create a file share in the Azure file storage account –– Configure BizTalk ports to access file share –– Test BizTalk access to Azure file storage share Creating an Azure File Storage Account You need to perform the following steps for creating an Azure file storage account: 1. Log in to the Azure portal. 2. Search for the storage account. 3. Click the Create button. The Create storage account window opens. 4. Enter the required details in the respective text boxes.

Exploring the Features Related to the Business to Business Integration 

 91

5. Click the Create button to create the storage account, as shown in Figure 3.43.

Figure 3.43: Creating Azure file storage account

92 

 Chapter 3: BizTalk Server 2016 Features

Creating a File Share in the Azure File Storage Account Once the Azure file storage account is created, you need to create a file share in the storage account. To do this, you need to perform the following steps: 1. Visit the Azure storage account. 2. Click the Files option to create a file share, as shown in Figure 3.44.

Figure 3.44: Clicking the Files option

The File service window opens. 3. Click the File share button. The File service window displays the New file share options. 4. Type the desired name for the file share in the Name text box. In our case, we have named the file share as testfileshare. 5. Click the OK button, as shown in Figure 3.45.

Figure 3.45: Creating a file share

Exploring the Features Related to the Business to Business Integration 

 93

Note You can specify the desired value for the Quota attribute. However, it will take 5TiB as the default value if no value is specified for it.

As you click the OK button, the file share is created. 6. Click the ellipsis ( ) icon on the right side of file share name. A pop-up menu appears. 7. Click the Properties option, as shown in Figure 3.46.

Figure 3.46: Click the Properties option

The Share properties dialog box opens. 8. Click the Copy icon beside the URL option, as shown in Figure 3.47.

Figure 3.47: The Share properties dialog box

9. Save the copied URL to the desired location. Note The copied URL should be saved to an appropriate location because it will be used to access the Azure storage file share from BizTalk ports.

94 

 Chapter 3: BizTalk Server 2016 Features

Configuring BizTalk Ports to Access File Share You need to configure BizTalk ports (the Receive Port and Send Port) to access the file share saved on the specified Azure storage account. To configure BizTalk ports, you need two file shares in the same storage account. Note In our case, we have created one more file share, named filesharebackup, in the same storage account.

You should note that Azure file shares cannot be accessed without the credentials that include Azure storage account name and Azure account access key. To get the access key for each file share, you need to perform the following steps: 1. Navigate to the Azure storage account. 2. Click the Access keys option under the SETTINGS group in the left pane. The settings related to Access keys appear in the right pane of the window, as shown in Figure 3.48.

Figure 3.48: Displaying the access keys

In Figure 3.48, you can see that there are two keys, named key1 and key2. You can click the Copy icon beside each key and save them at an appropriate location as they are used for configuring ports. After getting the access keys, you can configure BizTalk ports. You need to configure the Receive Port for receiving the file from Azure storage. Perform the following steps to configure the receive port: 1. Create a receive port. 2. Set the receive location to testfileshare (\\bts2016filestorage.file.core. windows.net\testfileshare), as shown in Figure 3.49.

Exploring the Features Related to the Business to Business Integration 

 95

Figure 3.49: Setting the receive location

3. Select the Authentication tab. 4. Select the Use these credentials when host does not have access to network share checkbox. The User name and Password fields become active. 5. Type the storage account name (bts2016filestorage) in the User name text box. 6. Type the access key (copied earlier) in the Password text box. 7. Click the OK button, as shown in Figure 3.50.

96 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.50: Configuring the receive port

In addition to the configuration of the Receive Port, you need to configure the send port that is used for sending the file to another Azure file share created earlier (named filesharebackup in our case). Perform the following steps to configure the send port: 1. Create a send port. 2. Set the Destination folder location to filesharebackup (\\bts2016filestorage. file.core.windows.net\filesharebackup), as shown in Figure 3.51.

Figure 3.51: Setting the Destination folder location

Exploring the Features Related to the Business to Business Integration 

 97

Note The send port should be subscribed to the receive port created earlier through BTS.ReceivePortName filter.

3. Select the Authentication tab. 4. Type the storage account name (bts2016filestorage) in the User name text box. 5. Type the access key (copied earlier) in the Password text box. 6. Click the OK button to complete the configuration. Testing BizTalk Server Access to Azure File Storage Share You can test BizTalk Server access to Azure file storage share by testing inbound and outbound accessibilities of BizTalk Server. To do this, you need to upload a file to the file share location named testfileshare. If the file disappears from the file share location after refreshing the file share, it means that the file has been picked up by BizTalk Server from the file share location “testfileshare.” If the file is still available in the testfileshare location, there must be a problem with the configuration of the receive port. If all goes well, a file is stored in the filesharebackup storage location. This means that both ports have been configured properly. In the filesharebackup storage location, you will get a file with the filename in the format “_.xml,” as shown in Figure 3.52.

Figure 3.52: Displaying the file received at the filesharebackup storage location

Note The filename format takes its form from the File name property specified in the send port configuration “%SourceFileName%_%MessageID%.XML.”

98 

 Chapter 3: BizTalk Server 2016 Features

The above discussions demonstrate that BizTalk can receive files from Azure storage share as well as send files to Azure storage share.

Refined B2B Import/Export Process BizTalk Server 2016 provides enhancements to the import/export options by separating them from the application level. Now, the binding files can be imported/ exported at the Parties and Agreement levels. This means that you no longer need to import/export the entire application for importing/exporting a party or an agreement. Import and Export operations can be performed in both Graphical User Interface (GUI) and Command Line Interface (CLI) modes. In CLI mode, BTSTask commands (ImportParties and ExportParties) are used for importing and exporting parties. Table 3.2 shows the modifications made to Import/Export operations. Table 3.2: Modifications to Import/Export Operations Import

Export

1. You can import the tracking settings with the help of the Application Import Binding wizard.

1. You can export all agreements associated with the selected parties.

2. You can exclude the parties or fallback settings while importing a binding file.

2. You can export specific parties to the same binding file.

3. You can import business profiles, parties, and agreements from a binding file (written in XML).

3. You can export single or multiple parties to the same binding file.

4. You can import only parties and agreements within the binding file while executing party-level import.

4. You can export EDI fallback settings, agreements or business profiles to the same binding file.

Perform the following steps to export multiple parties together: 1. Open the BizTalk Server Administration wizard. 2. Expand the BizTalk group. 3. Select the Parties option. The Parties and Business Profiles pane appears. 4. Press the CTRL key and click the left mouse button to select multiple parties in the Parties and Business Profiles pane. 5. Right-click the desired Name. A context menu appears. 6. Select the Export option from the context menu, as shown in Figure 3.53.

Exploring the Features Related to the Business to Business Integration 

 99

Figure 3.53: Displaying the Parties and Business Profiles pane

The Export Bindings: Parties dialog box appears. 7. Specify the location and file name to which you want to export the bindings in the Export to file text box. 8. Click the OK button, as shown in Figure 3.54.

Figure 3.54: Displaying the Export Bindings: Parties dialog box

100 

 Chapter 3: BizTalk Server 2016 Features

Note All the parties selected into the Parties and Business Profiles pane are exported into the same binding file. There is no separate file for each party in this case.

Perform the following steps to exclude EDI fallback settings while importing parties: 1. Expand the BizTalk Group option in the left pane of BizTalk Server Administration Console. 2. Right-click the Parties option. A context menu appears. 3. Select the Import option from the context menu. The Import Bindings: Parties dialog box appears. 4. Click the Browse button beside the Select the binding file text box to locate the path where the binding file is stored. 5. Select or clear the Exclude EDI Fallback Settings checkbox under the Import options section. 6. Click the OK button, as shown in Figure 3.55.

Figure 3.55: Importing parties

Perform the following steps to export all the parties at once: 1. Expand the BizTalk Group option in the left pane of BizTalk Server Administration Console. 2. Right-click the Parties option. A context menu appears.

Exploring the Features Related to the Business to Business Integration 

 101

3. Select the Export option from the context menu. The Export Bindings: Parties dialog box appears. 4. Specify the file path where you want to store the binding file in the Export to file text box. 5. Click the OK button, as shown in Figure 3.56.

Figure 3.56: Exporting all parties at once

As discussed earlier, CLI supports the export/import operations through BTSTask, which uses commands including ImportBindings and ExportBindings to perform import and export operations respectively. A list of commands supported by BTSTask is shown in Figure 3.57.

102 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.57: Displaying the commands supported by BTSTask

Improved FTP and SFTP Adapters BizTalk Server 2016 provides improved support for adapters including FTP and SFTP. In this section, you will learn about the improvements made to both the adapters. Supporting More FTP Servers In BizTalk Server 2016, the use of the SYST command is not always required for configuring FTP adapters, as with earlier versions of BizTalk Server. Now, you can use the FTP Server Type property that provides several values for different types of FTP servers. You can select the desired FTP server. The selection of FTP server examines the usage of the SYST command. Figure 3.58 shows different types of FTP servers for the FTP Server Type property.

Exploring the Features Related to the Business to Business Integration 

 103

Figure 3.58: Different types of FTP servers for the FTP Server Type property

Note The FTP Server Type property is displayed when the FTP adapter is configured to either the receive location or send port.

Improved SFTP Adapters In BizTalk Server 2016, an SFTP adapter uses the SSH file transfer protocol for transmitting messages. You can use the SFTP adapters to log on to SFTP sites to troubleshoot the errors. The SFTP adapter in BizTalk Server 2016 supports more SFTP servers with the help of WinSCP which is used to connect to SFTP. In addition, BizTalk Server 2016 provides more encryption values for SFTP adapters. Before BizTalk Server 2016, there were only three values for the Encryption Cipher option including Auto, AES, and TripleDES. Now, three more values are available for the Encryption Cipher option including Arcfour, Blowfish, and DES, as shown in Figure 3.59.

104 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.59: Displaying values for the Encryption Cipher option

Exploring the Improvements to Miscellaneous/UI Features In this section, you will learn about the following features: –– Simultaneous configuration of multiple hosts/host instances –– Searching artifacts –– Save multiple suspended messages simultaneously –– Connect to management REST APIs –– Operational data feed for Power BI –– Resizable schema window –– BizTalk Server 2016 Azure VMs in production

Exploring the Improvements to Miscellaneous/UI Features 

 105

Simultaneous Configuration of Multiple Hosts/Host Instances In earlier versions of BizTalk Server, you were required to change configuration settings for each host/host instance separately. For a single change in the configuration setting of each host, you were required to repeat the same process many times. This process has become simple in BizTalk Server 2016 as multiple hosts/ host instances can be configured simultaneously. You just need to select all the hosts/host instances collectively that require the same change in their configuration settings. Figure 3.60 shows the multiple hosts selection in BizTalk Server Administration Console. It also shows the BizTalk Settings Dashboard dialog box containing configuration settings that can be changed collectively for the selected hosts.

Figure 3.60: Configuring multiple hosts simultaneously

Note The number of context options varies based on the single host selection and multiple hosts selection.

106 

 Chapter 3: BizTalk Server 2016 Features

Searching Artifacts In the earlier versions of BizTalk Server, you were required to manually look for an artifact from the list of artifacts. BizTalk Server 2016 provides the Search feature that can be used to locate a particular artifact directly. The Search feature is shown in Figure 3.61.

Figure 3.61: Using the Search feature

Save Multiple Suspended Messages Simultaneously Before the introduction of BizTalk Server 2016, it was required to save each suspended message separately. It was a time-consuming process. This problem has been rectified in the latest version of BizTalk Server. Now, you can select multiple suspended messages with the help of the CTRL and SHIFT keys on the keyboard and save them simultaneously to a folder with a single click (using the Save to File option from the context menu that appears when you right-click after the selection). However, you should note that each suspended message is saved in its own separate file.

Exploring the Improvements to Miscellaneous/UI Features 

 107

Connect to Management REST APIs Prior to BizTalk Server 2016, either WMI or BizTalk explorer OM assembly was used to communicate with BizTalk Server. However, the communication could not be done through the Web. This issue has been resolved in BizTalk Server 2016 feature pack 1 through REST APIs. In fact, APIs refer to the endpoints that help in remotely updating and querying the status of artifacts, which are available on BizTalk Server. REST is used to add these endpoints that have their own swagger definitions. The REST APIs and their swagger definitions are installed through Windows PowerShell script. Note A list of references for REST APIs can be found at the following link: https://msdn.microsoft.com/en-us/library/mt801283.aspx

Did You Know? To connect BizTalk Server to Management REST APIs, you should have IIS and BizTalk 2016 feature pack 1 installed. In general, most of the BizTalk environments have IIS installed.

Enabling REST API After installing BizTalk Server 2016 feature pack 1, you need to enable REST API. Perform the following steps to enable REST API: 1. Open Windows PowerShell with administrator privileges. 2. Navigate to the file C:\Program Files (x86)\Microsoft BizTalk Server 2016\FeaturePack.ConfigureServices.ps1. 3. Run the following command and pass the desired values to parameters: FeaturePack.ConfigureServices.ps1 -Service management -WebSiteName ‘’ -ApplicationPool -ApplicationPoolUser \ -ApplicationPoolUserPassword -AuthorizationRoles ‘\, \’

The result of the command is shown in Figure 3.62.

108 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.62: Output of the command

4. Run IIS to look for an application named BizTalkManagementService. 5. Browse the URL http://localhost/BizTalkManagementService/swagger/ui/ index. The user interface of the BizTalk Management Service displays, as shown in Figure 3.63.

Figure 3.63: Displaying the user interface of the BizTalk Management Service

If the user interface is displayed, we can say that REST API has been enabled successfully.

Operational Data Feed for Power BI BizTalk Server 2016 allows you to send BizTalk operational data as an input/ feed to Power BI. Operational data is the data related to applications, batches,

Exploring the Improvements to Miscellaneous/UI Features 

 109

instances, tracked messages, transaction sets, etc. available in the BizTalk environment. The operational data can be fetched through a visualization tool. One of the most prominent visualization tools is Power BI. To get the operational data feed, you first need to enable operational data feed. Enabling Operational Data Feed for BizTalk Server 2016 The process of enabling operational data feed is almost like setting up Management REST APIs. To enable operational data feed, you need to change the service parameter value to “operationaldata.” Perform the following steps to enable operational data feed: 1. Run Windows PowerShell with administrator privileges. 2. Navigate to the folder where BizTalk Server 2016 is installed. 3. Run the following command and pass the desired values to the parameters: FeaturePack.ConfigureServices.ps1 -Service operationaldata -WebSiteName ‘’ -ApplicationPool -ApplicationPoolUser \ -ApplicationPoolUserPassword -AuthorizationRoles ‘\, \’

4. Press the Enter key. The script runs and gives some output, as shown in Figure 3.64.

Figure 3.64: Enabling operational data feed

5. Open the Web browser window. 6. Browse the http://localhost/BizTalkOperationalDataService link, as shown in Figure 3.65.

110 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.65: Browsing the IIS Application

Using Operational Data Feed as Power BI Template Perform the following steps to access the Power BI template file: 1. Download and install the Power BI desktop. 2. Open the IIS Manager window. 3. Expand the ADMINISTRATOR Sites Default Web Site nodes. 4. Right-click the BizTalkOperationalDataService service. A context menu appears. 5. Click the Explore button from the context menu, as shown in Figure 3.66.

Figure 3.66: Exploring the BizTalkOperationalDataService service

Exploring the Improvements to Miscellaneous/UI Features 

 111

The OperationalDataService folder opens. 6. Open the BizTalkOperationalData.pbix file, as shown in Figure 3.67.

Figure 3.67: Displaying the OperationDataService folder

After opening the file, a dialog box appears asking for the BizTalk operation service URL. 7. Enter the application URL in the BTSOPSURL text box. 8. Click the Load button, as shown in Figure 3.68.

Figure 3.68: Loading the BizTalkOperationalDataService service

Once the loading is done, the operational data is displayed, as shown in Figure 3.69.

112 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.69: Displaying the operational data

As shown in Figure 3.69, there are four tabs in the bottom. You can switch to different tabs for getting operational data related to the selected tab.

Resizable Schema Window BizTalk Server 2016 allows you to resize the schema picker (The Type Picker window). The Type Picker window appears when a source schema/destination schema is either added or replaced during map creation. Now, the schema tree created by the source and destination schemas can be expanded or collapsed as per your requirements. You can expand/collapse the schema tree either for exposing or hiding schema nodes. It helps in analyzing complete information about the schema including its assembly and name. You can drag the bottom-right corner of the BizTalk Type Picker window for increasing or decreasing its size, as shown in Figure 3.70.

Exploring the Improvements to Miscellaneous/UI Features 

 113

Figure 3.70: Resizing the BizTalk Type Picker window

Note You can use a key combination of CTRL+M or CTRL+E to expand the schema tree node. To collapse, you can use either the CTRL+C or the CTRL+E key combination.

BizTalk Server 2016 Azure VMs in Production In BizTalk Server 2016, production infrastructures are supported by Azure virtual machines. An Infrastructure as a Service (IaaS) service is introduced on Microsoft Azure to provide support for BizTalk Server on all kinds of platforms. During the launch of BizTalk Server 2016, only one edition of BizTalk Server was available, named the Enterprise edition. However, two more editions including Standard and Developer are available on Azure, as shown in Figure 3.71.

114 

 Chapter 3: BizTalk Server 2016 Features

Figure 3.71: Displaying editions of VMs on Azure

Summary BizTalk Server 2016, the latest version of BizTalk Server, supports most of the platforms including SQL Server 2016, Windows Server 2016, Visual Studio 2015, and Office 2016. It provides numerous new and enhanced features that can be categorized based on the integration pattern to which they belong. Some features added to EAI include support for SAP .NET Connector (NCo), artifacts analytics tracking, Azure Logic Apps adapter, and connection to SQL Server Always Encrypted column from BizTalk. Some features added to Business Process Control include map compilation, ordered delivery on dynamic Send Ports, and advanced location scheduling. The features added to business to business integration include connecting to Azure file share from the BizTalk file adapter, refined B2B import/ export process, and improved FTP and SFTP adapters. Other features, including simultaneous configuration of multiple hosts/host instances, search artifacts, saving multiple suspended messages simultaneously, etc. are put into the “miscellaneous” category.

Chapter 4 BizTalk in Microsoft Azure Services Cloud is the new frontier and Microsoft is a dominant provider of cloud services via Microsoft Azure. Microsoft provides several options for hosting BizTalk as a cloud or Azure service. One of these options is Microsoft Azure BizTalk Services (MABS). MABS is a cloud-based integration service. It implements cloud and hybrid integration solutions through business to business and Enterprise Application Integration (EAI) capabilities. BizTalk on Azure is available in the following two forms: 1. BizTalk Infrastructure as a Service – IaaS 2. BizTalk Integration Platform as a Service – PaaS There are several integration competencies provided for the Azure platform by MABS. Some of these are as follows: –– It uses X12 and EDIFACT messaging formats to allow business to business messaging –– It helps in expanding on-premises applications to the cloud – – It helps in processing messages on the cloud through rich messaging endpoints

BizTalk Infrastructure as a Service In order to build a BizTalk application, a BizTalk developer must set up the BizTalk environment. While setting up the environment, you are provided with the BizTalk development SDKs that are necessary to build the BizTalk applications. The prerequisites for setting up the BizTalk environment include the installation and setup of SQL Server, Web Servers, IIS, Visual Studio, and Network connections, etc. However, the installation and configuration of these prerequisites can take more time. You also need separate licenses for each product including Windows Server and SQL Server in case you want to implement a Production environment. To reduce this heavy configuration, a BizTalk Server Virtual Machine (VM) as IaaS (Infrastructure as a Service) is provided by Microsoft. The VM holds the complete BizTalk development environment. Microsoft has also added BizTalk Server 2016 images in the VM images gallery in the form of IaaS with the launch of BizTalk Server 2016, as shown in Figure 4.1.

DOI 10.1515/9781501505652-004

116 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.1: Displaying BizTalk Server virtual machines

BizTalk Integration Platform as a Service Azure BizTalk services were primarily used to resolve complex messaging transformation issues associated with the physical server-based enterprise applications, SAP system, and SQL Server, etc. However, these services are no longer available as the support for these services has been discontinued effective June 1, 2017. Therefore, we can say that new subscriptions for BizTalk services are not available. However, the subscription support for existing nonserver BizTalk services will continue until May 31, 2018. The main objective of Microsoft is to switch organizations toward Azure Logic Apps and Azure App service hybrid connections. Therefore, BizTalk PaaS services are available in the following different formats: 1. Azure Logic Apps 2. Integration account (part of Azure Logic Apps EIP) 3. Azure App hybrid connections

Azure Logic Apps As discussed in the previous chapter, Azure Logic App is a server-less service hosted on Azure that works as a workflow. Azure Logic Apps allow users to

BizTalk Integration Platform as a Service 

 117

organize business processes as a sequence of steps in the graphical mode. They allow users to execute workflows in the cloud. In other words, they allow users to provide integration solutions easily. Azure Logic Apps have an extensive set of connectors that connect to different enterprise applications. Benefits of Using Azure Logic Apps There are several benefits of using logic apps. Some of them are as follows: 1. Easy to design: As Azure Logic Apps are available on the Azure portal, it becomes easy to define a sequence of events in the browser. Also, you can add the desired number of actions with the help of connectors available in Azure Logic Apps. 2. Out-of-the-box Templates: There are many ready to use templates that can be utilized to create solutions, including SaaS connectivity and B2B solutions. 3. Extensibility: You can create your own set of connectors using the API and code. You can also use the Azure functions to execute code-blocks wherever required. 4. Simplified BizTalk Integration: Azure Logic Apps help in creating the projects used for communicating with legacy systems and cloud applications. The connectors and templates related to BizTalk Server have simplified the on-premises BizTalk Server integration. 5. Managed Connectors: Azure Logic Apps provide application-specific connectors that can be used for accessing data and services. These connectors can also be used to allow Azure Logic Apps to write or fetch new or existing data. These connectors can be classified into the following three types: a. Enterprise connectors: These require an additional cost. They are used with MQ and SAP systems. b. Integration account connectors: These are available with the integration account only. These connectors are generally required to perform BizTalk-related activities including encoding and decoding of flat files and processing B2B messages with AS2, EDI, or EDIFACT, etc. c. Standard connectors: These are available with Azure Logic Apps account, which means that they do not add additional cost. They are available automatically when Azure Logic App is created. 6. Easy to Compose SaaS: Azure Logic Apps provide the most reliable and the fastest way to build solutions for complex problems. Azure Logic Apps can be used to connect cloud marketing solution to the on-premises billing system. You can also use Azure Logic Apps to create a task in CRM on the basis of your account activities available on the social sites.

118 

 Chapter 4: BizTalk in Microsoft Azure Services

Integration Account An integration account is part of the Enterprise Integration Pack (EIP) of Azure Logic Apps. It helps in managing additional services provided by BizTalk Server in the EIP. EIP in Azure Logic Apps provides several new features including B2B processing and XML. It also supports several BizTalk features including EDI, EDIFACT, AS2 message types, XML validation, flat file processing (encode/ decode), and XSLT transformation, etc. All these integration artifacts are managed and secured by a scalable container known as Integration Account. Integration Account formulates the creation of B2B processes through Azure Logic Apps. An integration account can contain the following integration artifacts: –– Trading partners –– Trading partner agreements for EDI/EDIFACT/AS2 messages exchange –– XML schemas –– XSLT transforms (maps) –– Public/private key certificates Creating an Integration Account Perform the following steps to create an integration account: 1. Sign in to the Azure portal. 2. Click the New button. 3. Type the keywords “Integration Account” in the Search box. 4. Click the Integration Account option. The Integration Account window opens to display the description of the integration account. 5. Click the Create button, as shown in Figure 4.2.

BizTalk Integration Platform as a Service 

 119

Figure 4.2: Displaying the Integration Account window

The Integration Account window appears. 6. Enter the desired name for the integration account in the Name text box. 7. Select the desired type of subscription from the Subscription drop-down list. 8. Select either Create new or Use existing radio button to create a new resource group or use an existing resource group. In our case, we have selected the Create new radio button. 9. Enter the desired name for the new resource group. 10. Select the desired pricing tier from the Pricing Tier drop-down list. 11. Select the desired location from the Location drop-down list.

120 

 Chapter 4: BizTalk in Microsoft Azure Services

12. Click the Create button, as shown in Figure 4.3.

Figure 4.3: The Integration Account window

Eventually, a message box appears, which states that an integration account is deployed, as shown in Figure 4.4

Figure 4.4: The Deployment succeeded message box

BizTalk Integration Platform as a Service 

 121

Adding a Trading Partner in an Integration Account Perform the following steps to add a trading partner in an integration account: 1. Search for the existing integration account. 2. Click the Partners option under the SETTINGS group in the left pane of the Integration Account Properties pane. 3. Click the Add button in the right pane, as shown in Figure 4.5.

Figure 4.5: Clicking the Add button

The Add Partner dialog box opens. 4. Enter the desired name of the partner in the Name text box. 5. Select the desired qualifier from the Qualifier drop-down list. 6. Specify the desired value in the Value text box. 7. Click the OK button, as shown in Figure 4.6.

122 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.6: Displaying the Add Partner dialog box

Now, the new trading partner is created and displayed under the Partners category, as shown in Figure 4.7.

Figure 4.7: Adding a new trading partner

Adding Qualifiers to an Existing Trading Partner For B2B agreements, you can either use multiple trading partners or single trading partner in more than one business process. To use a single trading partner in multiple business processes, you need to add qualifiers to an existing trading partner. Perform the following steps to add more qualifiers to an existing trading partner: 1. Locate an existing trading partner.

BizTalk Integration Platform as a Service 

 123

2. Click the desired trading partner to which you want to add more qualifiers. When you select a trading partner, the Edit button is activated. 3. Click the Edit button. The Edit window appears, with multiple options to edit the partner profile. 4. Select the desired qualifier from the QUALIFIER drop-down list. 5. Specify the desired value for the selected qualifier. 6. Click the OK button, as shown in Figure 4.8.

Figure 4.8: Adding a qualifier to existing trading partner

After clicking the OK button, the new qualifier is added to the selected trading partner profile. Adding Agreements in the Integration Account Agreements are the two-way message exchange settings that both the parties have agreed to follow during inbound or outbound transactions. Perform the following steps to add an agreement in the integration account: 1. Search for an existing integration account. 2. Click the Agreements option under the SETTINGS group in the left pane of the Integration Account Properties pane.

124 

 Chapter 4: BizTalk in Microsoft Azure Services

3. Click the Add button in the right pane, as shown in Figure 4.9.

Figure 4.9: Adding an agreement

The Add window appears containing several settings for adding an agreement. 4. Enter the desired name for the agreement in the Name text box. 5. Select the desired agreement type from the Agreement type drop-down list. 6. Select the desired host partner from the Host Partner drop-down list. 7. Select the desired host identity from the Host Identity drop-down list. 8. Select the desired guest partner from the Guest Partner drop-down list. 9. Select the desired guest identity from the Guest Identity drop-down list. 10. Click the arrow icon beside the Receive Settings option, as shown in Figure 4.10.

BizTalk Integration Platform as a Service 

 125

Figure 4.10: Displaying the Add window

The Receive Settings window opens. This window contains receive settings that can be categorized into three types on the basis of the message type. The three receive settings include X12 (EDI), EDIFACT, and AS2. In our case, we are configuring X12 (EDI). The X12 (EDI) receive settings are used for identifying and handling received EDI messages. There are different sections to be configured while configuring EDI receive settings including Identifiers, Acknowledgement, Schemas, Envelopes, Validations, Control Numbers, and Internal Settings. 11. Configure ISA segments in the incoming messages under the Identifiers section, as shown in Figure 4.11.

126 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.11: Configuration under the Identifiers section

12. Set the desired configuration under the Acknowledgement section for handling different types of acknowledgments (4010–997, 5010–997, and 5010– 999), as shown in Figure 4.12.

Figure 4.12: Displaying settings under the Acknowledgement section

13. Set the desired configuration settings under the Schemas section, as shown in Figure 4.13.

Figure 4.13: Displaying configuration settings under the Schemas section

Note In the Schemas section, use the configuration settings to identify the type of schema to be used based on the transaction type (ST1) and sender application (GS2). The receive pipeline separates the inbound message by cross verifying the values for ST1 and GS2 in the incoming message with the values specified under the Schemas section.

BizTalk Integration Platform as a Service 

 127

14. Select the desired separator that you want to use for the ISA11 Usage property under the Envelopes section, as shown in Figure 4.14.

Figure 4.14: Selecting the desired separator

15. In the Control Numbers section, specify the desired settings for controlling message data, identifying duplicate messages using an interchange number, duplicating group control numbers, or duplicating transaction IDs, as shown in Figure 4.15.

Figure 4.15: Specifying the properties under the Control Numbers section

16. In the Validations section, specify the desired validation rules including validation messages, trim leading, or trailing separators, etc., as shown in Figure 4.16.

Figure 4.16: Specifying the properties under the Validations section

17. Specify the desired settings for the Internal Settings section, as shown in Figure 4.17.

128 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.17: Specifying the properties under the Internal Settings section

Note The settings under the Internal Settings section are used for splitting and preserving interchanges in the inbound messages. The properties under this section allow you to split a file with multiple transaction sets into individual files with a single transaction set. This section also contains properties that can be used to keep the file intact in case the user wants to process the complete file as a whole. This section also contains a property for creating empty XML tags when trailing separators are allowed. And last, it also contains a property for converting Nn format decimal values into base 10 numeric values.

18. Specify the send settings similar to the receive settings as they are also categorized into the same three types including X12 (EDI) send settings, EDIFACT send settings, and AS2 send settings. Similar to the receive settings, we are configuring X12 (EDI) send settings. The X12 (EDI) send settings are used to identify and handle outbound messages and add header information related to receiving partners, etc. In addition to the sections available in the receive settings, the Character Sets and Separators section contains properties that allow you to specify characters to be used as delimiters for segment terminator or data segment terminator. You need to configure these settings as per your requirement, as shown in Figure 4.18.

Figure 4.18: Specifying properties for the Character Sets and Separators section

19. Click the OK button to save all the settings.

BizTalk Integration Platform as a Service 

 129

20. Refresh the Agreements page. The created agreement is displayed in the list of agreements. Adding Maps to the Integration Account Perform the following steps to add a map to an integration account: 1. Search for an existing integration account. 2. In the Integration account pane, under SETTINGS, click Maps. 3. Click the Add button in the right pane, as shown in Figure 4.19.

Figure 4.19: Adding a map

The Add Map window appears. 4. Enter the desired name for the map in the Name text box. 5. Click the Folder icon beside the Map option to upload a map. The File select dialog box appears. 6. Select the desired map file and click the OK button to close the File select dialog box. The name of the selected file appears in the Map text box. 7. Click the OK button to save the uploaded map in the integration account, as shown in Figure 4.20.

130 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.20: Displaying the Add Map window

Adding Schemas in the Integration Account Perform the following steps to add schemas in the integration account: 1. Search for an existing integration account. 2. In the Integration account properties pane, under SETTINGS, click Schemas. 3. Click the Add button in the right pane, as shown in Figure 4.21.

Figure 4.21: Adding a schema

The Add Schema window appears. 4. Enter the desired schema name in the Name text box. 5. Select the desired tab to specify whether you want to upload a small schema file or large file (size more than 2 MB). 6. Click the Folder icon beside the Schema option to upload a schema. The File select dialog box appears. 7. Select the desired schema file and click the OK button to close the dialog box. The name of the selected file appears in the Schema text box. 8. Click the OK button to save the uploaded schema in the integration account, as shown in Figure 4.22.

BizTalk Integration Platform as a Service 

 131

Figure 4.22: Uploading a schema

Note Depending on the size of the schema file, the file upload process may vary. Schema files under 2 MB can be uploaded through the File select dialog box. On the other hand, schema files larger than 2MB are uploaded to the cloud. To access the files uploaded to the cloud, the file access URL is required.

Linking Integration Account to Azure Logic App As discussed earlier, you need to link the integration account to Azure Logic App to allow the logic app to access artifacts in the integration account. Perform the following steps to link integration account to Azure Logic App: 1. Log into to the Azure portal. 2. Look for the existing Azure Logic App through the Search feature. In our case, we have an Azure Logic App with the name “BTS2016TestLogicApp.” A list of results appears. 3. Click the logic app. In our case, we have clicked BTS2016TestLogicApp. 4. In SETTINGS, click Integration account, as shown in Figure 4.23.

132 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.23: Figure 4.23: Clicking the Integration account option

The options related to the integration account are displayed in the right pane. 5. Click the down-arrow button beside the Select an Integration account option. A drop-down list appears with the name of available integration accounts. 6. Select the desired integration account from the drop-down list, as shown in Figure 4.24.

Figure 4.24: Figure 4.24: Selecting an Integration account

7. Click the Save button to save the configuration. After clicking the Save button, a popup appears with the message “Integration account linked to logic app,” as shown in Figure 4.25.

Figure 4.25: Displaying the popup

BizTalk Integration Platform as a Service 

 133

Now, the integration account is linked to Azure Logic App, which means that Azure Logic App is capable of accessing all artifacts available in the integration account. Integration Account Connectors As discussed earlier, integration account connectors allow applications to do additional things with data including transformation and validation of XML, and encoding and decoding of flat files, etc. These connectors also allow you to expand your BizTalk workflows into Azure when used in BizTalk Server. The following integration account connectors can be used in Azure Logic Apps after creating the integration account: 1. AS2 decoding 2. AS2 encoding 3. EDIFACT decoding 4. EDIFACT encoding 5. Flat File decoding 6. Flat File encoding 7. Integration account 8. Transform XML 9. EDI (X12) decoding 10. EDI (X12) encoding 11. XML validation In this section, you will learn about different integration account connectors. AS2 Encoding and Decoding Connectors AS2 (Applicability Statement 2) is a standard devised by the Internet Engineering Task Force (IETF) for reliable and secure transmission of EDI messages/data over the Internet. The secure transmission is made possible through the digital certificates and the encryption technique. It also establishes a point-to-point connection between the client and the server. Did You Know? Similar to EDI messaging, an acknowledgment is also sent to the sender in AS2 messaging. This acknowledgment is called Message Disposition Notification (MDN).

As AS2 messages are transmitted in the encrypted format, there is a need for an encode connector (used at the sender end to encrypt messages) and decode connector (used at the receiver end to decrypt messages).

134 

 Chapter 4: BizTalk in Microsoft Azure Services

Encoding and Decoding AS2 Messages in Azure Logic App  As discussed in the earlier section, the Encode connector is used for encoding/ encrypting AS2 messages, while the Decode connector is used for decoding AS2 messages. Both connectors do not have triggers. Perform the following steps to encode AS2 messages in Azure Logic App: 1. Add a request trigger in the logic app workflow (Logic App Designer). In our case, we have added a request trigger and generated the JSON schema on the basis of the available sample JSON data. 2. Type “AS2” in the Search box. A list of related results appears. 3. Select the AS2 – Encode to AS2 message option to encode AS2 messages. The Encode to AS2 message form appears. 4. Enter details for the AS2-From field, the AS2-To field, and the body field, as shown in Figure 4.26.

Figure 4.26: Encoding AS2 messages

Similarly, the Decode connector can be added, as shown in Figure 4.27.

BizTalk Integration Platform as a Service 

 135

Figure 4.27: Decoding AS2 messages

EDIFACT Encode and Decode Connectors Electronic Data Interchange for Administration, Commerce, and Transport is an international EDI standard that is standardized by the United Nations Economic Commission for Europe (UNECE). The standard provides a set of syntaxes, data elements, segments, codes, and messages. Each EDIFACT message is notified by a six-character string, which means that the message AUTHOR points to an authorization message. More examples are listed in Table 4.1. Table 4.1: EDIFACT Messages Six-Character Name

Message

BANSTA CONITT COREOR DELFOR ENTREC FINPAY IFTICL INSPRE JOBOFF

Banking status message Invitation to tender message Container release order message Delivery schedule message Accounting entries message Multiple interbank funds transfer message Cargo insurance claims message Insurance premium message Job order message, etc.

136 

 Chapter 4: BizTalk in Microsoft Azure Services

Similar to EDI, an EDIFACT file can contain one or more interchanges wherein one interchange may contain one or more messages or transactions. An interchange starts with the UNB segment and ends at the UNZ segment. A group starts with the UNG segment and ends at the UNE segment. On the other hand, a transaction or an individual message starts with the UNH segment and ends at the UNT segment. Both EDIFACT connectors (Encode and Decode) are used to validate EDI and partner-specific properties. The Decode connector is also used for splitting interchanges into transaction sets and generating acknowledgments for the received transactions. The Encode connector is responsible for generating an XML document for each transaction set, and it adds a token in the message request for technical or functional acknowledgment. Decoding and Encoding EDIFACT Messages in Logic Apps As the Encode and Decode connectors for EDIFACT messages do not have triggers, you first need to add a request trigger in your logic app workflow. In our case, we have added a request trigger and generated JSON schema on the basis of the sample JSON data. Perform the following steps to decode EDIFACT messages in Azure Logic Apps: 1. Type “EDIFACT” in the Search box. The results related to the entered keyword appear. 2. Click the EDIFACT - Decode EDIFACT message option, as shown in Figure 4.28.

Figure 4.28: Choosing an action

BizTalk Integration Platform as a Service 

 137

The Decode EDIFACT message form appears. 3. Specify the desired data that you want to decode in the EDIFACT flat file message to decode text box. In our case, we have selected http request data as a flat file message, as shown in Figure 4.29.

Figure 4.29: Decoding EDIFACT message

4. Save the workflow. Once the workflow is saved, you will get an HTTP POST URL in the workflow trigger, as shown in Figure 4.30.

Figure 4.30: Displaying the HTTP POST URL

The generated URL can be used by any client with valid EDIFACT sample data as the request payload in the “data” object. It will return 200 as the status code, which means your request was accepted and the logic app workflow was trig-

138 

 Chapter 4: BizTalk in Microsoft Azure Services

gered. Once triggered, it will receive the value of the “data” object from the requested data, and it will be decoded to an EDIFACT message. In the same way, you can use the Encode-EDIFACT connector for encoding EDIFACT messages. EDI (X12), XML and Flat File Connectors Similar to EDIFACT and AS2 connectors, the integration account provides more connectors related to EDI and flat file processing. These connectors can be used to encode or decode flat files/EDI files. The integration account also provides connectors, called XML validation connectors, for validating schemas. The connectors can also be used for transforming one type of data into another form with the help of maps stored in the integration account.

Hybrid Connections A hybrid connection is a feature of the Azure BizTalk EIP that allows websites and mobile apps to communicate effectively with the on-premises data and resources that are located behind the firewall. This feature provides almost the same functionality as that provided by Azure Service Bus Relay. The only major difference between the two is the ability of a hybrid connection to communicate with on-premises resources through any port. It allows applications that are using a hybrid connection to query a database or communicate with an on-premises server resource located behind the firewall. In fact, the hybrid connection communicates with on-premises resources through an agent that runs on the on-premises server. The agent is communicated by a service running on the Azure portal. This service, in turn, can be used by applications that are running or executing on the Azure portal. Note All Azure services including Azure mobile services and Azure websites use hybrid connections to communicate with on-premises resources like a database, etc.

EAI Implementation Using BizTalk Azure Services In this section, you will learn about implementing an EAI scenario using integration account connectors. The implementation process will convert a flat file message with employees’ data into an XML message through a map. It requires

EAI Implementation Using BizTalk Azure Services 

 139

a flat file decoding connector for decoding flat file data and two XML connectors for XML validation and transformation. The transformation is done to generate the XML response.

Building Required Artifacts As you need to validate the flat file message, a schema is required. The schema for Azure Logic Apps can be built into the Visual Studio that requires Azure Logic Apps Tools installer ( download link at the URL: https://marketplace.visualstudio.com/items?itemName=VinaySinghMSFT.AzureLogicAppsToolsforVisualStudio). First download the file, and then install it. Once installed, the Logic Apps option will be available within the Templates category. As you select the Logic Apps option, you will see an Integration Account project in the right pane, as shown in Figure 4.31.

Figure 4.31: Displaying the Logic Apps option

You need to go through a number of steps while building the required artifacts. These steps are listed below: Step 1: Create a BizTalk project with the Azure Logic Apps template: In this step, you need to create a new project with the Azure Logic Apps template. Once the project is created, you need to right-click the project and select the Add New Item option from the context menu to add new items. A list of available templates is displayed in the right pane of the Add New Item window. You need to select the desired template to add the desired items, as shown in Figure 4.32.

140 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.32: Adding new items

Note You might face an issue after installing the Azure Logic Apps Tools installer. The installation removes the BizTalk templates from Visual Studio 2015. However, you will get the BizTalk templates back in Visual Studio 2015 after removing the Azure Logic Apps Tools from your machine. This means that you can only develop either a BizTalk project or an integration account project at the same time. In other words, you cannot run both the BizTalk project and the integration account project at the same time. Also, the tools available for the BizTalk project map templates are not available for the integration account map templates. You cannot find the tools in the Toolbox pane after adding a map in the integration account project, as shown in the following figure:

The nonavailability of tools may happen due to the fact that the integration account in Azure supports only XSLT maps, while BizTalk templates save the maps with .BTM extension. Therefore, you cannot upload maps created through the BizTalk template to your integration account in Azure. You will see only the XSLT extension in the File type combo box in the File Upload dialog box when uploading a map to the integration account. This means that the map that you are uploading to the integration account must be in XSLT format; otherwise, you will get an error, as shown in the following figure:

EAI Implementation Using BizTalk Azure Services 

 141

Step 2: Create schemas: In this step, you need to create two schemas including flat file schema (Employee_FF.xsd) and XML schema (Employee_XML.xsd). The flat file schema is created to validate the flat file input data and use it as the map source, as shown in Figure 4.33.

Figure 4.33: Displaying the flat file schema

The XML schema is created for use as the destination schema for maps. Figure 4.34 shows the XML schema.

Figure 4.34: Displaying the XML schema

Step 3: Transform the flat file schema into the XML schema: In this step, a map is created that transforms the flat file data into XML data. It adds the values available in the Salary and Allowance fields and multiplies the output value by 12 to

142 

 Chapter 4: BizTalk in Microsoft Azure Services

calculate the Total CTC. Finally, it stores the output data (outbound message) into the TotalCTC field, as shown in Figure 4.35.

Figure 4.35: Transforming the flat file schema into the XML schema

Step 4: Adding artifacts to the integration account: In this step, you need to add the already created artifacts to the integration account. This can be done by performing the steps mentioned in the earlier section “Integration Account.” After uploading schemas, the Schemas window appears, as shown in Figure 4.36.

Figure 4.36: Displaying the Schemas window

After uploading maps, the Maps window appears, as shown in Figure 4.37.

Figure 4.37: Displaying the Maps window

EAI Implementation Using BizTalk Azure Services 

 143

At this point, the required artifacts have been built and uploaded to the integration account instance. These artifacts can be used to implement a B2B process in Azure Logic Apps. Next, we are required to set up a process that receives a message having employee data in the flat file format and then converts it into the XML message.

Implementing Workflow in Azure Logic Apps using Integration Account In this section, you will learn how to build a workflow in Azure Logic Apps that requires the following operations: 1. Trigger to get the employee data from the http request body 2. Action to decode the http request data into flat file (XML) using the flat file decoding connector against the flat file schema (Employee_FF.xsd) 3. Action to validate the decoded data using XML validation connector against the flat file schema (Employee_FF.xsd) 4. Action to transform the decoded data using the map stored in the integration account 5. Action to send the output map (transformed XML) as response to the API which sent the http request Once all the listed operations are performed, the Azure Logic App workflow is displayed, as shown in Figure 4.38.

144 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.38: Displaying the Azure Logic App workflow

You can click the Save button to save the logic app workflow. Once the logic app workflow is saved, a URL (in http request trigger) is generated, which can be used to invoke the workflow.

Testing the Workflow We need to call the trigger URL through any REST API program for testing the workflow. We are using the Google advanced REST API client for testing the workflow. The required information is as follows:

EAI Implementation Using BizTalk Azure Services 

 145

–– Request URL: URL generated in the workflow trigger –– Request data: Sample flat file data, as follows: Emp-101,Rose Arnold,3560,55060 Emp-343,Ava Smith,3100,53960 Emp-983,Ruth Graham,3254,67560 Emp-403,Brandon Paterson,3933,34960 Emp-519,Gavin Lambert,8934,78900 Since a response action has been used in our workflow, we expect some data to be returned. The expected data is the map output in the XML format. We have used the URL (generated in the earlier section) to invoke the workflow. We get the result as HTTP-status-code: 200 OK (Accepted) and a response data in XML format, as shown in Figure 4.39.

Figure 4.39: Getting the result as HTTP-status-code: 200 (Accepted)

The response data received in XML format is shown in Figure 4.40.

146 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.40: Displaying the response data in XML format

From the response data shown in Figure 4.40, you can analyze that there are three fields, namely employee number, employee name, and total CTC. If we match it with the map code, we will find that the code is as expected. Therefore, we can say that the response data is the map output as expected. From the above scenario, we can say that we have implemented an end-toend workflow, which decodes and validates the flat file data using the schemas stored in the integration account. After decoding and validating, the workflow converts the data into XML data using a map stored in the integration account.

Extending the Scenario using Hybrid Connection The complete EAI scenario can be extended to make it smaller. This can be done by implementing the hybrid connection to the SQL Server database residing on the on-premises SQL Server. The SQL Server database stores the map output data. The hybrid connection connects the on-premises resources to the SQL Server

EAI Implementation Using BizTalk Azure Services 

 147

database. The hybrid connection can be implemented through the on-premises data gateway. Note In this scenario, we will use the existing data gateway that was implemented in Chapter 3. In Chapter 3, we created a data gateway with the name testdatagateway on the Azure portal.

Until now, a process has been set up in the workflow for transforming the data received through the http request payload and sending the transformed data back as a response in the XML format. Now, we need to store the transformed data into the SQL Server database residing on the on-premises SQL Server. A data gateway is required to connect to the on-premise SQL Server. As we already have the data gateway, our next task would be to use the data gateway in our logic apps. Understanding SQL Connectors SQL connectors can be used to insert data into the SQL Server database. There are several SQL connectors available in Azure Logic Apps for communicating with the SQL Server. These connectors are either trigger-based or action-based. A list of action-based connectors is shown in Figure 4.41.

Figure 4.41: Displaying the action-based SQL connectors

148 

 Chapter 4: BizTalk in Microsoft Azure Services

Inserting Data into the On-Premise Database As we need to insert data for many individuals, the map output will also have multiple records. Therefore, we need to iterate through each of the records. The For each control in Azure Logic Apps can be used to iterate through the map output. The SQL connector for inserting rows in a table is used inside this map output. The For each control is shown in Figure 4.42.

Figure 4.42: Displaying the For each control

As shown in the above figure, the expression under the Select an output from previous steps text box is “json(xml(body(‘Transform_XML’)))?[‘NS0:Employees’]?[‘Employee’].” This expression takes the Map output (Transform_XML), converts it into JSON, and finally iterates through each “Employee” record in map output (“Employees”). Perform the following steps to insert data into the on-premises database: 1. Navigate to SQL server connectors. 2. Click the SQL Server – Insert row option for inserting a row. The Insert row box appears. 3. Specify the desired connection name in the Connection Name text box. 4. Specify the instance name of the on-premises SQL server in the SQL Server name text box. 5. Enter the desired name for the database in the SQL database name text box. 6. Enter the username that is used to access the SQL server in the Username text box. 7. Enter the password that is used to access the SQL server in the Password text box. 8. Select the Connect via on-premise data gateway checkbox. 9. Select the desired authentication type from the Authentication Type dropdown list.

EAI Implementation Using BizTalk Azure Services 

 149

10. Select the existing gateway from the Gateways drop-down list. 11. Click the Create button to create the connection, as shown in Figure 4.43.

Figure 4.43: Creating a connection

The Insert Row dialog control appears. 12. Select the desired table in which you want to insert data from the Table name drop-down list, as shown in Figure 4.44.

Figure 4.44: Selecting the table

Once the table is selected, provide values for the respective fields in the table. The values reside within the Employee record in the For each control. However, you need to specify some expressions for the values. 13. Type item()[‘TotalCTC’] in the CTC text box. 14. Type item()[‘EmpNo’] in the EmpNo text box.

150 

 Chapter 4: BizTalk in Microsoft Azure Services

15. Type item()[‘EmployeeName’] in the EmpName text box, as shown in Figure 4.45.

Figure 4.45: Specifying expressions

Definition The item() is a logic app function evaluation to the iterating object i.e. each employee in Employees (map output).

16. Click the Save button to save the workflow. The complete workflow is shown in Figure 4.46.

EAI Implementation Using BizTalk Azure Services 

 151

Figure 4.46: Complete workflow

Testing the Complete End-to-End EAI Scenario in Our Extended Workflow To test the complete EAI scenario, we need to use the same URL that was generated after saving the logic app workflow, and the same data in the http request payload. If the workflow executes without error, we should see five rows inserted into the on-premises SQL Server database. First, call the logic app trigger URL through the Advanced REST Client API, which returns a response in XML (map output) with status-code 200 OK (accepted) as expected, as shown in Figure 4.47.

152 

 Chapter 4: BizTalk in Microsoft Azure Services

Figure 4.47: Calling the logic app trigger URL

From the above figure, you can see that the http call has triggered the workflow, which means that the request data is decoded and converted into XML through a map. The map output is sent as a response. Finally, the map output data (Employees) is iterated one by one and each employee’s data is inserted into the next action. In Figure 4.48, you will see the portal where the Azure Logic App shows the current trigger and the execution status of all subsequent actions.

EAI Implementation Using BizTalk Azure Services 

 153

Figure 4.48: Displaying the execution status of all subsequent actions

We can also check the on-premises database (TestDB) and table (Employee) to verify the successful execution of the logic app workflow, as shown in Figure 4.49.

Figure 4.49: Verifying the execution of the logic app workflow

154 

 Chapter 4: BizTalk in Microsoft Azure Services

From the above figure, it can be summarized that the data related to all employees, which is sent through http request, has been transformed and inserted into the on-premise SQL server. It signifies that the logic app workflow has been executed successfully.

Summary Microsoft provides BizTalk Azure Services in two forms, namely, Infrastructure as a Service (IaaS) and Platform as a Service (PaaS). You can use any BizTalk service depending on your requirements. If you do not want to host on-premises physical servers, you can provision BizTalk Server 2016 Azure VMs either for development or for production. You can also set up BizTalk services in the cloud if your requirements are not too large. To set up BizTalk services in the cloud, you will need Azure Logic Apps that act as a replacement for BizTalk Azure services. You can implement the desired option according to your requirement.

Chapter 5 Web 3.0, Custom BizTalk Azure Application Introduction In this chapter, we will look at creating a custom integration solution as an alternative to BizTalk Server, leveraging the next generation of web technologies and microservices, widely known in the software industry as Web 3.0. As discussed in the previous chapter, we can build and implement complete EAI solutions using Microsoft Azure Services, which is a classic example of the innovation and capabilities Azure brings to the software community. In this chapter we will utilize the Web 3.0 service alternatives to build out a typical BizTalk EAI and B2B processing capabilities such as those listed below: –– B2B: Decoding EDI to XML, and decoding flat file to flat file XML –– EAI: Performing XML validation, and executing a set of steps in a workflow

BizTalk Alternatives As we know, BizTalk B2B and EAI processing require three major component groups, which are messaging, transformation, and workflow (orchestration). The incoming message undergoes message processing and is then forwarded to orchestration for business logic or rules execution. Once the message is processed by orchestration; it is delivered to the destination/intended location in the required format. The pipeline processing involves passing the messages through a set of pipeline components. A pipeline component can be either an out-of-box component or a custom component. In fact, the pipeline component is a .NET assembly itself. Therefore, as an alternative to BizTalk Server, you just need to find out the business logic of the component and write a library for it.

Pipeline Processing Alternatives Generally, BizTalk Server processes three forms of inputs which are EDI, XML, and flat file. There are also some other types of messages (inputs). However, they are converted into XML after being parsed using their respective pipeline components. All processing and message transmission in BizTalk Server is accomplished only DOI 10.1515/9781501505652-005

156 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

after the messages are converted into XML. However, if you want to process EDI, XML, or flat file messages without using BizTalk Server, you can use the following alternatives: 1. EDI parsing 2. XML parsing 3. Flat file/CSV parsing 4. MIME parsing In this section, we will discuss EDI parsing only. EDI Parsing In BizTalk Server, EDI parsing is accomplished through the EDI disassembler component. This component is responsible for processing the received EDI interchanges. The EDI disassembler component handles the following processing tasks: –– Splitting or preserving the interchanges –– Splitting or preserving the transaction sets –– Identifying the schema against which individual transaction sets are validated –– Generating XML after validation There are several EDI parsers available that you can use as an alternative to the EDI disassembler component. However, these EDI parsers may or may not execute all of the same tasks as handled by the EDI disassembler. You should always select an EDI parser that performs the basic tasks of EDI parsing, including validating EDI interchanges/transaction sets against the designated schema and generating XML. In this section, we will discuss the following EDI parsers: –– X12Parser –– EDIReader –– BOTS –– EDIFabric X12Parser X12Parser is one of the best EDI parsers used for reading and examining EDI files. It does not require any database integration for parsing the file. It creates a generic EDI XML representation of the envelopes, and the transactional data available in the X12 document. It is an open-source parser. It has built-in specifications for all 4010 version EDI schemas and some of the 5010 version EDI schemas. It takes the EDI document file (full or relative path) as an input, decides the type of EDI schemas to use for validation, and then generates an XML document that is easy to understand.

BizTalk Alternatives 

 157

EDIReader EDIReader is an application developed in Java that follows EDI standards for analyzing business documents. This application can also work in an XML-based system through XML interfaces, including SAX and JAXP. The support for XML interfaces helps in configuring a custom SAX parser. Some of the advantages associated with EDIReader are as follows: –– Simple to use –– Good transformation engine –– Third-party backup support –– Java-based cross-platform In addition to the advantages associated with EDIReader, there are some limitations too. Two major limitations of EDIReader are its incompleteness as a workflow application and the fact that it does not have an active support forum. BOTS BOTS is an open source EDI application and is considered the complete EDI solution. Several features of BOTS are as follows: –– Follows all the workflow rules –– Supports most of the EDI data formats including EDIFACT, XML, and X12, etc. –– Provides transformation capabilities (X12 to XML and vice versa) –– Provides third-party support –– Supports most operating systems including Windows, Linux, OSX, and Unix, etc. –– Provides an active support forum Some limitations associated with BOTS are as follows: –– It has fundamental user interface –– It requires knowledge of Python scripts EDIFabric EDIFabric is one of the best EDI parsers available on the web. Some of the most prominent features of EDIFabric are as follows: –– Supports both EDI 4010 and 5010 versions –– Supports HIPAA, EDIFACT, EANCOM, and VDA other than EDI –– Provides seamless integration –– Supports all EDI standards –– Allows users to read, validate, and write EDI documents –– Generates acknowledgments

158 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Some limitations associated with EDIFabric are as follows: –– The available customer evidence and case studies are limited –– Community discussions and peer support is limited

Transformation Alternatives In this section, you will learn about the transformation of messages without using BizTalk Server. In BizTalk Server, BizTalk Maps uses two types of documents including, source schema, and target schema. However, you can also implement a transform solution in C# that would include steps, as shown in Figure 5.1. Generate a C# class from the source XSD or create a new class Deserialize source file data into an instance of the created class (source object) Generate a C# class from the target XSD or create a new class as per the requirement Create an instance of the class created in the earlier step (target object) Perform object-to-object mapping (transformation) based on mapping requirements/specifications Serialize target object into XML

Send the generated XML as map output Figure 5.1: Steps to implement a transform solution in C#

As most of the data transfer occurs through JSON, the probabilities of receiving the JSON data as the source file are high. In such cases you need to implement a JSON to JSON transformation. JSON schema is like XSDs we use in BizTalk Server. JSON to JSON transformation contains a sequence of steps, as shown in Figure 5.2.

BizTalk Alternatives 

 159

Receive the JSON file and validate it using the JSON schema

Generate a C# class from the source JSON schema

Convert the received data into the JSON object of the created class (source JSON)

Generate a C# class from the target JSON schema

Create an instance of the class created in the earlier step (target object) Perform the object-to-object mapping (transformation) based on the mapping requirements/specifications

Serialize the target object into JSON

Send the generated JSON as the map output Figure 5.2: Steps involved in the JSON to JSON transformation

Orchestration/Workflow Alternatives As discussed in earlier chapters, orchestration is a workflow. You can also create a workflow without using BizTalk Server. The following two options help you in creating a workflow without requiring BizTalk Server: 1. Azure Logic App (an example of Web 3.0 capabilities) 2. Windows Workflow Other Workflow Alternatives There are some workflow engines available on the web that act as alternatives to the BizTalk workflow. These workflow engines add workflow functionality to the application through C# coding.

160 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Workflow Engine .NET The WorkflowEngine.NET is a lightweight component. This component allows you to build workflows. You can use this component as a stand-alone service implemented as a web service. You can also integrate this component with the application completely. This component provides several features. Some of them are as follows: –– Allows you to design a code-free process in a web browser –– Provides high performance –– Allows you to add a custom workflow to the project easily –– Allows you to generate incoming list automatically –– Supports real-time modification in the scheme process –– Supports several database providers including MS SQL Server, MongoDB, RavenDB, Oracle, MySQL, and PostgreSQL, etc. Wexflow: Open Source Workflow Engine Wexflow is an open source workflow engine. It has a cross-platform manager that provides a Graphical User Interface (GUI) for managing workflows. With the help of Wexflow, building automation and workflow processes become easy. Wexflow also helps in making the long-running processes straightforward. The communication between systems or applications becomes easy through this powerful workflow engine. Some of the features of Wexflow are: –– It creates process workflows wherein tasks are executed sequentially. –– It helps in building flowchart workflows. –– It provides three events which are triggered by the final result received once the workflow finishes its job. The OnSuccess event is triggered when the final result is a success. The OnWarning and the OnError events are triggered when the final results are warning and error respectively. –– It allows you to manage the workflow through the Wexflow manager. –– It provides an application called Wexflow Web Designer to design workflows. –– It provides an Android application named Wexflow Android Manager which is used to manage workflows. –– It offers 41 built-in tasks which can be used to perform different types of jobs. –– It allows you to use Wexflow Web Designer for modifying, adding, or deleting the workflow files.

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 161

Note A flowchart workflow has flowchart nodes (If/While/Switch) in its execution graph wherein a flowchart task acts as an input to the flowchart node. The flowchart task gives either true or false as a result of execution. Other flowchart tasks are executed in a sequence. You can change the sequence in which the tasks are executed, by making modifications to the execution graph of the flowchart node.

A Simple Stateless End-to-End Process Implementation Using Alternatives Let us take a simple mapping scenario where a file is received at the configured location (receive location in BizTalk Server). Then, the file is picked/archived from the configured location. It is then decoded and transformed as per the mapping requirements. Finally, it is sent to the destination location. This scenario can be extended later through complex mapping specifications. Since we need automation as in BizTalk Server, we must create a Windows Service with a FileSystemWatcher component. This component fires an event when a file is dropped at its configured location. We can then write the desired code to specify the operations to be performed on the dropped file. Note In this scenario, we will process a file containing order data (Order number, Item name, Quantity, and Unit price). As soon as the file is dropped, it will be taken and transformed into the processed order data, which will be in the XML format. The transformed data (mapping output) will contain an Order number, Item name, and Total price. This implementation looks like the one used in Chapter 4 in the case of Azure Logic Apps. However, this implementation is done without BizTalk Server and Azure Services.

The automated end-to-end implementation process involves the following components: 10. A Windows Service program with a FileSystemWatcher component, a configuration file, a Service installer, and a Service process installer. 11. The “CREATED” event of the FileSystemWatcher component, which picks a file as soon as it is dropped in the receive location. 12. Event-related tasks are: a. Archive a file in the archive location if enabled in the configuration file b. Get the content of the dropped file c. Deserialize the data from the received file into an object (source object for mapping) d. Get an instance of the target object (mapping target)

162 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

e. Perform object to object mapping f. Serialize the target object (after mapping) into XML (output data) g. Archive the output data in the archive location if enabled in the configuration file h. Send the output data in the form of a file to the Send (output) location configured in the configuration file i. Delete the dropped file from the receive location The next section will teach you how to implement this.

Windows Service Perform the following steps to create a Windows Service: 1. Open Visual Studio. 2. Click the New Project option. The Add New Project window opens. 3. Select the Classic Desktop option in the left pane. The related options appear. 4. Select the Windows Service template in the middle pane. 5. Type the desired name for the project in the Name text box. In our case, we have specified OrderProcessing.WindowsService. 6. Click the OK button, as shown in Figure 5.3.

Figure 5.3: Adding Windows Service

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 163

A new project is created with the specified name, and a Windows Service is created with the name “Service1.” You can change the Windows Service name by right-clicking it and selecting the Rename option from the context menu. In our case, we have changed the Windows Service name to OrderProcessingService. As you rename it, the Microsoft Visual Studio pop-up appears asking you to rename all the references of the selected Windows Service. You need to click the Yes button to rename all the references. Once you click the Yes button, all references are renamed, as shown in Figure 5.4.

Figure 5.4: Renaming all references of Windows Service

The FileSystemWatcher Component After creating a Windows Service, add a FileSystemWatcher component in the Service Design window. This component contains four events, which are Changed, Created, Deleted, and Renamed. The FileSystemWatcher keeps an eye on the configured receive location, which means that whenever a file or folder is dropped in the configured location, the respective event is raised. Here, we are using the Created event that fires an event as soon as a file is dropped in the configured receive location. We also need to add an event handler, as shown in Figure 5.5.

Figure 5.5: Adding an event handler

164 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Configuration File Now, we need to add an application configuration file to the project. This file is used by Windows Service to fetch configuration values required in the business logic. By default, a configuration file is added to the project when a Windows Service project is created. However, you can also add a configuration file by clicking the Add New Item option from the context menu, and selecting the Application Configuration File option from the Add New Item window. You can specify the desired name for the configuration file in the Name text box and click the Add button to add the configuration file, as shown in Figure 5.6.

Figure 5.6: Adding an application configuration file

Adding Configuration Values Once the configuration file is added to the project, we need to add configuration values to it. The configuration values are used in the business logic. We can also archive inbound and outbound files in the service. However, it can be made optional by adding configuration values, as shown in Figure 5.7.

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 165

Figure 5.7: Adding configuration values

Configuring FileSystemWatcher Path Next, we need to wire and configure the FileSystemWatcher component to fire its Created event based on the file dropped in the receive location, which is configured in the application configuration file. For this, we need to use the OnStart event of the Windows Service, as shown in Figure 5.8.

Figure 5.8: Using the OnStart event

It allows the FileSystemWatcher component to fire its event when a file is dropped in the path configured as the receive location.

Moving to Business Process As discussed above, we receive a file containing orders and process it. In the processing phase, we receive and archive the file. Then, we transform the file into another document having a different structure like BizTalk Maps. Finally, the transformed data is sent as an output to the send location.

166 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Transform/Mapping To Transform/Map, we need to perform object-to-object mapping. This means that we need to deserialize the received order data into an object (source object) and create an instance of the Target type document object. This requires the class representation of the source schema as well as the target schema. This can be done by creating classes from schemas through the Visual Studio tool named XSD.exe. In our case, we have two schemas, one for received orders and the other for transformed orders. These schemas are used for generating .CS classes using XSD tools. Creating Schemas and Generating Classes As discussed in the previous section, we are creating two schemas for generating .CS classes. The first schema is the flat file Orders schema (Orders.xsd), which is used as a schema for received orders, as shown in Figure 5.9.

Figure 5.9: Creating flat file Orders schema

Another schema is the Orders XML schema, which is used as the processed Orders schema (Orders_XML.xsd), as shown in Figure 5.10.

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 167

Figure 5.10: Creating the Orders XML schema

After creating schemas, we need to generate classes from these schemas. Figure 5.11 shows a class generated from the flat file Orders schema (Orders.xsd).

Figure 5.11: Generating a class from the flat file Orders schema

Figure 5.12 shows a class generated from the processed Orders schema (Orders_ XML.xsd).

Figure 5.12: Generating a class from the processed Orders schema

168 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

We now have two classes that represent two schemas. We must add these classes to our Windows Service project. For better readability, we have added these classes to their respective folder names, as shown in Figure 5.13.

Figure 5.13: Adding classes to their respective folder names

We now have created two classes wherein one represents the source order data, and the other represents the target order data. These classes are used for serialization and deserialization in the mapping process. Transform (Object-to-Object Mapping) To transform the received flat file data into processed orders, we first need to get the received flat file data in the FileSystemWatcher CREATED event. We can get the dropped file name by using the following code snippet: private void FileSystemWatcher_FileDropped (object sender, System.IO.FileSystemEventArgs e) { //Get dropped file Name string sourceFileName = e.FullPath; }

Deserialize Input Data into Source Object and Instantiate Target Object We have written XMLSerializer class for serializing and deserializing XML data into an object and vice versa. This class helps us in deserializing source data into

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 169

a specific type of object and then serialize the output data into an XML string. Figure 5.14 shows the code for the XMLSerializer class.

Figure 5.14: Code for the XMLSerializer class

170 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Transforming Source Data into Target Data For transforming source data into target data, we have created a separate class (OrderTransform). This class performs object-to-object mapping based on mapping requirements. Figure 5.15 shows the code for the OrderTransform class.

Figure 5.15: Code for the OrderTransform class

Serialize Target Object After performing object-to-object mapping, we need to send the transformed data to the send location. However, the output data is in the form of an object. Therefore, we need to serialize the target object for generating XML. For serializing the target object, we need to use the XMLSerializer class. Once the target object is serialized, we get the XML data. This XML data is our output data. Send Output Data As stated earlier, we get the XML data after serializing the target object. Once we get the XML data, we need to send it to the send location. However, we need to archive the output data into the configured location before sending it to the send location. We also need to identify that archiving is enabled in the configuration file.

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 171

Delete Source/Dropped File Once all the steps of file processing are complete, we finally need to delete the dropped file from the configured receive location.

Code Implementation This section outlines the sequence of steps to complete the process outlined in the Transform/Mapping section. These steps are performed when the FileSystemWatcher CREATED event is fired. Figure 5.16 shows the code for the FileSystemWatcher CREATED event handler.

Figure 5.16: Code for the FileSystemWatcher CREATED event handler

Figure 5.17 shows the code for the archive source file.

Figure 5.17: Code for archive source file

172 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

After archiving, the central business logic starts. It includes several tasks, such as getting received file data, deserializing it into flat file orders object, performing the transformation (object-to-object mapping), serializing target data object (received after transformation process) to generate XML, and finally sending the generated output to the send location. Figure 5.18 shows the code for all these tasks.

Figure 5.18: Code for main business logic

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 173

Building the Solution Once the coding is complete, we can rebuild the solution. To do so, navigate to the build location after rebuilding the solution. Here, we will see multiple files. However, we only need the OrderProcessing.WindowsService.exe and the OrderProcessing.WindowsService.exe.config files. The OrderProcessing.WindowsService.exe file is required to install OrderProcessingService as a Windows Service. The OrderProcessing.WindowsService.exe.config file is used by the order processing service as a configuration file, as shown in Figure 5.19.

Figure 5.19: Displaying the required files

Installing Order Processing Service As discussed earlier, the OrderProcessing.WindowsService.exe file is used to install OrderProcessingService as a Windows Service. To install this Windows Service, we need to use the installutil command followed by the complete path of the EXE file, as shown in Figure 5.20.

174 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.20: Installing the OrderProcessingService as a Windows Service

Verifying the Installation Once the Service is installed successfully, we can verify it in the Windows Services list. Locate the Service “Order Processing Service” and double-click it. The Order Processing Service Properties (Local Computer) dialog box opens and displays the name of the service and its executable path that should point to the EXE location, as shown in Figure 5.21.

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 175

Figure 5.21: Verifying the installation

Testing the Order Processing Service After installing the Service and verifying it in the Windows Services list, we need to test it. To do this, start the Windows Service. As soon as the Windows Service is started, an event will be logged with the receive location path, as shown in Figure 5.22.

Figure 5.22: Testing the Order Processing Service

176 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Note The receive location path is retrieved from the Service configuration file.

Business Process Testing To test the business process implementation, we need to drop a file into the receive location, just as we did in BizTalk Server. For this scenario, we are using an XML flat file that contains order data. Figure 5.23 shows the sample XML flat file.

Figure 5.23: Sample XML flat file

We need to drop this XML data file into the receive location (“E:\Order Processing Service\ReceiveOrder”), which is configured in the configuration file. Once the file is dropped, we need to check both of the configured locations. We must review the archive location and the output location because archiving is enabled in the configuration file. Figure 5.24 shows the ArchiveReceivedData location.

A Simple Stateless End-to-End Process Implementation Using Alternatives 

 177

Figure 5.24: The ArchiveReceivedData location

Figure 5.25 shows the processed data archive location (ArchiveProcessedOrder).

Figure 5.25: The ArchiveProcessedOrder location

Figure 5.26 shows the output location (ProcessedOrder):

Figure 5.26: The ProcessedOrder location

From Figure 5.24–5.26 it can be analyzed that a file is received, and the output file is generated and archived at the configured locations. Examine Output File Now we can check the output file generated at the send location, as shown in Figure 5.27.

178 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.27: Examining the output file

In Figure 5.27, we can see that the output file is a transformed file. The OrderNum and ItemName fields are mapped directly from the source file. The TotalOrderAmount field contains the values received by multiplying the values in the Quantity and the UnitPrice fields of the source file.

Extending the Scenario (EDI Processing and Persistence) At this point, we have implemented a business scenario where a file is received, processed/transformed, and the output is sent to the configured send location. Next, we will extend this business scenario. As demonstrated, we have implemented XML to XML mapping. Now, we can add functionalities to it such that a flat file data is received, converted into XML, and then processed. We can also extend our business scenario further, in such a way that an EDI file is received, parsed to generate XML, and mapping is applied to it. Note As discussed earlier, there are many EDI parsers available on the web. Here, we are using the EDIFabric parser for parsing the EDI files. This parser provides an open-source SDK, which can be downloaded from https://github.com/EdiFabric/Sdk.

Extending the Scenario (EDI Processing and Persistence) 

 179

In addition to the EDI processing, you will learn about adding persistence logic like BizTalk Server, where we will store basic message details that help in identifying the following in case of errors: –– What is the cause of the message error? –– Which file has the message or transaction error? –– How many transactions does a file have when it reaches the system?

EDI Processing In EDI processing, we need a program (EDI parser) that can validate EDI and then disassemble it so that all transactions in the EDI file can be processed individually. The program should also reassemble it while sending an output. As mentioned earlier, we will use the EDIFabric parser for these tasks. Configure Processing Type As we are implementing an EDI scenario, we need to use a configuration key and specify the processing type required, as shown in Figure 5.28.

Figure 5.28: Specifying the processing type

Adding References to EDIFabric in the Project To add references to EDIFabric in the project, we first need to download EDIFabric open-source SDK from GitHub at https://github.com/EdiFabric/Sdk. Once it is downloaded, we need to add basic assemblies to our project, as shown in Figure 5.29.

180 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.29: Adding basic assemblies to the project

As we are implementing the EDI processing scenario, we do not require the assemblies related to EDIFACT or VDA protocols. In the current scenario, we need the following assemblies: 1. EdiFabric.Core.dll 2. EdiFabric.Framework.dll 3. EdiFabric.Rules.Hipaa005010.dll (for EDI 5010-version) 4. EdiFabric.Rules.X12004010.dll (for EDI 4010-version) Did You Know? The name of the assembly indicates its purpose.

Persistence Scenario In persistence implementation, we need the following: 1. Received file details 2. Received file data 3. Transactions (messages) data 4. Outbound messages (processed messages ready to be delivered) data 5. Outbound file information 6. Information related to failed messages in case of an error/exception

Extending the Scenario (EDI Processing and Persistence) 

 181

Implementation In this section, we discuss the step-by-step implementation of EDI processing and persistence logic. Receive and Validate EDI As discussed earlier, we used the EDIFabric parser for parsing EDI data. In addition, we used X12Reader for reading EDI files that need to pass EDI data in the form of a stream to X12Reader, as shown in Figure 5.30.

Figure 5.30: Using X12Reader

This process loads an assembly based on the EDI version, as shown in Figure 5.31.

Figure 5.31: Loading an assembly

Based on the EDI version, there are two assemblies, which are X12 and HIPAA. These assemblies contain the rules for EDI documents. HIPAA rules assembly is

182 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

used for EDI version 5010 validation, while X12 rules assembly is used for EDI version 4010 validation. After successful validation, EDIReader returns five or more items in the following sequence: 1. ISA Segment values (ISA object) 2. GS Segment values (GS object) 3. Transaction 1 object 4. Transaction 2 object 5. Transaction 3 object and so on 6. GE Segment values (GE object) 7. IEA Segment values (IEA object) For example, the Sample_834 file has two transaction sets, as shown in Figure 5.32.

Figure 5.32: A Sample_834 file

Extending the Scenario (EDI Processing and Persistence) 

 183

As we can see in Figure 5.32, there are two transaction sets. These transaction sets contain file contents in the following sequence: 1. ISA segment 2. GS segment 3. ST-SE Segment (transaction sets) 4. GE Segment 5. IEA segment We can parse this file through EDI Parser (EDIFabric) in debug mode. Figure 5.33 shows the output after parsing the file through EDI Parser.

Figure 5.33: Parsing the file through EDI Parser

Referring to Figure 5.33, you can see that the EDI Parser returns six items in the same sequence the EDI file contents are arranged. This means that we have two transaction data sets in the form of TS834 objects, and the other objects are the ISA, GS, GE, and IEA segment data. Till this point, we have converted EDIReader items into a custom object called ReceivedFileData, which contains all transactions data in the form of an object list, as shown in Figure 5.34.

Figure 5.34: Transactions data in the form of object list

184 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Process Each Transaction and Save Received File Data Enabling Multiple Type Documents Processing Since we are extending the business scenario discussed earlier, we have made few changes in the FileSystemWatcher file Created event. Now, this event will instantiate a different type of object based on the processing type configured in the configuration file. A different type of class is initiated based on the processing type, including EDI processing, Flat file processing, EDIFACT processing, or XML processing. All classes follow a contract (interface), as shown in Figure 5.35.

Figure 5.35: A contract (interface) followed by classes

All other file processing classes inherit this contract. Figure 5.36 shows our EDI processing scenario inheriting the contract (interface).

Figure 5.36: Inheriting the contract

Extending the Scenario (EDI Processing and Persistence) 

 185

Figure 5.37 shows the code snippet for instantiating the respective class based on the selected processing type.

Figure 5.37: Code snippet for instantiating the respective class

Figure 5.38 shows the code snippet to get the file processing service.

186 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.38: Code snippet for getting the file processing service

Database Operations Based on the information available in the earlier section, it can be concluded that EDIReader (EdiFabric) has parsed the received EDI file into a list of different objects. The object information can be used to store file-related data. For received data: We have created the following three tables for storing different types of data: 1. Table: [ReceivedFileData]: This table stores the inbound file details, as shown in Figure 5.39.

Figure 5.39: The ReceivedFileData table

2. Table: [MessageData]: This table stores each transaction data in the received file, as shown in Figure 5.40.

Extending the Scenario (EDI Processing and Persistence) 

 187

Figure 5.40: The MessageData table

3. Table: [ReceivedEDIDetails]: This table stores interchange data when the received file is the EDI file, as shown in Figure 5.41.

Figure 5.41: The ReceivedEDIDetails table

For outbound data: 4. Table: [OutboundMessageData]: This table stores the processed messages that are routed to the output location, as shown in Figure 5.42.

Figure 5.42: The OutboundMessageData table

188 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

For exception data: 5. Table: [ExceptionData]: This table stores details about the messages that have errors, as shown in Figure 5.43.

Figure 5.43: The ExceptionData table

Saving Received File Data The ReceivedFileData table schema information is available in the configuration file. This file contains the received file properties. You need to call the DBOperations method to save the received file data, as shown in Figure 5.44.

Figure 5.44: Saving the received file

Extending the Scenario (EDI Processing and Persistence) 

 189

Saving EDI Interchange Information EDI interchange information can only be saved when the dropped files are in EDI format. To save EDI interchange information, we need ISA segment details available in EDI items. These EDI items are returned by the EDI reader after parsing the received EDI file. We need to call the DBOperations method for saving EDI message data, as shown in Figure 5.45.

Figure 5.45: Saving EDI interchange information

Processing Transactions Individually Now, we need to iterate through each transaction separately. We have two transactions in our file, as follows: –– Saving each transaction data in the database –– Applying transform (mapping) Figure 5.46 shows both the transactions.

190 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.46: Displaying both transactions

Saving Each Transaction Data As shown in Figure 5.46, each transaction is iterated and processed individually. Figure 5.47 shows how each separate transaction is saved.

Figure 5.47: Saving each separate transaction

Extending the Scenario (EDI Processing and Persistence) 

 191

Transform For transform/mapping, we will use 834 to 834 mapping because we want to process EDI and get the output as EDI. The 834 to 834 mapping is used so that map output can be converted back into EDI after output validation. For this, we need to project the transaction object into 834 object and create a new 834 object, as shown in Figure 5.48.

Figure 5.48: Creating a new 834 object

After creating a new 834 object, we need to map and serialize the map output into XML, as shown in Figure 5.49.

Figure 5.49: Mapping and serializing the map output into XML

Create Outbound Message Data Once all the individual transactions are processed, a list of output messages to the FileSystemWatcher Created event handler is received, as shown in Figure 5.50.

Figure 5.50: A list of output messages

Here, we will create outbound message data that consists of send and archive location details and output messages. It returns a list of output messages in the FileSystemWatcher Created event handler, as shown in Figure 5.51.

192 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.51: Creating outbound message data

Sending Processed Messages (Output) to the Output Location After creating outbound message data, we need to convert it based on the processing type (EDI), send it to the output location, and finally archive it (if enabled), as shown in Figure 5.52.

Figure 5.52: Converting output into EDI format

Creating EDI from XML (Processed Message Output) To create EDI from XML data, we need to use the EDIReader (EDIFabric) class. We can pass ISA segment and GS segment and write them into the stream using the X12writer class. Figure 5.53 shows the code snippet to create the ISA segment for outbound EDI.

Extending the Scenario (EDI Processing and Persistence) 

Figure 5.53: Creating ISA segment for outbound EDI

Figure 5.54 shows the code snippet to create GS segment for outbound EDI.

 193

194 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.54: Creating GS segment for outbound EDI

Creating an Outgoing EDI File After creating the ISA segment and the GS segment for outbound EDI, we need to create an outgoing EDI file wherein we add the ISA segment, GS segment, and transaction data. Then, we need to load the string from the memory stream, as shown in Figure 5.55.

Figure 5.55: Creating an outgoing EDI file

Extending the Scenario (EDI Processing and Persistence) 

 195

Sending and Archiving Outbound Message As we have the outbound message in the required EDI format, we can send it to the configured location and archive it (if enabled in the configuration file), as shown in Figure 5.56.

Figure 5.56: Sending and archiving outbound message

Saving Outbound Message Data Now, we can save the outbound message data to keep track of each message. The following information related to each outbound message is saved: –– Message ID (outbound message) –– Source message ID –– Main message ID (message ID assigned to the dropped file, i.e., whole message) –– Interchange details Figure 5.57 shows the code to save outbound message data

196 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.57: Code to save outbound message data

Testing the Extended Scenario Since we have updated the existing service, we need to reinstall it to affect all the changes made to it. After reinstalling the service, we can update the configuration file, as shown in Figure 5.58.

Figure 5.58: Updating the configuration file

Extending the Scenario (EDI Processing and Persistence) 

 197

Dropping the File in the Receive Location Next, drop a test file in the configured receive location. In this case, we have dropped the same file (Sample_834) that has two transaction sets of 834 type, as shown in Figure 5.59.

Figure 5.59: Dropping a test file in the configured receive location

As discussed earlier, the dropped file has two transaction sets. The testing phase works the same way it worked in BizTalk Server. Therefore, the file should split into two different transaction files, process individually, and produce two files in the output location. Checking the Output Location Figure 5.60 shows the output location (the ProcessedOrder folder).

Figure 5.60: Checking the output location

From Figure 5.60, we can see that two files are produced in the output location wherein each file name has GUID associated with it. The GUID is added to the output file name through coding. Checking Data in the Database Figure 5.61 shows the received file data in the database.

198 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.61: Received file data

Figure 5.62 shows the received EDI details in the database.

Figure 5.62: Received EDI details

Figure 5.63 shows individual transaction data from the received file.

Figure 5.63: Individual transaction data

Figure 5.64 shows the outbound message data related to the outbound message ID, correlated source message ID, source file message ID, and outbound location.

Figure 5.64: Outbound message data

Figure 5.65 shows the outbound message data related the outbound message ID, message data, sender ID, receiver ID, interchange date, interchange time, and interchange control number.

Extending the Scenario (EDI Processing and Persistence) 

 199

Figure 5.65: Outbound message data

From the above figures, we can see that the EDI file has been processed successfully and the related data is saved in the database. We can also see that the source message ID and individual message ID could be used at either receive stage or send stage to correlate the information related to file processing.

Extending a Bit More: Generate ACK In our examples thus far, we have set up an EDI end-to-end scenario (receivevalidatetransformsend). Next, we will generate acknowledgments (ACK) for received EDI without using BizTalk. Earlier, we used EDIFabric for EDI validation and XML to EDI transform/mapping. Now we will use EDIFabric for generating acknowledgments. The EDIFabric component allows you to configure ACK settings, as shown in Figure 5.66.

Figure 5.66: Configuring ACK settings

200 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Now, we can use the Acknowledgement manager to generate ACK, as shown in Figure 5.67.

Figure 5.67: Generating ACK through the Acknowledgement manager

Next, we need to build ACK into the readable string, as shown in Figure 5.68.

Figure 5.68: Generating ACK as a readable string

Adding an Entry for ACK Location After building ACK into the readable string, we need to add an entry for ACK location in the configuration file, as shown in Figure 5.69.

Extending the Scenario (EDI Processing and Persistence) 

 201

Figure 5.69: Adding an entry for ACK location in the configuration file

Saving Generated ACK in the Configured Location After adding an entry for ACK location in the configuration file, the next task is to save the generated ACK in the configured location, as shown in Figure 5.70.

Figure 5.70: Saving generated ACK in the configured location

Testing the ACK Generation For generating ACK, we used the same file (Sample_834) that was used in the previous implementation. As we drop the file into the configured location and look in the ACKs folder (configured for storing ACK files), we see the generated file, as shown in Figure 5.71.

Figure 5.71: Testing the ACK generation

You need to double-click the file to open it and read its contents, as shown in Figure 5.72.

202 

 Chapter 5: Web 3.0, Custom BizTalk Azure Application

Figure 5.72: Examining the file content

Figure 5.72 shows that the source file had two transactions and both transactions were accepted.

Summary In this chapter, you have learned about building an end-to-end solution similar to BizTalk Server using Web 3.0 services while removing the dependency on BizTalk Server. We have used C# code to implement the equivalent of BizTalk Server processes. We have also used parser components by downloading them from the internet. The result is a complete EDI end-to-end solution through parsers available on the web, and C#.Net code—a true Web 3.0 solution.

Chapter 6 Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration Introduction In the previous chapter, we discussed the features of BizTalk Server 2016, BizTalk Azure Services, and BizTalk alternatives (custom integration through .NET). In this chapter, we focus on the comparisons between these three integration models and provide information that will help you to select the appropriate option for your real-world scenario.

Key Comparison Pivots The parameters we use for comparison include the following features. Feature Comparison: 1. Application integration capabilities 2. Support for long-running transactions 3. Dynamic message processing and routing 4. Business rules execution 5. Workflow support 6. End to end transaction processing 7. Tracking system 8. EDI business process 9. (Other) message protocol support 10. Transport protocol and line of business connectivity Enterprise Readiness Comparison: 1. Developer experience 2. Scalability 3. Maintainability 4. Ease of implementation 5. Cost of ownership

DOI 10.1515/9781501505652-006

204 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

In the following sections, we elaborate on each of the above key features and enterprise readiness pivots and assist with selecting the most viable solution for your scenario.

Application Integration Capabilities BizTalk IaaS/BizTalk Server You are already familiar with the enterprise application integration (EAI) capabilities within BizTalk Server. The previous chapter walked you through this complete process. As you know, EAI implies connecting two or more diverse systems. These systems can be new or existing, and they can be located within the same enterprise or in different enterprises. BizTalk Server provides a reliable EAI solution through its multi-protocol messaging, workflow engine, tracking, and reporting capabilities. BizTalk PaaS BizTalk PaaS services support Azure Logic Apps that act as a native tool for implementing the EAI solution. BizTalk PaaS also provides several types of connectors used to expand the functionalities of applications running on cloud or onpremise. Like BizTalk Orchestration Engine, BizTalk PaaS provides a workflow engine that allows you to draw the desired set of actions and triggers. Custom Integration With a custom integration, we need to build connectors for each application to which we want to connect. We also need to set up a common communication mode. Accordingly, we need to design the solution to parse the messages coming from other systems. These messages are then processed and transformed back into the format accepted by the system that receives the messages. In simple words, we need to build everything from scratch. However, we can use the existing features of C# to design our application as per our requirements.

Long-Running Transactions BizTalk IaaS/BizTalk Server BizTalk Server supports long-running transactions. Long-running transactions are used when a transaction needs to run for a long time. They do not require complete ACID properties. In other words, these transactions satisfy only the consistency (C) and durability (D) properties. BizTalk Server supports long-running

Long-Running Transactions 

 205

transactions through the Orchestration Scope shape in BizTalk Orchestration Engine, as shown in Figure 6.1.

Figure 6.1: Illustration of transaction types

Several features of long-running transactions are listed as follows: –– Nesting: A long-running transaction can contain two or more long-running transactions, wherein each long-running transaction may possess atomic transactions. –– Commit: The transaction commit and transaction rollback in a long-running transaction should be done by the code itself because there is no specific transaction manager in long-running transactions. –– Rollback/Compensate: Compensation is similar to transaction rollback and is performed after transaction commit. You can use a compensation shape when you need transaction rollback. However, you should note that the compensation block for any scope can only be used when scope’s Transaction Type property is either Atomic or Long Running, as shown in Figure 6.2.

206 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

Figure 6.2: Clicking the New Compensation Block option

From the information mentioned above, we can say that BizTalk Server supports long-running transactions out of the box. BizTalk PaaS BizTalk PaaS services do not have built-in capability for long-running transactions. As discussed in the previous chapter, Azure Logic Apps work on REST-based architecture. This means that Azure Logic Apps use REST constraints. These constraints allow Azure Logic Apps to timeout the connection before the completion of the long-running processes. By default, Azure Logic Apps wait for one minute, and then the request is timed out. However, Azure Logic Apps can be designed in such a way that they support long-running processes. To do so: –– Send a response 202 (Accepted) as soon as a request is received for triggering Azure Logic App workflow. –– Check whether a transaction is complete or not when a job status is checked. If the transaction is completed, then return a 200 (OK) response, otherwise, send another 202 (Accepted) response. Custom Integration With custom integration, we can design the solution through coding using the same approach mentioned in the BizTalk PaaS section. We can also implement the same approach using transaction scope.

Dynamic Message Processing and Routing 

 207

Dynamic Message Processing and Routing BizTalk IaaS/BizTalk Server BizTalk Server provides an Enterprise Service Bus (ESB) Toolkit, which is used in combination with Business Rules Engine (BRE) policies to support dynamic message processing. The dynamic message processing can be related to dynamic message routing and dynamic message transformation. For example, ESB and BRE policies can be used collectively to design a solution where a message can be routed to different locations, and different maps can be selected based on the message type. A single message can be routed to multiple recipients through the ESB itinerary. ESB Toolkit works on the resolver connection strings for resolving maps and endpoints for received messages. These connection strings can be attached with a message in the pipeline, or they can be added to the SOAP header of the messages. The Resolver and Adapter Provider Framework supports itinerary, transformation, endpoint resolution, and routing of messages. The framework allows you to dynamically resolve endpoints and set outbound adapter properties. Primarily, a resolver component performs the endpoint resolution, for example the Universal Description, Discovery, and the Integration (UDDI) component helps to look up an outbound Web service endpoint. Then, an adapter provider component is used to set the properties of the registered Microsoft BizTalk Server adapters. For example, the WCF-BasicHttp adapter provider is used to set the BizTalk-specific message context properties for the endpoint URI to use the appropriate BizTalk adapter. In the same way, the FTP adapter provider is used to set the properties that are specific to the FTP adapter. BizTalk ESB Toolkit configuration contains details of all the registered resolvers and adapter providers. At the execution time, the resolver managers and adapter managers read details of the registered resolvers and adapter providers respectively from the configuration files, load the appropriate assemblies, and store them in a BizTalk host-level cache. This caching technique eliminates the need for repeated tasks for reading configuration files and loading assemblies for each submitted message. BizTalk PaaS BizTalk PaaS provides the ability to implement itinerary-based message routing with the help of Azure Logic Apps and service bus actions. For example, the service bus actions feature can be used to define the routing of the messages to the next service endpoint. Alternatively, we can set properties to match them with

208 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

the next subscriber filter. When the property’s values satisfy the filter condition, a service bus action is executed. Custom Integration With custom integration, we can use the existing capabilities of C#. We can write and design the solution in such a way that all the operations supported by ESB for dynamic message processing can be executed. This requires significant work.

Business Rules Execution BizTalk IaaS/BizTalk Server Business Rules Engine (BRE) is used for executing business rules in any application. Microsoft provides the .NET class library for executing these rules, which can be created by a local tool called the Microsoft Business Rule Composer, as shown in Figure 6.3.

Figure 6.3: Microsoft Business Rule Composer

BRE provides an efficient solution for implementing declarative and readable rules to an XML document or business object. It helps in implementing the straightforward design, reusing code, and using business modules. BizTalk Server allows users to implement BRE policies, which can be used in BizTalk Orchestration Engine by using the Call Rules shape, as shown in Figure 6.4.

Business Rules Execution 

 209

Figure 6.4: Implementing a BRE policy

BizTalk PaaS Unlike Microsoft Business Rule Composer, BizTalk PaaS does not have rules. Previously, Azure BizTalk Rules Service could be used, which supported rules and policies. However, it has been deprecated and removed from the Azure portal. Its documentation has also been removed from the portal. However, as an alternative to Azure BizTalk Rules Service, we can use the on-premise BizTalk BRE policies and access them in our Azure Logic Apps through BizTalk Management APIs. After configuration, we can use Azure functions as rules engines. Azure functions act as a good option for running small code snippets in the cloud. They can also be edited and deployed on the portal. Custom Integration With custom integration, we can create our own set of rules in the database and retrieve them from the database while using them in our code. Also, there are several open source and commercial rules engines available on the web. Note The CodeEffects rules engine provides .Net compliant business rules. Visit the following link for more information: http://codeeffects.com/Business-Rule-Demo

210 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

Workflow Support BizTalk IaaS/BizTalk Server The support for Business Process Workflow is available as orchestrations in BizTalk Server and is managed by BizTalk Orchestration Engine runtime. This BizTalk Orchestration Engine is a state machine. It monitors the entire lifecycle of the workflow, which includes: –– Instantiating the workflow –– Executing the workflow –– Scheduling tasks in the workflow –– Receiving and sending the messages –– Monitoring the interaction with external systems or a local system BizTalk Orchestration Engine is tied to the BizTalk MessageBox database, which helps Orchestration Engine maintain its state at different stages. In other words, BizTalk Orchestration Engine state persists irrespective of transactions (long-running or small) or failure at any stage (information/message is guaranteed not to be lost at any stage). If a failure occurs, the issue can be resolved, and the same instance can be reactivated. Re-initiating the process for the same message is not required. These capabilities of BizTalk Orchestration Engine make it an idle workflow engine. BizTalk PaaS The Azure Logic App Service within BizTalk PaaS services is a workflow-based service provided on the Azure portal. The connectors associated with it make the integration easy. You can implement an end-to-end document processing solution within hours. Custom Integration With custom integration, we can leverage Windows Workflow Foundation (WWF). WWF enables a developer to write a workflow-enabled service. It allows developers to create custom activities, which can be a .Net component or a third-party library. Like BizTalk Orchestration Engine, WWF allows developers to continue the state of the workflow and serialize and save it to the local disk through its in-built provider model. This provider model is used to save the workflow state to the SQL server database to attain its persistence capabilities. However, this solution would need to be hosted on a Windows Server Compute node.

End-to-End Transactional Support 

 211

End-to-End Transactional Support BizTalk IaaS/BizTalk Server BizTalk Server provides complete transactional support through the workflow engine (BizTalk Orchestration Engine). BizTalk Orchestration Engine allows you to design long-running processes. In a design scenario, we can implement a process wherein either all the steps are executed, or none are executed. This means we would implement a process where an error-occurring step would cause it to revert all the previous steps/actions that were executed. To achieve this task in the context of programming, we would need to group all related steps/actions into a single process. This can be done easily in BizTalk Server by implementing transactions, which could be a long-running business process or timed transaction. Therefore, we can design a transaction that is either executed completely or not at all. For transactional support in BizTalk Server, we can use a Scope shape (in BizTalk Orchestration Engine) and set the Transaction Type property to Atomic, Long Running, or None based on the requirement, as shown in Figure 6.5.

Figure 6.5: Setting the transaction type

BizTalk PaaS BizTalk PaaS services provide stateless transactional support. However, to create state-oriented transactional support, we need to design the solution ourselves. Azure Logic Apps provide a control named “Scope.” Scope allows you to group a set of actions together. This implementation works best for exception handling, as shown in Figure 6.6.

212 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

Figure 6.6: Illustration of Scope control

Custom Integration With custom integration, C# provides its own solution for transactional support. Similar to every large-scale development framework, C# also provides a built-in class for transactional management. The name of the built-in class is “TransactionScope.” This class can be used to manage local as well as distributed transactions. This class is available in the System.Transactions namespace and can be used as shown in Figure 6.7.

Figure 6.7: Using the System.Transactions namespace

Tracking System BizTalk IaaS/BizTalk Server As discussed in earlier chapters, BizTalk Server keeps track of each message and process running in its domain through its MessageBox database. In addition, there is a specific feature in BizTalk Server, called Business Activity Monitoring (BAM). BAM monitors and generates alerts. It tracks almost every artifact created in BizTalk Server. The ability of BAM to track artifacts allows users to analyze and monitor data that is processed through BizTalk Server.

Tracking System 

 213

BAM is an ASP.Net website hosted on IIS and can be configured through the BizTalk configuration wizard. It provides a real-time report of the business processes that are either completed or being executed. BAM uses the BAMPrimaryImport database as its primary source on the BAM portal. BizTalk PaaS In BizTalk PaaS services, there is an Azure Diagnostics Tool. This tool is used for monitoring or tracking the business process data involved in any Azure BizTalk Service. It also tracks Azure Logic Apps workflow events. To view performance related metrics of Azure Logic Apps, use the Metrics tab under the Monitoring section, as shown in Figure 6.8.

Figure 6.8: Selecting the Metrics tab

You will see multiple filters for metrics, as shown in Figure 6.9.

214 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

Figure 6.9: Representation of filters

Similarly, you can use alert rules and diagnostics logs. In the case of an integration account, you can get data related to integration account tracking events by clicking the “Turn on diagnostics” link, as shown in Figure 6.10.

Figure 6.10: Clicking the Turn on diagnostics link

From the above discussions, you can see that tracking options are available for BizTalk PaaS services. Custom Integration With custom integration, we can build our own tracking system. In the previous chapter, we implemented an end-to-end scenario with persistence support.

EDI Business Processes 

 215

Similarly, we can create our own portal that displays reports based on filters and queries data from our tracking database.

EDI Business Processes BizTalk IaaS/BizTalk Server BizTalk Server supports processing of EDI documents and provides several EDI specific features which are used to process EDI messages. It supports all types of EDI documents related to Retail Commerce, Shipping, Health Insurance, Product, and Payment. It processes all EDI documents with ease and ensures accurate EDI protocol specific configuration. In BizTalk Server, the processing of EDI messaging includes: –– Receive side EDI processing –– EDI schemas (X12, EDIFACT, EANCOM, etc.) –– Agreements –– Send side EDI processing BizTalk PaaS Although BizTalk PaaS does not provide any specialized tool for processing EDI messages, it can still process EDI messages. As discussed in earlier chapters, we can use Azure BizTalk Services and integration accounts to store artifacts, such as EDI schemas and Maps, related to EDI processing. BizTalk PaaS provides different types of connectors that can be used by Azure Logic Apps to validate and disassemble an EDI document. Custom Integration With custom integration, we can use the helpful EDI parsers that are available on the web. These EDI parsers provide SDKs, which can be used to implement an EDI processing scenario.

Message Protocol Support BizTalk IaaS/BizTalk Server In addition to EDI, BizTalk Server supports different types of documents including RosettaNet, HL7, and SWIFT. These types of documents are called messaging formats. These documents are used for exchanging data electronically at the enterprise level. BizTalk Server also provides accelerators for these formats.

216 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

BizTalk PaaS BizTalk PaaS supports all message formats that are supported by BizTalk Server. These message formats are supported with the help of the Enterprise Integration Pack (EIP) within Azure Logic Apps. Custom Integration With custom integration, third-party libraries are available for different message formats including EDI, EDIFACT, VDA, and XML. In the previous chapter, you were introduced to the process of implementing libraries using the EDIFabric parser. You can look for other parsers on the web for remaining message formats. A significant amount of effort is required to stitch the solution together.

Protocol and Line of Business Connectors BizTalk IaaS/BizTalk Server BizTalk Server is enriched with several types of connectors (or adapters), which makes it the most powerful and the best integration platform. BizTalk Server supports several adapters for common transport standards including the following: –– FTP –– SFTP –– MSMQ –– SMTP –– SOAP –– WCF In addition to the above-listed adapters, BizTalk Server supports several line-ofbusiness (LOB) or application-specific adapters. A few of the most popular adapters are: –– SharePoint –– Oracle –– SAP Note Visit the following link to understand the SharePoint Services adapter, its features, and its working: https://docs.microsoft.com/en-us/biztalk/core/what-is-the-windows-sharepoint-servicesadapter

Developer Experience 

 217

BizTalk PaaS BizTalk PaaS connectors/adapters are an essential part of the BizTalk PaaS services. These connectors allow applications running on-premise or on cloud to connect with other on-premise applications. The communication between applications and resources happens through data gateways. All the PaaS connectors are used through Azure Logic Apps, where the workflow can be defined, and connectors can be used based on the application requirement. Like BizTalk Server, Azure Logic Apps supports different connectors for different applications. For example, you can connect your Azure Logic App to: –– Outlook –– Salesforce –– CRM online –– SharePoint –– Twitter –– DB2 –– Oracle You can also connect to the on-premises BizTalk Server from Azure Logic Apps. It also supports a specific BizTalk connector. Custom Integration In custom integration, there are no built-in connectors. However, you can build your own connectors. The functions of these connectors may be the same or different from other connectors available on different platforms. These adapters or connectors can be developed in native C# itself.

Developer Experience BizTalk IaaS/BizTalk Server BizTalk Server provides SDKs, which can be used in the Visual Studio environment to create BizTalk projects. These SDKs provide a good environment for development and testing. A BizTalk process, “btsntsvc.exe,” can be added to BizTalk Server to debug most of the artifacts from Visual Studio itself. In addition, maps can be debugged independently in the development mode, that is,before deployment as well as after deployment. You may also deploy the solution/artifacts directly to the BizTalk solution from within Visual Studio.

218 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

BizTalk PaaS BizTalk PaaS can be developed on the Azure portal. Additionally, the artifacts can also be developed in Visual Studio after the introduction of the integration account SDKs and Azure Logic Apps tools for Visual Studio. Then, Azure Logic Apps can be deployed on the Azure portal from Visual Studio. Custom Integration With custom integration, development can be done through Visual Studio. It supports several languages. It is platform independent, which means that you do not need to switch to other IDEs for development purposes.

Scalability BizTalk IaaS/BizTalk Server Throughput is considered the most important parameter while designing an application or integration solution and infrastructure. However, there could be a scenario where throughput (message processing in BizTalk Server) increases beyond its capability. In such a situation, we need to scale the BizTalk system. Scaling can be done either by adding extra hardware (servers) in a scale-out buildout or by upgrading the existing hardware and scaling up. This can be done through BizTalk Server architecture. The BizTalk Server architecture helps in scaling up and scaling out any BizTalk Server setup. In the process of scaling the BizTalk Server setup, you can scale out: –– BizTalk Server –– SQL Server –– Receiving hosts –– Processing hosts –– Sending hosts –– Scaled-out databases You need to select the applicable option depending on the increase in throughput and scenario that you are trying to implement. Note Visit the following link to look at a few considerations that define the type of scale-out suitable for different situations: https://docs.microsoft.com/en-us/biztalk/core/what-is-scalability

Scalability 

 219

BizTalk PaaS The scalability of BizTalk PaaS can be defined easily by comparing it to that of BizTalk IaaS (servers). Azure BizTalk Services are offered in five tiers, namely, Free, Developer, Basic, Standard, and Premium. Depending on the selection of tier, the scalability can be switched easily. Figure 6.11 shows different tiers.

Figure 6.11:  Different tiers of Azure BizTalk Services

Note Information on BizTalk Azure Services is available at the following link: https://azure.microsoft.com/en-in/pricing/details/biztalk-services/

Custom Integration Scalability in custom integration primarily depends on the type of integration solution being designed/implemented. There are several important considerations for custom integrations, some of which are mentioned here: –– Type—whether stateless implementation or implementation with persistence capabilities –– Memory size required by executables to run –– Database size based on the implementation –– Throughput that could be handled by the application Once the above items have been considered, scaling is effortless, as the developer holds complete control.

220 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

Maintainability BizTalk IaaS/BizTalk Server Maintenance is one of the most important aspects of the development lifecycle. Let us take a scenario where a partner document delivery connection is broken in the BizTalk environment, which means we are unable to receive files for the day. We will need support to resolve this problem. If there is no production support available, we need to hire a person with specific capabilities, which may take days. The hired person will also require adequate time to learn and understand the environment. If the problem remains unresolved, it impacts the overall productivity and performance of the organization. In terms of cost, you may lose millions of dollars if the problem is not fixed in a timely manner. To support the solutions, it is best to hire an experienced BizTalk developer with deep exposure to BizTalk administration. In addition, you can leverage Microsoft Product Support Services. BizTalk PaaS For BizTalk PaaS, you also need a developer who has a good understanding of BizTalk Server and Azure. Although BizTalk services in Azure are designed from a learning and understanding perspective, a good BizTalk developer can easily adapt to the development environment of BizTalk PaaS. Debugging in the PaaS environment is not the easiest and can be considered the most challenging aspect of the integration option. Custom Integration For designing a custom end-to-end business scenario, you will need a well-experienced C# developer who has a good understanding of BizTalk processes. However, you do not need a dedicated BizTalk developer with administration experience.

Ease of Implementation BizTalk Server/BizTalk IaaS In the context of rapid integration, BizTalk IaaS is a reliable integration solution. It has a set of tools and application services for almost all applications including CRM, SAP, Salesforce, and Dynamics AX. The integration between applications becomes straightforward through available connectors/adapters.

Cost of Ownership 

 221

BizTalk PaaS The small integration in BizTalk PaaS can be quicker than that in BizTalk IaaS. Custom Integration The rapid integration facility is not available for custom integration, as there is nothing to reuse, and everything must be built from scratch. However, SDKs are available for EDI-related processes. You can also use the built-in XML capabilities of C#.

Cost of Ownership BizTalk IaaS/BizTalk Server Cost is one of the key factors that should be taken into consideration while comparing BizTalk Server and its alternatives. BizTalk Server 2013 onward, the licensing model has changed to per core license. Per core licensing allows customers to choose the desired license based on the required computing power. Now, the licensing does not depend on the environment, which means there is no difference whether the solutions are deployed on the local physical server, or on the cloud, like Azure Logic Apps. When BizTalk Server is installed on a physical machine, all cores must be licensed. You need a minimum of four core licenses for each physical processor on the server system. Figure 6.12 shows the core-based pricing model.

Figure 6.12: Core-based pricing

222 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

As discussed earlier, all cores on the BizTalk Server machine should be licensed. You can find information about the number of cores by executing the MsInfo32. exe application, which is in the Windows folder, as shown in Figure 6.13.

Figure 6.13: Location of MsInfo32.exe application

As you run this application, you will get the System Information window. This window displays all information about your entire system, including its OS, system name, processor, and BIOS, etc., as shown in Figure 6.14.

Figure 6.14: The System Information window

Note Pricing also varies based on the manufacturer of the processor, which means pricing should be different for Intel and AMD-based processors. If you have an Intel processor having four cores, you need four core licenses. If you have an AMD processor with four cores, you need three (4 * 0.75) core licenses.

Cost of Ownership 

 223

BizTalk PaaS BizTalk PaaS services are available in five tiers. The actual cost of implementation depends on the selected tier. The five tiers are: –– Free (preview) –– Developer –– Basic –– Standard –– Premium Figure 6.15 shows different features of the above-listed tiers.

Figure 6.15: Different features of BizTalk Azure Services tiers

As we have shown, we use an integration account and Azure Logic Apps for the integration scenario in BizTalk PaaS. Azure Logic Apps implementation account is charged based on the number of actions executed. The price per action execution for Azure Logic Apps is shown in Figure 6.16.

224 

 Chapter 6: Compare Azure BizTalk Applications: BizTalk IaaS, BizTalk PaaS, and Custom Integration

Figure 6.16: Azure Logic Apps actions

The pricing list shown in Figure 6.16 does not have data transfer prices. In Azure, you do not need to pay anything for inbound data transfer. However, you must pay for outbound data transfers. You also need to pay integration account charges at $1000/month; enterprise connection charges are $800 per connection per month. Custom Integration With custom integration, you need to build most of the things yourself through coding. Therefore, you need to pay only for third-party resources that you use. For example, while implementing an end-to-end EDI scenario, you need a third-party library that supports EDI transaction validations and processes including ACKs and parsing, etc. These libraries need to be purchased. In the context of EDI integration, there are many EDI SDKs available on the web. EDIFabric SDKs are one of the EDI SDKs that are available in two pricing plans, as shown in Figure 6.17.

Summary In this chapter, we compared three BizTalk Azure application options: BizTalk Server (hosted as IaaS on Azure), BizTalk PaaS (Azure Logic Apps), and Custom Azure Integration solution against a set of key features and enterprise readiness pivots. The bottom line is that all three options are viable, and customers can select the desired integration option based on their specific requirements and scenarios.

Summary 

 225

Figure 6.17: Pricing plans of EDIFabric SDKs

If you want to implement a solution to communicate with applications available on the cloud, you should use Azure PaaS—Azure Logic Apps. However, if you are dealing with on-premise resources in your integration solution, it is better to implement BizTalk Server in an IaaS role using a hybrid integration pattern. Finally, if you have very simple processing needs, say a few EDI files, you may consider custom integration hosted on Azure. Let us consider a scenario where we need to process one to five EDI files per day where each file contains 100–500 transactions. In this instance, it is better to implement a custom integration solution, as there is no need to invest in BizTalk licensing, BizTalk specific development, or maintenance costs with such a small number of transactions. In a more complex situation where we need to process many transactions in the range of 100–500 files per day, with different EDI transactions like 834, 835, 999, 997, and 850, we need to select BizTalk Server (IaaS on Azure) due to its capability of handling a large number of transactions and supporting the required features for large volume transaction processing. Mid-sized loads with short life cycles are better suited for BizTalk PaaS or Azure Logic Apps.

Chapter 7 Partner Solutions for BizTalk Azure Applications Introduction BizTalk Server covers a lot of ground—EAI, B2B, and managing business process coordination. BizTalk Server provides a wide range of services with a breadth of deployment scenarios, allowing partners to build complementary solutions, which further enhances the value of BizTalk Server. In this chapter, we discuss the following three different but complementary solutions that greatly enhance the BizTalk Server ecosystem: 1. BizTalk360: An effective monitoring solution that is especially important for large deployments. 2. WPC–HIPAA Database Toolkit: Ideal solution for integrating HIPAA-mandated EDI with backend or Line-of-Business applications. 3. Neudesic–ESB Toolkit: Ideal for scenarios that require complex rules for routing business documents including repairing and re-introducing invalid or non-compliant electronic documents.

BizTalk360 BizTalk360 is a Rich Internet Application (RIA) that provides a single platform for operating, managing, and monitoring your BizTalk Server environment. It was developed by Mr. Saravana Kumar in 2010, who is a well-known BizTalk Server expert and a Microsoft Most Valuable Professional (MVP). Mr. Kumar drew from his extensive experience with several large enterprise customers deploying BizTalk Servers. While providing support for these enterprises, he faced several issues related to operating, monitoring, and administering BizTalk Server. This led to the birth of BizTalk360. As mentioned earlier, there are several gaps in BizTalk Server, especially in its administration feature offering, BizTalk Admin Console. BizTalk Admin Console lacks important features, such as security, auditing, and monitoring. Additionally, you would need significant expertise in BizTalk Server to support and monitor BizTalk Server environments. To deal with these issues, Mr. Kumar formed a company, Kovai Ltd, to build BizTalk360, which supports BizTalk Server environments. The company added DOI 10.1515/9781501505652-007

228 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

two more products: BizTalk360 for Cloud and BizTalk360 for Service Bus into its product line in conjunction with the Microsoft launch of BizTalk Azure Services and BizTalk Server Azure VM. Important Points —— There are two ways to download a trial version. First, use a corporate email ID, which receives the trial code and trial key in the GUID format. Second, use your personal email ID. In the second case, you must go through an additional verification process wherein you will receive a call from the sales team. The sales team verifies you and asks for your requirements. Based on the information you provide, you will receive a trial code and trial key. —— The trial version is available for fourteen14 days. —— You cannot download the trial version directly from the BizTalk360 website. You can only download it from the dedicated download link received on your email ID.

As already discussed, Kovai offers three products, which are as follows: –– BizTalk360 –– BizTalk360 Azure –– ServiceBus360 These products possess common features. A few of the most important features of BizTalk360 are: 1. Environment management 2. User access policy 3. Monitoring and notification 4. Analytics health 5. BizTalk360 health 6. BizTalk360 Health Monitor 7. Advanced event viewer 8. ESB portal settings 9. Service Bus registration 10. Gateway/proxy settings 11. System settings 12. Custom widgets 13. API documentation

BizTalk360 

 229

A list of options associated with these features in the BizTalk360 environment is shown in Figure 7.1.

Figure 7.1: Options associated with features

Some of the BizTalk360 features are available for free. For example, BizTalk Health Monitor can be configured for free, as shown in Figure 7.2.

230 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.2: Configuring BizTalk Health Monitor

However, most features require a license (trial or full) to perform any operation. For example, the Advanced Event Viewer feature throws an exception, called Valid License Required, when you try to enable it, as shown in Figure 7.3. As discussed earlier, BizTalk360 provides several features in the following four categories: –– Operations –– Monitoring –– Analytics –– Extras

BizTalk360 

 231

Figure 7.3: Enabling the Advanced Event Viewer feature

Operations In BizTalk Server, there is a separate console or interface for a specific operation. To check the overall status of a BizTalk group, you need to look across several interfaces. For example, you need SQL Server to check the running status of SQL jobs related to BizTalk Server. BizTalk Admin Console is used to check the status related to ports and processes. To check logs, you need the Event Viewer console. Also, a separate console is required to create and deploy policies to the BizTalk group. Kovai’s major objective was to streamline these operations and provide a single console through which all operations could be monitored easily. So, the group introduced a single console, called Operations, in BizTalk360. This console also contains some valuable functionalities related to BizTalk operations. Figure 7.4 shows the BizTalk360 Operations dashboard:

232 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.4: BizTalk360 Operations dashboard

The features related to the BizTalk360 Operations console are divided into the following sections: 1. Dashboard/Home 2. Application support 3. Business rules 4. Data access 5. ESB portal 6. Azure services 7. Electronic Data Interchange 8. Health check tools 9. Security 10. Infrastructure setting Dashboard/Home This section contains several links that act as shortcuts to specific features. It also displays all artifacts (applications, orchestrations, Receive, and Send Ports), BizTalk Group properties, and BizTalk Server license information, etc. Application Support This section displays all applications available in your BizTalk environment. This section also contains the Search Artifacts option, which allows you to search for all artifacts that are available in your BizTalk Server environment. Figure 7.5 shows the Search BizTalk Artifacts window:

BizTalk360 

 233

Figure 7.5: The Search BizTalk Artifacts window

Business Rules This section displays the Business Rules Composer window where you can view and manage existing rules. You can also add or delete rules to new or existing policies, save, and deploy them. This section provides an almost identical interface as that of Microsoft’s Business Rules Composer, as shown in Figure 7.6.

Figure 7.6: The Business Rules Composer window

234 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Data Access This section contains features that allow you to view and execute queries and track data. It contains the following subsections: –– Message Box (Queries): This section allows you to select the desired type of query from the following alternatives: ○○ All In-Progress Service Instances ○○ Running Service Instances ○○ Suspended Service Instances ○○ Messages After selecting the query type, the user can click the Execute Query button to execute the selected query, as shown in Figure 7.7.

Figure 7.7: The Message Box (Queries) window

–– Graphical Flow (Tracking): This section allows you to view the process message flow. BizTalk Server’s publish and subscribe model makes it hard for the developer to understand the message flow. The message flow becomes too complex to understand when rules and itinerary services are added to it. The graphical message flow viewer of BizTalk360 helps a developer or administrator understand the message flow effortlessly.

BizTalk360 

 235

–– Secure SQL Queries: As discussed, the BizTalk Server environment is managed by two different teams, namely the BizTalk database management team, and the BizTalk operation team. The BizTalk operation team cannot track data, as it does not have the required privileges. This creates a lot of problems in the case of an emergency. This issue has been resolved in BizTalk360. Now, applicable users can track data easily. However, you should execute only the Select Query for security purposes, as shown in Figure 7.8.

Figure 7.8: Selecting a query

–– Business Activity Monitoring (BAM): BizTalk360 provides an integrated BAM portal, which means that you do not need to open BizTalk Admin Console to check BAM related activities. It has tools that can be used by business users to establish data points to get to information about a business process. The Business Activity Monitoring (BAM) Portal window is shown in Figure 7.9.

236 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.9: The Business Activity Monitoring (BAM) Portal window

–– Advanced Event Viewer: In BizTalk Server, the Event Viewer does not allow you to debug the suspended instances from various host instances, and the tedious workaround is to start a new console each time you receive an error. This problem has been addressed in BizTalk360, because it provides an integrated solution for Event Viewer. It displays events for all environments added to the BizTalk360 dashboard, as shown in Figure 7.10.

BizTalk360 

 237

Figure 7.10: The Advanced Event Viewer (Queries) window

ESB Portal This section provides the Enterprise Service Bus (ESB) Toolkit that extends the capability of BizTalk Server to implement a service-oriented architecture. BizTalk360 provides an integrated ESB exception management feature that allows you to configure the ESB portal and ESB exception database. This feature also allows you to resubmit messages through SOAP and WCF with itinerary headers. You can also validate the configuration settings before saving them, as shown in Figure 7.11.

238 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.11: The Configure Your ESB Exception Database window

Azure Services BizTalk360 also allows you to manage Azure Logic Apps as well as the integration account. To use this feature, you need to add Azure subscription details to BizTalk360. This can be done from the Add New Azure Subscription window, which can be opened by clicking SETTINGS MONITORING AND NOTIFICATION. The Add New Azure Subscription window also contains an option to use the Publish Settings file and the Enable Subscription for Monitoring Or Operation option to enable subscriptions, as shown in Figure 7.12.

BizTalk360 

 239

Figure 7.12: The Add New Azure Subscription window

Electronic Data Interchange BizTalk360 has integrated several EDI operations into the EDI dashboard. Now, you can view existing parties, agreements, fallback settings, and information about business profiles in the EDI dashboard. The Reports section allows you to view reports based on several types of Select queries including Interchange/ ACK Status, Batch Status, AS2/MDN Status, Transaction Set Details, Interchange Aggregate Report, Transaction Set Aggregate Report, and Functional ACK Status, as shown in Figure 7.13.

240 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.13: The EDI Status Reports window

Health Check Tools BizTalk360 provides the following two health check tools: –– Backup/DR Visualizer: Provides information related to backup and disaster recovery of BizTalk database servers. The visualizer displays information about complete backup, log settings, basic configuration, live environment, and standby environment. It also allows you to configure standby SQL Server instances. Figure 7.14 shows the Backup and Disaster Recovery Visualizer window. –– BizTalk Health Monitor: Allows you to configure and schedule BizTalk Health Monitor. You should note that BizTalk Health Monitor is not bundled with BizTalk360. You need to download it separately. You can download it from https://www.microsoft.com/en-in/download/details.aspx?id=43716. Once the installation is complete, you can configure it by providing a BizTalk Health Monitor installation directory and configuring the schedule for BizTalk Health Monitor execution, as shown in Figure 7.15.

BizTalk360 

Figure 7.14: The Backup and Disaster Recovery Visualizer window

Figure 7.15: Configuring and scheduling BizTalk Health Monitor

 241

242 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Security In the context of security, BizTalk360 provides an auditing feature that audits all the activities performed on artifacts and components related to the BizTalk Server environment. Within BizTalk Server, there is no method or interface available for operational auditing in the BizTalk Admin Console. For example, a support person can perform several activities including stopping the host instance and starting or stopping a business process or orchestration. These activities cannot be tracked in the BizTalk Admin Console. However, BizTalk360 tracks each activity performed in your BizTalk Server environment through the options available in the Governance/Audit window, as shown in Figure 7.16.

Figure 7.16: The Governance/Audit window

Using the options available in Figure 7.16, you can keep track of application artifacts and service or host instances, ESB or BRE messages, and user and business rules activities. Infrastructure Setting The Infrastructure Setting section contains several options that can be used to view information related to the BizTalk Server environment/infrastructure. It allows you to view the topology of the BizTalk Server environment, as shown in Figure 7.17.

BizTalk360 

 243

Figure 7.17: Topology of the BizTalk Server environment

The Infrastructure Setting section allows you to view the list of BizTalk hosts, as shown in Figure 7.18.

Figure 7.18: Viewing the list of BizTalk hosts

In addition, the Infrastructure Setting section contains options to start/stop host instances, as shown in Figure 7.19.

Figure 7.19: Starting/stopping host instances

244 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

This section also allows you to view information on BizTalk Server, as shown in Figure 7.20.

Figure 7.20: Viewing information of BizTalk Server

You can use the Tracking Manager window to view tracking settings of artifacts for each application, as shown in Figure 7.21.

BizTalk360 

 245

Figure 7.21: Using the Tracking Manager window

The Infrastructure Setting section also contains the Adapters option that allows you to check the BizTalk Adapters, as shown in Figure 7.22.

246 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.22: Checking BizTalk Adapters

Monitoring As discussed throughout the book, BizTalk Server integrates different applications running in the same or different enterprises. However, BizTalk Server faces some difficulties while monitoring the status, database health, memory and other resources of the connected applications.

BizTalk360 

 247

BizTalk360 offers its own Monitoring Dashboard that overcomes the difficulties faced by BizTalk Server. BizTalk360 Monitoring Dashboard allows you to add alerts on individual BizTalk components, as shown in Figure 7.23.

Figure 7.23: BizTalk360 Monitoring Dashboard

BizTalk360 Monitoring Dashboard also allows you to manage, create, edit, copy, and delete alarms, as shown in Figure 7.24.

Figure 7.24: The Manage Alarms window

After creating an alarm, you may need to associate or map it with different components, such as applications or other artifacts at application levels, SQL Servers, queues, or Azure subscriptions, etc. In our case, we have created an alarm for

248 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

threshold monitoring, mapped it to monitor the host instance, and set the Expected State option to Started, as shown in Figure 7.25.

Figure 7.25: Creating an alarm for threshold monitoring

Data Monitoring You can set up data monitoring that allows you to monitor data flowing through BizTalk processes, some of which include Receive Ports, Receive Locations, and Send Ports. To set up data monitoring, you need to select the Data Monitoring option from the left pane of the Monitoring Dashboard window. This opens the Process Monitoring window. From there, add the required details under applicable sections and click the Save and Close button to save data alarm, as shown in Figure 7.26.

Figure 7.26: The Process Monitoring window

BizTalk360 

 249

This adds an alarm for data monitoring in the Alarms – Data Monitoring window, as shown in Figure 7.27.

Figure 7.27: Creating an alarm in the Alarms – Data Monitoring window

Similarly, we can add more alarms for data monitoring related to: –– MessageBox data –– Tracking data –– BAM data –– EDI data –– ESB data –– Azure Logic Apps

Analytics Process and performance analysis of any business is a critical task. However, there are several conditions in which critical decisions can be made based on the reports of various components of the integration environment. BizTalk360 offers a very robust and integrated Analytics service for performance analysis. The Analytics Environment Settings window allows you to enable analytics for different kinds of artifacts (which are either running or deployed in the environment) and other involved components, such as SQL Server and IIS. Figure 7.28 shows the Analytics Environment Settings window with the performance data analytics enabled.

250 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.28: Enabling performance data analytics

Figure 7.29 shows the Analytics Environment Settings window with the tracking data analytics enabled.

BizTalk360 

 251

Figure 7.29: Enabling tracking data analytics

Messaging Pattern Visualizer One of the most useful features of BizTalk360 is that it displays the message flow of your process setup wherein the process ID (GUID) can be selected to display the message flow diagram, as shown in Figure 7.30.

252 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.30: Displaying the message flow diagram

To summarize, BizTalk360 contains an invaluable set of extensions provided by Kovai to manage complex BizTalk deployments.

WPC (WPC-EDI) Washington Publishing Company (WPC) is a Microsoft Partner Company based in Seattle, Washington. It was founded in 1975 and is well known in both EDI and healthcare domains. It provides many relevant services related to EDI, healthcare documentation, and reference solutions. The company also provides: –– The FirstPass HIPAA transaction testing tool –– Windows desktop application that validates X12 EDI data to X12 syntax and other formats specified in the 5010 X12 Type 3 Technical Reports (TR3) adopted under HIPAA –– A HIPAA Database Toolkit (which is the most relevant for further discussion)

HIPAA Database Toolkit HIPAA Database Toolkit is a Microsoft BizTalk Adapter developed by WPC (WPCEDI) (http://www.wpc-edi.com/products/edi-overview/). It supports both the versions of EDI transactions (4010 and 5010) and serializes EDI Transaction sets to insert the transaction data into the SQL Server database. The Toolkit includes an adapter that can only be used with the BizTalk Send Pipeline. This adapter

HIPAA Database Toolkit 

 253

receives the XML data generated by disassembling EDI and inserts the transactional data for each transaction into the SQL Server database. The HIPAA Database Toolkit includes a schema generator feature, which creates a DDL script for each HIPAA EDI transaction. When a user runs the script in SQL Server, appropriate database objects are created as elaborated below: –– A set of tables wherein each field is included in the EDI transaction –– Stored procedure for inserting the EDI data –– A set of views referring to each loop and subloop of the EDI transaction Note HIPAA Database Toolkit Adapter supports all versions of BizTalk Server starting from BizTalk Server 2006.

Installing Database Toolkit The installation file of the HIPAA Database Toolkit Adapter is a standard MSI file, titled HIPAADatabaseToolKitSetup5010.msi. Perform the following steps to install the HIPAA Database Toolkit adapter: 1. Double-click the installation file. The DB Toolkit Setup wizard appears with the Welcome page. 2. Click the Next button. The End-User License Agreement page appears. 3. Select the “I accept the terms in the License Agreement” checkbox. 4. Click the Next button, as shown in Figure 7.31.

Figure 7.31: The End-User License Agreement page

254 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

The Destination Folder page appears. 5. Specify the destination folder location in the Install DB Toolkit to text box. You can either select default location, i.e., C:\Program Files (x86)\WPC\, or click the Change button to change the location. 6. Click the Next button, as shown in Figure 7.32.

Figure 7.32: The Destination Folder page

The Ready to install DB Toolkit page appears. 7. Click the Install button, as shown in Figure 7.33.

Figure 7.33: The Ready to install DB Toolkit page

HIPAA Database Toolkit 

 255

The installation process starts. 8. Click the Finish button to finish the installation.

Enabling the CLR in the SQL Server HIPAA Database Toolkit uses a .NET CLR-based stored procedure to parse the EDI messages. For this, we need to enable the CLR in SQL Server. By default, the CLR is disabled in SQL Server for security reasons. It can be enabled in SQL Server by executing the query shown in Figure 7.34.

Figure 7.34: Enabling the CLR in SQL Server

Generating EDI (Version 5010) Transaction DDL Scripts As discussed earlier, a schema generator file (SchemaGenerator5010.exe) generates the DDL scripts for a BizTalk HIPAA EDI schema. This file is located in the SchemaGenerator folder at the Database Toolkit installation location, as shown in Figure 7.35.

Figure 7.35: Locating the schema generator file

The SchemaGenerator5010 application is only compatible to either HIPAA EDI schemas distributed with BizTalk Server, or schemas modified from the original XSD files without changing their structure and naming conventions. The syntax for executing the schema generator file is as follows: Cmd> [-d | -i]

256 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

A description of options associated with the schema generator file is as follows: –– The -d option is used to eliminate the fields marked as unused –– The -i option is used to include the fields marked as unused This is easier to understand with the help of an example. In our example, we will generate the database schema for the HIPAA-5010–834 transaction. We have used the “–i” option with the SchemaGenerator5010.exe to generate EDI-834 schema DDL script in the “E:\Generated DDL Scripts\WPC.DBToolkit.5010_834_Script. sql” file, as shown in Figure 7.36.

Figure 7.36: Generating the DDL script

Once the script is generated, we receive a SQL script file at the “E:\Generated DDL Scripts” location, as shown in Figure 7.37.

Figure 7.37: Viewing the SQL script file

Using DDL Script to Configure EDI Transaction Database Once the DDL script is generated, you can use it to create database objects for the 834 transaction on SQL Server. In our case, we have created a database (EDI_5010_834) manually in SQL Server. Now, we need to run the generated script against the created database, as shown in Figure 7.38.

HIPAA Database Toolkit 

 257

Figure 7.38: Running the generated script against the created database

Once the script runs, we can check the database for created objects. We can see that the following database objects are created: –– Stored Procedures: A total of four stored procedures are created, as shown in Figure 7.39.

Figure 7.39: The Stored Procedures

–– Tables: A total of forty-seven tables are created. The table names refer to the schema nodes in the EDI_5010_834 schema, as shown in Figure 7.40.

258 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.40: Schema nodes table

–– Views: A total of forty views are created, as shown in Figure 7.41.

HIPAA Database Toolkit 

 259

Figure 7.41: System Views

Using the Database Toolkit (DBToolKit) Adapter Next, you will learn how to use the Database Toolkit Adapter. For this, we need to set up a simple EDI process that contains an EDI Receive Port and a Send Port with the Database Toolkit Adapter subscribing to the Receive Port. Figure 7.42 shows an example of a Receive Port.

260 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.42: Creating a Receive Port

Figure 7.43 shows a Send Port subscribing to the created Receive Port.

Figure 7.43: Creating a Send Port

HIPAA Database Toolkit 

 261

When creating the Send Port, you need to set transport options that can be viewed by selecting the Transport Advanced Options option in the left pane of the Send Port Properties dialog box. As you select the Transport Advanced Options menu in the left pane, the related settings appear in the right pane. Select the DBToolKit option from the Transport type (Adapter) drop-down list and click the Configure button to configure the selected transport type. This opens the DBToolKit Transport Properties dialog box, which contains several properties, as shown in Figure 7.44.

Figure 7.44: The DBToolKit Transport Properties dialog box

A description of properties under the DBToolKit Transport Properties dialog box is as follows: –– Connection String: This property contains the connection string of the database. In our case, we have used the connection string: “Data Source= ‘Administrator’; Initial Catalog=‘EDI_5010_834’; Integrated Security=‘SSPI’” –– CorrelationPropertyNamespace: Database Toolkit supports external correlation through a correlation token associated with each message to be stored in the database. Generally, when you run a DDL script, a table named EDITransaction is created with a field name externalCorrelationToken, as shown in Figure 7.45.

262 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.45: Displaying the EDITransaction table

The externalCorrelationToken field is indexed; however, it is not a unique index. To use the correlation feature of the Database Toolkit Adapter, we need to add a property schema and promote the externalCorrelationToken property in this schema. This property schema namespace should be used against the CorrelationPropertyNamespace property while configuring the Database Toolkit Adapter. –– SimpleParsingProcedure: This property is automatically set by the Adapter and requires a stored procedure (from the available stored procedures) as a value to load the BizTalk XML into the SQL Server database. –– StreamingTransactionThreshold: This property is automatically set by the Adapter and does not require manual configuration. –– XPath Filter: This property is set when you want to filter the messages to be inserted into the database. In a nutshell, we need to make modifications to the connection string. Testing the Database Toolkit Implementation For testing the Database Toolkit implementation, we need to drop an 834 EDI file at the Receive Location that was configured while creating the Receive Port, as shown in Figure 7.46.

HIPAA Database Toolkit 

 263

Figure 7.46: Dropping an 834 EDI file in the Receive Location

Once this process is complete without any glitches, check the database tables. In our example, as there are forty-seven tables and forty views, we will have to check one or two of them. Let us check the table named [X12_EnvelopeHeader]. To view all data available in the table, we need to create two queries, as shown in Figure 7.47.

264 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.47: Creating queries to check data

Our results show that the received data matches the test file data dropped at the Receive Location. Note HIPAA Database Toolkit allows you to save the complete transaction data without writing any code and with minimal configuration.

In summary, the WPC HIPAA Database Toolkit simplifies the integration of the HIPAA EDI Transaction set with your backend/Line-of-Business application. The Toolkit creates the database artifacts required to store the serialized EDI Transaction set data into the SQL Server database because the data consumption from the SQL Server tables is relatively easy.

Neudesic Neudesic, LLC is a Microsoft Partner Company, which is based in Irvine, California, founded in 2001. It provides software products and services related to Enterprise Service Bus (ESB) Integration, enterprise social software, Microsoft technology solutions, and Customer Relationship Management (CRM) solutions. The company also provides: –– CRM services –– Application development services –– Application and system integration services –– Cloud computing services –– Enterprise collaboration services –– Business intelligence services –– Enterprise mobility services

Neudesic 

 265

The Enterprise Service Bus (ESB) solution is one of the most widely used solutions and is now included with BizTalk Server releases.

What is Enterprise Service Bus (ESB)? Enterprise Service Bus (ESB) is an architecture that allows you to implement a service-oriented infrastructure. It is a collection of: –– Web Services patterns –– .NET and Java interoperability –– Enterprise Application Integration –– Message-oriented middleware –– Interoperability with service registries and asset repositories –– Host system integration ESB controls the way different applications communicate with each other in a rather loosely coupled manner.

BizTalk ESB Toolkit BizTalk ESB Toolkit, as the name indicates, is a set of tools that provide: –– SOA based architecture patterns, practices, and guidance –– Sample applications –– BizTalk Server applications –– ESB web applications –– .NET framework components These tools can be used for developing enterprise-level applications to implement a loosely coupled message environment, and build dynamic message-based applications.

BizTalk ESB Toolkit Components There are several components within the BizTalk ESB Toolkit. Some of these components are as follows: –– ESB web services –– ESB management portal –– ESB pipeline interop components –– Exception management framework

266 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

–– ESB resolver and adapter provider framework –– Itinerary based routing and processing –– ESB Toolkit sample applications

Installation From BizTalk Server 2013 onwards, ESB Toolkit setup is included in the BizTalk Server installation. Installation of the ESB Toolkit requires the following steps: 1. Run the BizTalk Server installer with Administrator privileges. The Microsoft BizTalk Server window appears. 2. Click the Install Microsoft BizTalk ESB Toolkit option, as shown in Figure 7.48.

Figure 7.48: Install Microsoft BizTalk ESB Toolkit option

The Microsoft BizTalk ESB Toolkit 2.4 Installation Wizard appears with the License Agreement page. 3. Select the “Yes, I accept the terms of the license agreement” radio button to accept the license agreement. 4. Click the Next button, as shown in Figure 7.49.

Neudesic 

 267

Figure 7.49:  Accepting the license agreement

The Component Installation page of the Microsoft BizTalk ESB Toolkit 2.4 Installation Wizard appears. 5. Select the components that you want to install from the Available components section. 6. Specify the location where you want to install components in the Install to text box. 7. Click the Next button, as shown in Figure 7.50.

268 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.50: The Component Installation page

The Ready to install page appears. 8. Click the Install button to start the installation. The installation process begins. 9. Click the Finish button to complete the installation.

Configuring ESB Toolkit After installing ESB Toolkit, we need to configure it. ESB Toolkit uses the database for storing message persistence state and keeping track of all messages and operations. Perform the following steps to configure ESB Toolkit: 1. Locate the ESB Configuration Tool option in the Start menu. 2. Right-click the ESB Configuration Tool option. A context menu appears. 3. Select the Run as Administrator option from the context menu to run the tool with administrative rights. The ESB Configuration Tool window appears. 4. Specify the configuration settings, such as the name of the database server and user credentials, in the ESB Configuration Tool window. 5. Click the Apply Configuration button, as shown in Figure 7.51.

Neudesic 

 269

Figure 7.51: Applying configuration settings in the ESB Configuration Tool window

After clicking the Apply Configuration button, the Exception Management components and ESB Core components are configured. You can verify the configuration in the IIS management console. If the configuration is made correctly, you will see several applications added under the Default Web Site option with the names starting with ESB, as shown in Figure 7.52.

270 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.52: Verifying the configuration

Configuring ESB BizTalk Applications The ESB BizTalk Applications configuration includes: –– Enabling ESB core components in BizTalk Server –– Enabling ESB JMS/WMQ components in BizTalk Server You must manually complete this configuration. The configuration can be done with the help of the ESB Configuration Tool window. Perform the following steps to configure ESB BizTalk Applications: 1. Open the ESB Configuration Tool window. 2. Select the ESB BizTalk Applications option in the left pane. The related configuration options appear in the right pane. 3. Select the Enable ESB Core Components in BizTalk Server checkbox.

Neudesic 

 271

4. Select the Enable ESB JMS/WMQ Components in BizTalk Server checkbox. 5. Click the Apply Configuration button, as shown in Figure 7.53.

Figure 7.53: Configuring ESB BizTalk Applications

Upon completion, a status message appears in the bottom of the ESB Configuration Tool window stating “Status: EsbBizTalkApplications: Configured,” as shown in Figure 7.54.

Figure 7.54: Configuration status

272 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

In Figure 7.54, you can see that the checkboxes are deactivated. This means that all applied settings are configured. The configuration changes can be verified by locating the applications named “Microsoft.Practices.ESB” and “Microsoft.Practices.JMS” in the BizTalk Server Administration Console window, as shown in Figure 7.55.

Figure 7.55: Verifying the configuration changes

We can now say that we are done with the process of installing and configuring ESB WCF services and ESB BizTalk Applications. However, from the management perspective, we also need to add the ESB Management Portal to our IIS so that we can create, edit, or view fault settings and ESB exceptions.

ESB Management Portal ESB Management Portal is an application that manages exceptions and fault messages. It manages both types of exceptions including system and business process exceptions that occur during message processing. It also keeps track of

Neudesic 

 273

the message processed during exception. It allows you to create alerts based on specific conditions and send notifications to the subscribers’ list. Configuring Management Portal ESB Management Portal must be configured explicitly. When you install ESB Toolkit, a zip file named “ESBSource.zip” is created in the installation folder, as shown in Figure 7.56.

Figure 7.56: Locating the ESBSource.zip file

The ESBSource.zip file contains ESB sample projects (source codes). You can extract this file to view its content. Once the file is extracted, you can navigate to ESBSourceSourceSamplesManagement Portal to view the ESB.Portal folder, as shown in Figure 7.57.

Figure 7.57: Navigating to the ESB.Portal folder

274 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

The ESB.Portal folder contains a web application (source code). You can run this application in Visual Studio and rebuild the solution to ensure that all required assemblies are available in the Machine Assemblies folder. Note We need to install the Microsoft Report Viewer application, which is used by the ESB.Portal web application. The installer file of the Microsoft Report Viewer application can be obtained from the following links based on different versions: —— https://www.microsoft.com/en-us/download/details.aspx?id=6576 —— https://www.microsoft.com/en-us/download/details.aspx?id=6442

Next, run the script that installs the ESB Portal in IIS, and creates the configurations (including creating the “ESBAdmin” database) in the local SQL Server. This script is available in the “\install\Scripts” folder, and its name is “Management_Install.cmd.” To run the script, we need to enable the unrestricted execution policy. For this, we need to run PowerShell with Administrator privileges and run the following command: Cmd>set-executionpolicy –executionpolicy unrestricted

The output of the above command is shown in Figure 7.58.

Figure 7.58: Running a command

We will run the script, as shown in Figure 7.59.

Neudesic 

 275

Figure 7.59: Running the script

When the script execution starts, all projects in the solution are built. In addition, in the last stage of script execution, a database (ESBAdmin) and an application (ESB.Portal) are created, as shown in Figure 7.60.

Figure 7.60: Displaying the created database and application

276 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

In a nutshell, the script is executed successfully if there are no red lines (exceptions). We can verify this by checking the IIS Default Web Site option, as shown in Figure 7.61.

Figure 7.61: Verifying the IIS Website

From Figure 7.61, we see that four new applications are created in IIS under the default website after the script execution. Last, check the ESB portal by browsing to “http://localhost/ESB.Portal/” to verify that the application is running correctly, as shown in Figure 7.62.

BizTalk ESB Implementation 

 277

Figure 7.62: Browsing http://localhost/ESB.Portal/

BizTalk ESB Implementation In the earlier sections, you learned how to set up and configure ESB Toolkit, ESB Management Portal, and BizTalk ESB applications. In this section, you will learn to use ESB components in a BizTalk application or solution. ESB Toolkit does not require redeploying the entire application.

Implementing Dynamic Routing Based on Message Type The implementation process can be better understood through an example. In our example, we have two trading partners, partner A and partner B, that exchange orders and invoices. In this scenario, we are going to implement the solution for trading partner A. A sells its products to B, and B sends its orders to A. As B also sells its products to A, B will accept orders from A. Therefore, B will send invoices to A after processing the received orders. To implement the solution for A, we should be able to receive and process two types of documents, that is, orders and invoices. Also, there is a single channel through which A receives both the documents/files, which means that we have only one Receive Port and one Receive

278 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Location to receive both files, that is, orders and invoices. These documents/files need to be sent to different locations based on the type of document or message so that the respective department (Order Processing Department/Invoice Processing Department) can receive its intended file. Once the file/document is received by the respective department, it can be used for further processing. Therefore, we need to implement a solution that dynamically decides the message outbound location based upon the message type. As a note, we could also implement this solution without using ESB. In this scenario, we could use a custom pipeline component and C# code to promote the message outbound location property based on the message type. However, this approach would require updating code, testing, and a redeploy of the entire application in every instance where we receive some other type of document. With our initial example solution, we can take advantage of ESB’s loose coupled messaging architecture where resolvers are used to determine the message outbound location based on message type. The loosely coupled messaging architecture means that its components can be replaced by alternative implementations, which can provide the same functionalities. Creating Artifacts Before starting the itinerary coding or development, we need to create ports to be used by the itineraries. In this section, we will create different artifacts. BizTalk Application Primarily, we have created an application named ESBTestApp in BizTalk Admin Console. We have added ESB applications as references to the created application so that ESB artifacts can be used with the created application, as shown in Figure 7.63.

BizTalk ESB Implementation 

 279

Figure 7.63: Adding ESB applications as references

On-Ramp Receive Port In this section, you will learn to create a Receive Port, which receives both types of files/documents, that is, orders and invoices. An On-Ramp Receive Port is a BizTalk Receive Location for receiving messages to be processed through ESB. The Receive Port will have a single Receive Location, and we will use the ESB ItinerarySelectReceiveXml pipeline. This will be used to receive the XML file and then select the itinerary, which should be called for further processing on the file. In our case, we have created a Receive Port and configured the selected pipeline, as shown in Figure 7.64.

280 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.64: Creating a Receive Port

Off-Ramp Dynamic Send Port An Off-Ramp component in the ESB itinerary model corresponds to the BizTalk dynamic Send Port. For sending messages in an itinerary model, we need a dynamic Send Port, as shown in Figure 7.65.

Figure 7.65: Creating a Send Port

BizTalk ESB Implementation 

 281

We need to specify filters for the dynamic Send Port, which identifies the messages to be routed to the Send Port from the database. The dynamic Send Port should have the following filters: Microsoft.Practices.ESB.Itinerary.Schemas.ServiceState == Pending And Microsoft.Practices. ESB.Itinerary.Schemas.ServiceType == Messaging And Microsoft.Practices.ESB.Itinerary.Schemas.ServiceName == BTS_ESB_MessageType_Routing

Figure 7.66 shows the options under the Filters section of the Send Port Properties dialog box.

Figure 7.66: Filters section of the Send Port Properties dialog box

Schemas for Testing Since we are going to route the files based on message type, we need two schemas each for orders and invoices. These schemas are required for testing. In this section, we have created two schemas, Orders and Billing, for orders and invoices respectively. These schemas have been deployed to the created application, that is, ESBTestApp. Figure 7.67 shows the Orders schema.

282 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.67: Table of Orders schema

Figure 7.68 shows the Billing schema:

Figure 7.68: Table of the Billing schema

As discussed earlier, we need to deploy these schemas to the BizTalk ESBTestApp application, as shown in Figure 7.69.

Figure 7.69: Deploying schemas to the BizTalk ESBTestApp application

Now, we have the required schemas that will be used for validation during file processing.

BizTalk ESB Implementation 

 283

Creating an ESB Itinerary Application Next, we need to create an application in Visual Studio with the BizTalk ESB Itinerary Designer template, as shown in Figure 7.70.

Figure 7.70: Creating an ESB itinerary application

The BizTalk ESB Itinerary Designer template creates an ESB Itinerary application and adds an itinerary designer (BTS_ESB_MessageType_Routing.itinerary). As you select this option, you will see an Itinerary Toolbox and Itinerary Designer area, as shown in Figure 7.71.

Figure 7.71: The Itinerary Designer area

284 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

You should always note that an itinerary model requires one OnRamp and at least one ItineraryService element, as shown in Figure 7.72.

Figure 7.72: Displaying the requirements

On-Ramp Service Now, we need to add an On-Ramp component in the itinerary and configure the already created Receive Port to use this itinerary, as shown in Figure 7.73.

Figure 7.73: Adding an On-Ramp component

Messaging Broker Service Messaging broker service is used to make decisions in an itinerary to check the message type and perform further processing based on the selected message type. Therefore, we will use a message broker service with filters. To add a message broker, we need to add an Itinerary Broker Service tool to the itinerary surface and configure it, as shown in Figure 7.74.

BizTalk ESB Implementation 

 285

Figure 7.74: Adding an Itinerary Broker Service tool and configuring it

Adding a Resolver A resolver is used to support message routing. It is also used in other decision-making processes in the itinerary model. In our implementation, we need to add a resolver to the message broker. In the Itinerary Designer area, right-click the Resolve option in the Messaging Broker Extender component, and select the Add new Resolver option, as shown in Figure 7.75.

Figure 7.75: Adding a resolver

In the Properties window, select Message Context Resolver, which is used to resolve the message type from message context, as shown in Figure 7.76.

286 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.76: Selecting Message Context Resolver

Filters Now we need to add filters. These filters work as conditions. Since we are required to check two types of messages, we need to add two filters, that is, InvoiceFilter and OrdersFilter. Figure 7.77 shows the Properties dialog box for the InvoiceFilter filter:

Figure 7.77: The Properties dialog box for the InvoiceFilter filter

Figure 7.78 shows the Properties dialog box for the OrdersFilter filter.

Figure 7.78:  The Properties dialog box for the OrdersFilter filter

BizTalk ESB Implementation 

 287

Itinerary Broker Out Port At this point, we have created a broker and two filters. Now, we need to add an output path (normally called Out Port) to both filters. These are equivalent to the decision shape connectors of BizTalk Orchestration Engine. These Out Ports connect the filters to other services in the Itinerary Designer area. Therefore, we have created two Out Ports, one each for the OrdersFilter and InvoiceFilter filters. Figure 7.79 shows the Out Port (marked with the red rectangle) for the InvoiceFilter filter:

Figure 7.79: The Out Port for the InvoiceFilter filter

Figure 7.80 shows the Out Port (marked with the red rectangle) for the OrdersFilter filter:

Figure 7.80: The Out Port for the OrdersFilter filter

From the above figures, we can see that the InvoiceFilter filter should be resolved at the first index as the Order property is set to 0, while the OrdersFilter filter should be resolved at the next index as the Order property is set to 1.

288 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Itinerary Service Itinerary Service primarily contains processing instructions. In this section, you will learn how to add two itinerary services. Figure 7.81 shows an itinerary service for the Orders type messages.

Figure 7.81: Adding an itinerary service for the Orders type messages

We also need to add a resolver in the added itinerary service so that it can process the Orders type documents/messages, as shown in Figure 7.82.

Figure 7.82: Adding a resolver

Figure 7.83 shows an itinerary service for invoices type messages.

BizTalk ESB Implementation 

 289

Figure 7.83: Adding an itinerary service for invoices type messages

Next, we need to add a resolver to process invoices type messages/documents, as shown in Figure 7.84.

Figure 7.84: Adding a resolver to process invoices type messages/documents

Off-Ramp Service and Off-Ramp Extender In our next step, we need to design the outbound service. For this, we need to create two Off-Ramp services for both types of documents, that is, orders and invoices. We also need to create two itinerary services (Off-Ramp extenders) for both types of documents. An Off-Ramp service receives the message from the Off-Ramp extender (itinerary service) and pushes that message to the dynamic Send Port. Finally, it sets the outbound location of the Send Port dynamically based on the messaging extender (itinerary service) created earlier. Figure 7.85 shows an Off-Ramp service for the orders type documents.

290 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.85: Creating an Off-Ramp service for the orders type documents

Figure 7.86 shows an Off-Ramp extender (itinerary service) for the orders type documents. Figure 7.86: Creating an Off-Ramp extender (itinerary service) for the orders type documents

Figure 7.87 shows an Off-Ramp service for invoices type messages/documents.

Figure 7.87: Creating an Off-Ramp service for invoices type messages/documents

Figure 7.88 shows an Off-Ramp extender (itinerary service) for invoices type messages/documents:

BizTalk ESB Implementation 

 291

Figure 7.88: An Off-Ramp extender (itinerary service) for invoices type messages/documents

Defining the Sequence of Execution (Using Connectors) Next, we need to use connectors to associate the shapes in their order of execution, as shown in Figure 7.89.

Figure 7.89: Associating the shapes in their order of execution

Understanding the Process Flow The process flow would be as follows: 1. A message is received by an On-Ramp service. 2. The On-Ramp service forwards the message to the Messaging Broker Extender. 3. The MessageContextResolver resolver in the Messaging Broker Extender extracts the message context to fetch the message type. 4. The filters start functioning based on the message type. 5. If the message type is orders, then the following tasks are performed:

292 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

a. The Messaging Broker Extender is invoked for the orders message type. It assigns the message outbound location and sends the message to the OrderOffRampExtender (itinerary service). b. The OrderOffRampExtender (itinerary service) associates the message to the Order_ Off_Ramp service. c. The Order_Off_Ramp service invokes the dynamic Send Port and sets its outbound location. d. BizTalk Server sends/pushes the message to the outbound location. Note A similar sequence of events is executed for the invoices message type through invoice-specific ESB services.

Validating the Itinerary Once the itinerary design is complete, you need to validate it. To validate the itinerary, you need to right-click anywhere in the Itinerary Designer area and select the Validate option from the context menu, as shown in Figure 7.90.

Figure 7.90: Validating the itinerary

Once the validation process is complete, you will receive the validation output that shows errors, warnings, and messages in the Error List window, as shown in Figure 7.91.

BizTalk ESB Implementation 

 293

Figure 7.91: Validation output

In Figure 7.91, we can see that there is no error. However, there is one warning to use an X509 Certificate for encryption.

Deploying the Itinerary The next task after successful validation is deploying the itinerary. The deployment process for an itinerary is straightforward and contains the following two steps: 1. Set the value of the Model Exporter type as Database Itinerary Exporter, as shown in Figure 7.92. 2. Right-click anywhere in the Itinerary Designer area and select the Export Model option from the context menu, as shown in Figure 7.93.

294 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.92: Setting the value of the Model Exporter type

Figure 7.93: Selecting the Export Model option

Figure 7.94 shows the output of the export process of the model.

BizTalk ESB Implementation 

 295

Figure 7.94: Output of the export process of the model

In Figure 7.94, we can see the message “The model ‘BTS_ESB_MessageType_ Routing’ was exported successfully to Itinerary Database ‘EsbItineraryDb.’” This means the export process was successful. You can also view the exported itineraries in the EsbItineraryDb database, as shown in Figure 7.95.

Figure 7.95: Viewing the exported itineraries in the EsbItineraryDb database

Testing ESB Implementation To test ESB implementation, we need to drop at least two files in the Receive Location. We have dropped two files called Billing_Input.xml and Orders_input.xml. Figure 7.96 shows the content of the Billing_Input.xml file (Message type: http:// Test.XML.Schemas.Billing#Invoice).

296 

 Chapter 7: Partner Solutions for BizTalk Azure Applications

Figure 7.96: Displaying the content of Billing_Input.xml

Figure 7.97 shows the content of the Orders_input.xml file (Message type: http:// IntegrationAccountDemo.Orders_XML#Orders).

Figure 7.97: Displaying the content of the Orders_input.xml file

Here we have dropped both the files in the same Receive Location, that is, F:\ BTS_Drops\On_Ramp_RecvLoc, as shown in Figure 7.98.

Figure 7.98: Dropping files in the Receive Location

Now, we can check the output locations to view the respective files, as shown in Figure 7.99.

Summary 

 297

Figure 7.99: Checking the output locations

In Figure 7.99, we can see that the output locations set through Off-Ramp Itinerary service contain the respective output files. We can also see that each message is routed to a different location based on the message type.

Summary In this chapter, we discussed three very essential Microsoft partner solutions that significantly contribute to the integration capabilities of BizTalk Server. BizTalk360 is a popular partner that fills the gaps left in BizTalk Admin Console. It provides integration services for on-premises BizTalk Server and BizTalk Cloud services. WPC is known for its contribution in the BizTalk domain by providing EDI schemas, documentation, and implementation references for healthcare EDI projects. It also provides several tools including FirstPass Editor and Database Toolkit Adapter. Neudesic is known for its contribution in enterprise integration solutions with service bus implementation and loosely coupled message environment. Armed with these three solutions you can get much more from your BizTalk solution.

Chapter 8 Summary—Three Valuable BizTalk Azure Application Options Introduction BizTalk Server is the essence of integration; the integration of external (B2B), and internal (EAI) business processes. This book presented three distinct options of delivering BizTalk Applications on Azure. In this summary we will do a quick recap and walk through the major talking points covered in the preceding chapters.

History of BizTalk Chapter 1 introduced the history of BizTalk Server. Prior to BizTalk Server, there were several complementary and dev-oriented technologies delivered by Microsoft. Component Object Model (COM) was introduced by Microsoft in 1993 as a standard that defined programming requirements and object models, allowing objects to communicate with other objects. In 1995, an extension of the COM technology was introduced which was called Distributed COM (DCOM) technology. This technology enabled communication among components in a distributed environment. It used the Remote Procedure Call (RPC) mechanism for transmitting information between COM components across the network. DCOM was the first avatar in the integration space; however, its use required developer expertise. In the 1990s, websites and web applications were the rage and in 1996, Microsoft released its first version of Site Server (Microsoft Site Server 1.0) to manage the stateful websites and e-commerce web applications. In October 1996, Microsoft introduced Microsoft Merchant Server to provide e-commerce solutions. This version contained limited features, and its associated technologies were migrated to Microsoft Site Server 2.0, which was released in December 1996. Microsoft Site Server 2.0, also called Microsoft Site Server Commerce (MSSC), provided simplified e-commerce solutions. In 2000, Microsoft Site Server was replaced by Microsoft Commerce Server, which provided several tools for web developers to manage e-commerce applications effectively, and simplify setup, management, and administrative tasks. EDI is now a cornerstone of business communication between trading partners. Microsoft wanted to deploy a solution that could manage EDI messages DOI 10.1515/9781501505652-008

300 

 Chapter 8: Summary—Three Valuable BizTalk Azure Application Options

and automate business processes. This led to the development of BizTalk Server. The first version was launched in 2000 and was named BizTalk Server 2000. It included the BizTalk Orchestration Engine feature to manage business processes, the Mapper feature to allow data to be read by different systems, and the XML Editor to enable cross-platform communication. Over time, several new versions of BizTalk Server, such as BizTalk Server 2002, BizTalk Server 2004, BizTalk Server 2006, BizTalk Server 2009, and BizTalk Server 2010, were released, each with an updated list of features. One major improvement included in BizTalk Server 2013 came in the form of its cloud offering. In this version, Virtual Machines (VMs) became available on the Azure portal. Within the same year, Windows Azure BizTalk Services (WABS) was released. It provided built-in support for managing EDI relationships between trading partners, and setting up EAI bridges with on-premises systems. Later, BizTalk Server 2016 was launched in October 2016. It is one of the most commonly used versions of BizTalk Server, as it provides integration services, such as on-premise to cloud integration, cloud to cloud integration, hybrid integration, and Azure Logic Apps integration, etc. Nowadays, most industries are executing their applications on the cloud. They want to create and store artifacts on the Azure portal and create an independent integration solution through Azure Logic Apps. For this, they need to switch to Azure BizTalk Services, that is, Platform as a Service (PaaS).

BizTalk Integration Patterns Before delving into BizTalk Azure Application options, it was important for us to understand the integration patterns. So Chapter 2 outlined the integration patterns of BizTalk Server, such as Enterprise Application Integration (EAI), business workflow, and B2B integration. Enterprise Application Integration is a set of rules that integrate applications running in an enterprise into an automated business process. This integration pattern helps businesses save on different types of costs including maintenance, administrative, and operational costs. BizTalk Server provides a wide range of services in the EAI platform. Additional products/services, such as Microsoft Host Integration Server, Microsoft SQL Server, and Windows core services, can be integrated with BizTalk Server to enhance additional service requirements. Tasks like order processing can automatically run through BizTalk Server within a specified time frame; however, it does require a one-time configuration. A workflow describes a sequence of events that must be applied to the data received through the Receive Pipeline. It presents an executable business process

Features of BizTalk Server 2016 

 301

through XLANG/s language. The major elements of a workflow include the message, the operating actions, and the transmission ports. Business to Business (B2B) integration manages Electronic Data Interchange (EDI), Health Insurance Portability and Accountability Act (HIPAA), and XML. Essentially, EDI allows users to transmit and receive data in a standardized electronic format between business entities. The commonality of the format (aka the schema) allows us to quickly integrate and orient to business processes. BizTalk Server assists and integrates HIPAA (EDI in Healthcare Domain) rules through built-in EDI schemas and allows organizations to automate the business process so that the exchange of health-related documents remains secure. In fact, HIPAA legislation was enacted in 1996 in the United States to set standards for protecting sensitive data related to patients. BizTalk Server provides separate configurations for each trading partner. Therefore, organizations that are implementing BizTalk Server can easily manage multiple trading partners. Also, BizTalk’s ability to integrate third-party applications makes it a better choice for organizations to implement.

Features of BizTalk Server 2016 Chapter 3 discussed the first of the three BizTalk Azure Application options. It outlined the features of the latest version of BizTalk Server, that is, BizTalk Server 2016, which was released by Microsoft in December 2016. This version of BizTalk Server bundles many exciting features. The most notable features added to EAI include support for SAP.Net Connector (NCO), artifacts analytics tracking, Azure Logic Apps adapter, and the ability for BizTalk Server to connect to the SQL Server Always Encrypted column. BizTalk Server 2016 supports both classic RFC SDK and NCO (.Net Connector) to integrate SAP systems with BizTalk Server. BizTalk Server 2016 is integrated with the cloud-level analytics feature called Application Insights, which makes tracking data for each artifact available anytime. The analytics feature can be enabled on a single artifact as well as on a BizTalk group. BizTalk Server 2016 provides an Azure Logic Apps adapter used to enable communication between BizTalk Server and Azure Logic Apps hosted on the Azure portal. BizTalk Server receives messages from Azure Logics Apps through a connector called the BizTalk Connector. BizTalk Server 2016 allows users to connect to the Always Encrypted column of SQL Server. This column stores confidential data in an encrypted form. To allow BizTalk Server to connect to the SQL Server Always Encrypted column, you need to create an Always Encrypted column in SQL Server and enable the Always Encrypted option in BizTalk Server.

302 

 Chapter 8: Summary—Three Valuable BizTalk Azure Application Options

The most common features added to Business Process Control include map compilation, ordered delivery on dynamic Send Ports, and advanced location scheduling. BizTalk Server 2016 also provides the map compilation feature by allowing users to select the desired API (XslTransform or XslCompiledTransform) depending on their requirement. It solves the issues faced by earlier versions of BizTalk Server in performing map compilation. The ordered delivery on dynamic Send Ports feature states that the outbound port receives the next message only after the successful transmission of the previous message. In BizTalk Server 2016, you can set the advanced scheduling properties such as time schedule and recurrence schedule, in Receive Locations. The major features associated with B2B integration include a connection to Azure File share from the BizTalk File adapter, a refined B2B import/export process, and improved FTP and SFTP adapters. BizTalk Server 2016 also allows users to connect to Azure File share with the help of a File adapter. This can be done by performing many tasks, including creating an Azure Storage Account and a File share on it, configuring BizTalk ports to access Azure File share, and testing the BizTalk Server access to Azure File share. In BizTalk Server 2016, importing/exporting a party or an agreement does not require importing/exporting the entire application; separate commands can be used for performing unique operations. Some miscellaneous features of BizTalk Server 2016 include simultaneous configuration of multiple hosts, and the ability to simultaneously search artifacts and save multiple suspended messages, etc. BizTalk Server 2016 allows the configuration of multiple hosts concurrently. It also provides the search feature, which is used to navigate to an artifact directly instead of manually looking for it from the list of artifacts. In BizTalk Server 2016, multiple suspended messages can be saved to a folder at the same time. However, each suspended message creates its own separate file. BizTalk Server 2016 also provides the facility to send BizTalk operational data as an input/feed to Power BI. It also allows users to resize the Schema Picker/Type Picker window, which appears when the source schema/destination schema is either added or replaced during map creation. BizTalk Server 2016 as an application hosted on Azure Virtual Machines supports the production infrastructures very well.

BizTalk in Microsoft Azure Services 

 303

BizTalk in Microsoft Azure Services Chapter 4 discussed Microsoft Azure BizTalk Services (MABS). MABS is a cloudbased integration service that implements cloud and hybrid integration solutions through the common integration patterns of BizTalk Server, that is, Business to Business (B2B) and Enterprise Application Integration (EAI). There are two forms of Microsoft Azure BizTalk Services, including Infrastructure as a Service (IaaS), and Platform as a Service (PaaS). Microsoft provides a BizTalk Server Virtual Machine (VM) as IaaS to reduce the heavy configuration in setting up the BizTalk Server environment. Currently, support for new subscriptions of IaaS services is not available. The major reason behind stopping the support for IaaS services is switching the organizations to BizTalk PaaS services, which deal with Azure Logic Apps, integration account, and Azure App Service Hybrid Connections. Azure Logic App is a serverless service hosted on Azure that allows users to easily execute workflows in the cloud and provides accessible integration solutions. There are several connectors associated with Azure Logic Apps that can be used to connect to different enterprise applications. Integration account is a scaleable container that manages all the integration artifacts including trading partners, trading partner agreements, XML schemas, XSLT transforms (maps), and public/private key certificates. Qualifiers must be added to an existing trading partner to use it in more than one business process. In addition, agreements need to be added to an integration account that should be agreed and followed by both trading partners during inbound or outbound transactions. The integration account must be linked to Azure Logic App to allow it to access artifacts in the integration account. The integration account connectors can be used to expand BizTalk workflows into Azure. Some of the integration account connectors are AS2 encoding and decoding connectors, EDIFACT encode and decode connectors, EDI (X12), XML, and flat file connectors. A hybrid connection allows communication between websites/mobile apps and on-premise data/resources with the help of an agent located on the on-premise server. You can also implement an EAI scenario with the help of integration account connectors.

Web 3.0 as a BizTalk Alternate Chapter 5 talked about implementing a Web 3.0 solution as an alternative to BizTalk Server with the help of C# code. The Web 3.0 Service alternatives help you build out a typical BizTalk EAI and B2B processing environment.

304 

 Chapter 8: Summary—Three Valuable BizTalk Azure Application Options

B2B processing involves decoding EDI to XML and decoding flat file to flat file XML. EAI implementation includes performing XML validation and executing a set of steps in a workflow. Three major component groups of BizTalk B2B and EAI processing include messaging, transformation, and workflow. Pipeline processing deals with transmitting the messages across pipeline components, which are in fact .NET assemblies. As an alternative to BizTalk Server, you need to determine the business logic of the component and write a library for it. You can use different parsing techniques outside of BizTalk Server to process EDI/XML/flat file messages. EDI parsing is one option and is accomplished through the EDI disassembler component. This component performs several tasks including splitting or preserving the interchanges and transaction sets, identifying the schema and generating XML after validation. You can use an EDI parser as an EDI disassembler component. Some of the EDI parsers are X12Parser, EDIReader, BOTS, and EDIFabric. You could also use a transformation solution implemented in C# instead of using BizTalk Server. The process for this include generating a C# class from the source XSD, deserializing source file data into an instance of the created class, generating a C# class from the target XSD, creating an instance of the target object, performing object to object mapping, serializing target object into XML, and finally sending the generated XML as map output. As an alternative to BizTalk Orchestration Engine, you can create a workflow through the Azure Logic App and Windows Workflow. Several workflow engines are available on the web that add workflow functionality to the application through C# coding. Commonly used workflow engines are WorkflowEngine.NET and Wexflow. These alternatives can be used to implement an end-to-end process. To achieve BizTalk-like automation, you would need to create a Windows Service with a FileSystemWatcher component. This component fires an event when a file is dropped at its configured location. Later, the desired code can be written to specify the operations to be performed on the dropped file. The earlier scenario can be extended such that an EDI file is received, parsed to generate XML, and mapping is applied to it. For processing EDI messages, you would need an EDI parser to validate EDI and then disassemble it so that all the transactions in the EDI file can be processed individually. From this, you would get a complete EDI end-to-end solution through C#.Net code.

Compare BizTalk IaaS, BizTalk PaaS, and Custom Integration 

 305

Compare BizTalk IaaS, BizTalk PaaS, and Custom Integration Chapter 6 compared the integration models BizTalk Server 2016, BizTalk Azure services (BizTalk PaaS services), and BizTalk alternatives (custom integration through .Net) based on several parameters. The comparison between the integration models helps users in identifying the correct integration model for their organization. In the context of rapid integration, BizTalk IaaS is a reliable integration solution. BizTalk PaaS is a good choice for small integration while there is no such facility in custom integration. In the context of connectors/adapters, BizTalk Server supports several adapters including FTP, SFTP, MSMQ, SMTP, SOAP, and WCF. PaaS connectors/ adapters are an essential part of the BizTalk PaaS services and allow users to connect Azure Logic App to different applications including Outlook, Salesforce, CRM online, SharePoint, Twitter, DB2, and Oracle. On the other hand, there is no in-built connector for custom integration. However, they can be built in native C#. In the context of a workflow engine, BizTalk Server provides the BizTalk Orchestration Engine that monitors the entire lifecycle of the workflow. BizTalk PaaS services provide an Azure Logic App that is considered a workflow-based service on the Azure portal. In custom integration, the desired workflow engine can be downloaded from the web. One of the most commonly used workflow engines is Windows Workflow Foundation (WWF), which allows a developer to write a workflow-enabled service. In the context of scalability, BizTalk Server architecture can be used to scale (scale in or scale out) any BizTalk Server setup. In BizTalk PaaS, scalability can be managed by switching the desired tier among the five Free, Developer, Basic, Standard, and Premium tiers. In custom integration, scalability depends on several factors including the type of integration solution, the memory size required by executables, database size, and throughput. In the context of the tracking system, the MessageBox database in BizTalk Server helps in tracking each message and process running in its domain. Business Activity Monitoring (BAM) in BizTalk Server also tracks each artifact. The Azure diagnostics tool in BizTalk PaaS services tracks the business process data involved in any Azure BizTalk service and traces Azure Logic Apps workflow events. In the case of custom integration, you would need to build your own tracking system. In the context of EDI capabilities, all types of EDI documents can be processed more swiftly in BizTalk Server. In BizTalk PaaS, there is no specialized tool for processing EDI documents, however, Azure BizTalk services can store arti-

306 

 Chapter 8: Summary—Three Valuable BizTalk Azure Application Options

facts related to EDI processing. In custom integration, EDI parsers can be used to implement an EDI processing scenario with the help of SDKs. In the context of transactional support, BizTalk Server provides complete transactional support through the workflow engine (BizTalk Orchestration Engine). BizTalk PaaS services provide stateless transactional support. In custom integration, C# provides a built-in class (TransactionScope) for transaction management, which manages local and distributed transactions. For long-running transactions, BizTalk Server supports them completely. BizTalk PaaS services do not have built-in capability for long-running transactions. However, Azure Logic Apps can be designed to support them. In custom integration, you would also need to design the solution through coding to support long-running transactions. In the context of EAI, BizTalk Server provides a reliable EAI solution through its multi-protocol messaging, tracking, and reporting capabilities. In BizTalk PaaS services, Azure Logic Apps can be used to implement an EAI solution. In a custom integration, you would need to build connectors, set up a common communication mode, design the solution to parse the incoming messages, and then process and transform data into the format accepted by the system. BizTalk Server allows users to implement BRE policies, which can be used in BizTalk Orchestration Engine through the Call Rules shape. BizTalk PaaS does not have rules. However, Azure functions can be used as a rules engine by using the on-premise BizTalk BRE policies, and access them in Azure Logic Apps through BizTalk Management APIs. In custom integration, rules can be created independently and can be retrieved later from the database. In the context of development experience, BizTalk Server provides SDKs. These SDKs are used in Visual Studio to create BizTalk projects and provide an environment for development and testing. BizTalk PaaS services can be deployed on the Azure portal as well as in Visual Studio. In custom integration, the development is done through Visual Studio. Cost is one of the prominent factors while selecting an integration model. In BizTalk Server, pricing is based on the number of cores for each processor of the server system. It means you need to invest more if you want more computing power. In BizTalk PaaS, the implementation cost depends on the tier selected. Azure Logic Apps implementation account is also charged based on the number of actions executed. Additionally, you need to pay $1000/month for the integration account and enterprise connection charges of $800 per connection per month. In custom integration, you need to pay only for third-party resources. In the context of maintenance, BizTalk Server needs a regular resource for maintenance support as BizTalk Server is used in industries that require regular interaction with different enterprises. For BizTalk PaaS, a developer is needed

Microsoft Partners in BizTalk Domain 

 307

who has a good understanding of BizTalk and Azure, which costs around $12,000–$15,000 per month. In custom integration, a reasonably experienced C# developer is required for designing an end-to-end business scenario costing around $8,000–$10,000 per month.

Microsoft Partners in BizTalk Domain Chapter 7 reviewed the Microsoft partners and organizations that are available in the BizTalk domain, and who assist BizTalk Server in functioning properly. We discussed three organizations, namely BizTalk360, WPC, and Neudesic. BizTalk360 is a support product introduced by Saravana Kumar in 2010 to solve the issues related to operating, monitoring, and administering BizTalk Server. It has several features including environment management, user access policy, monitoring and notification, analytics health, BizTalk360 health, BizTalk360 health monitor, and advanced event viewer, etc. All these features can be categorized into four categories including Operations, Monitoring, Analytics, and Extras. The Operations console is a single console in BizTalk360 that is used for monitoring all operations which require separate consoles for performing different operations in BizTalk Server. BizTalk360 Monitoring Dashboard solves all issues faced by BizTalk Server while monitoring the status, database health, and memory and other resources of the connected applications. It allows users to add alerts to individual BizTalk components and manage alarms. BizTalk360 Analytics service allows users to perform a process and performance analysis by enabling analytics for different kinds of artifacts and components, such as SQL Server and IIS. WPC (Washington Publishing Company) is a Microsoft partner company that deals in the healthcare domain and provides services related to healthcare documentation. It provides the FirstPass HIPAA transaction testing tool and Windows desktop application that validates X12 EDI data to X12 syntax. WPC-EDI developed a BizTalk adapter called HIPAA Database Toolkit (DBToolKit) that supports both versions of EDI transactions and processes them to insert the transaction data into the SQL Server database. The schema generator feature of DBToolKit creates a DDL script for each HIPAA EDI transaction, which in turn creates a database when you run the script in SQL Server. DBToolKit allows users to save the complete transaction data without writing any code and with very few configurations. Neudesic, LLC is also a Microsoft partner company that deals in software products and services related to Enterprise Service Bus (ESB) integration, enterprise social software, Microsoft technology solutions, and CRM solutions.

308 

 Chapter 8: Summary—Three Valuable BizTalk Azure Application Options

ESB is an architecture that helps users in implementing a service-oriented infrastructure. It controls the way different applications communicate with each other in a loosely coupled manner. BizTalk ESB Toolkit contains several tools used to develop enterprise-level applications where a loosely coupled message environment can be created. BizTalk ESB Toolkit components include ESB web services, ESB management portal, ESB pipeline interop components, Exception management framework, ESB resolver and adapter provider framework, Itinerary based routing and processing, and ESB Toolkit sample applications. These components can be used in a BizTalk application to implement the BizTalk ESB solution.

Conclusion You now understand information required for BizTalk Server implementation in an organization. Organizations have several options besides implementing BizTalk Server. They can opt for BizTalk PaaS services to contain all work on the Azure portal so services are always available from anywhere. Implementation of any solution depends entirely upon the requirements of an organization. If an organization does not want to invest more, it can develop a C# solution that requires a specialized programmer with adequate programming skills. If an organization opts for BizTalk Server, it will likely require the help of Microsoft partners who provide several tools to perform the tasks easily. BizTalk360 is one of the tools that has solved the issues related to operating, monitoring, and administering BizTalk Server.

Index A AAA 43, 45, 48 Access file share 90, 94 Access keys 94–95, 97 ACK Location 200–201 Acknowledgments 12, 14–15, 33, 37–38, 43, 126, 133, 157, 199 Action configuration 71–72 ActiveX Data Object (ADO) 26 Adapter providers 207 Adapters 12–13, 16–17, 26, 28–29, 102, 216–17, 252, 261–62, 305 – install Azure Logic Apps 66 – registered Microsoft BizTalk Server 207 Add Application option 66–67 Add button 121, 124, 129–30, 164 Add New Azure Subscription 238–39 Administrator/ReceiveWCFService/ service 74–75 ADO (ActiveX Data Object) 26 Agreements 28, 32–33, 98, 122–24, 129, 215, 239, 302–3 – license 66, 253, 266–67 Alarms 247–49, 307 American National Standards Institute (ANSI) 10, 32 Analytics 21–22, 53, 56–58, 60, 230, 249, 307 Analytics Environment Settings 249–50 Analytics feature 57–58, 301 Analytics on artifacts 53, 57 ANSI (American National Standards Institute) 10, 32 Application and system integration services 264 Application configuration file 164–65 Application Insights 21–22, 52–55, 301 – new instance of 53 Application Insights for BizTalk artifacts 20 Application Insights instance 53–55, 59–60 Application Insights option 54, 56 Application integration capabilities 203–4 Application pool 67 Application pool identity 67 DOI 10.1515/9781501505652-009

Application services 26, 220 Application support 232 ApplicationPool 107, 109 Applications 3, 25–26, 37, 157, 159–60, 204, 217–20, 246–47, 274–78 – distributed 3 – e-commerce 8–9, 299 – exe 222 Archive 43–44, 161–62, 164–65, 170, 192, 195 Archive location 161–62, 176 Archive source file 171 ArchiveReceivedData location 176–77 Artifacts 51–53, 56–57, 60, 106–7, 142–43, 217–18, 232, 278, 300–302 AS2, 10, 15–16, 117, 125, 128, 133–34 AS2 adapter 16 Assemblies 6, 107, 112, 180–81, 207 AuthNet 35–36, 43 AuthNet application 39, 42–45 Azure 19, 21–24, 113–16, 140, 220, 224–25, 299, 303, 307 Azure BizTalk Rules Service 209 Azure BizTalk service and traces Azure Logic Apps workflow events 305 Azure BizTalk Services 23, 116, 213, 219, 228, 300, 305 Azure BizTalk Services and integration accounts 215 Azure diagnostics tool in BizTalk PaaS services tracks 305 Azure File share 20, 51, 90, 94, 96, 114, 302 Azure File Storage Account 90–92 Azure File Storage Share 90, 97 Azure Logic Apps 65–66, 68–69, 74–77, 116–18, 133–34, 143–44, 215–18, 223–25, 300–301 Azure Logic Apps Tools 139–40 Azure portal 53, 59–62, 64–65, 117–18, 138, 209–10, 218, 300–301, 305–6 Azure services 22–23, 115, 138, 161, 232, 238 Azure VMs 19, 22, 61, 104, 113, 154

310 

 Index

B BAM (Business Activity Monitoring) 17, 212–13, 235–36, 305 Base EDI adapter 14–15 Binding file 98, 100–101 BizTalk Admin Console 42, 83, 227, 231, 242, 278, 297 BizTalk alternatives 155, 157, 159, 203, 305 BizTalk Analytics service 307 BizTalk application, configure ESB 270 BizTalk Azure applications 227–28, 230, 232, 234, 236, 238, 240, 242, 300–301 BizTalk Azure services 138–39, 141, 143, 145, 147, 149, 151, 153–54, 219 BizTalk Cloud services 297 BizTalk components 247, 307 BizTalk Connector 66–67, 217, 301 BizTalk developers 13–14, 115, 220 BizTalk environment 21, 107, 109, 115, 220, 229, 232 BizTalk ESB applications 277 BizTalk ESB Implementation 277, 279, 281, 283, 285, 287, 289, 291, 293 BizTalk ESB Itinerary Designer 283 BizTalk ESB Toolkit 265, 308 BizTalk ESBTestApp application 282 BizTalk file dapter 21, 51, 90, 114, 302 BizTalk Flat File Schema Wizard 17 BizTalk group 53, 56, 98, 231–32, 301 BizTalk Group option 56, 100 BizTalk Health Monitor 20, 228–29, 240, 307 BizTalk hosts 30, 243 BizTalk IaaS 203–4, 206, 208, 210, 212, 214, 216, 218–22, 305 BizTalk IaaS/BizTalk Server 204, 207–8, 210–12, 215–18, 220–21 BizTalk in Microsoft Azure Services 115–16, 118, 120, 122, 124, 126, 128, 130, 303 BizTalk integration patterns 25–26, 28, 30, 32, 34, 36, 38, 40, 42 BizTalk Integration Platform as a Service 115–17, 119, 121, 123, 125, 127, 129, 131, 133 BizTalk Management Service 108 BizTalk message 28 BizTalk messaging 27–28 BizTalk operation service URL 111

BizTalk operation team 235 BizTalk operations 231–32 BizTalk Orchestration Designer 30–31 BizTalk Orchestration Engine 11, 22, 31, 204–5, 208, 210–11, 287, 300, 304–6 BizTalk PaaS 203–4, 206–25, 305–6 BizTalk PaaS services 23, 206, 210–11, 213–14, 217, 223, 303, 305–6, 308 BizTalk PaaS services support Azure Logic Apps 204 BizTalk processes 217, 220, 248 BizTalk projects 139–40, 217, 306 BizTalk Server – on-premises 20, 73, 217, 297 – orchestration in 30, 210 BizTalk Server Administration Console 21, 100, 105, 272 BizTalk Server applications 265 BizTalk Server setup 218, 305 BizTalk services 23, 116, 154 BizTalk services in Azure 220 BizTalk Settings Dashboard 56, 105 BizTalk templates 140 BizTalk Type Picker 112–13 BizTalk workflows 133, 159, 303 BizTalkManagementService, named 108 BizTalkOperationalData.pbix file 111 BizTalkOperationalDataService service 110–11 BOTS 156–57, 304 BPM (Business Process Management) 11, 25 BRE (Business Rules Engine) 207–8 BRE policies 207–9, 306 BTS 281, 283, 295–96 BTS configuration file 41 Business Process Control Integration 84–85, 87, 89 Business processes 17, 25–26, 30, 117, 122, 235, 242, 299–301, 303 Business profiles 98–100, 239 Business Rules Engine (BRE) 207–8 Business to business 25, 32–33, 51, 301, 303 Business to Business Integration 90–91, 93, 95, 97, 99, 101, 103 C Classes 6, 158–59, 166–68, 170, 184, 192, 212, 304, 306

Index  Clearing House 33–36 Cloud 20, 23, 115, 117, 131, 154, 225, 300, 303 Cloud integration 19–20, 300 Code 41, 47, 169–72, 195–96, 202, 205, 209, 303, 307 Commands 101–2, 107–9, 274 Compare Azure BizTalk Applications 203–4, 206, 208, 210, 212, 214, 216, 218, 220 Component Installation page 267–68 Component Object Model 1, 26, 29, 299 Components 1–6, 28–29, 155–56, 160–61, 249, 265, 299, 304, 307–8 Configuration 64, 74, 86, 96–97, 269–70, 274, 300, 302, 307 – simultaneous 51–52, 104–5, 114, 302 Configuration file 161–62, 164, 170, 173, 176, 184, 195–96, 200–201, 207 Configuration settings 105, 126, 237, 268 Configuration values 164–65 Configure 38, 42, 61, 73, 94, 96, 237–38, 240, 284 Configure BizTalk ports 90, 94 Configured location 161, 163, 170, 176–77, 195, 201, 304 Configuring 38, 42, 58, 61–62, 64, 80, 96, 105, 157 Configuring BizTalk Ports 94, 302 Configuring ESB BizTalk Applications 270–71 Connection string 41, 207, 261–62 Connections 5, 16, 35, 53, 114, 149, 206, 302, 306 Connectors 61, 117, 133–34, 138, 204, 215–17, 291, 301, 305–6 Context menu 56–57, 66–67, 78, 98, 100–101, 110, 163–64, 268, 292–93 Core licenses 221–22 Covast 13–14, 16 Covast AS2 Adapter 13, 16 Covast EDI Accelerator for BizTalk Server 13 Cumulative update packs (CUPs) 18–19 CUPs (cumulative update packs) 18–19 Custom BizTalk Azure application 155–56, 158, 160, 162, 164, 166, 168, 170, 172 Custom integration 203–4, 206, 208–10, 212, 214–22, 224–25, 305–7

 311

D Data gateway 61, 63, 66, 73, 147, 217 Data gateway resource 61, 64–65 Data monitoring 248–49 Data warehouse 35–37 Database 27, 77–78, 138, 147–48, 197–99, 209, 256–57, 261–62, 275 Database objects 253, 256–57 Database server 41, 268 Database Toolkit Adapter 259, 262, 297 DBToolKit Transport Properties dialog box 261 DCOM (Distributed Component Object Model) 1, 4–6, 299 DCOM components 5–6 DDL script 253, 255–56, 261, 307 Decode 118, 136–38, 143, 146 Decode connector 133–36, 303 Deployment 4, 21–22, 42, 120, 217 Developers 2–3, 8, 12, 85, 210, 219–20, 223, 234, 305–7 Development 8, 17, 154, 217–18, 225, 278, 300, 306 DLL Surrogates 6 Domain 107, 109, 212, 305 Dynamic Send Ports 19–20, 51, 84, 87, 114, 280–81, 289, 292, 302 E EAI (Enterprise Application Integration) 12, 23, 25–27, 34–35, 51, 114–15, 204, 299–301, 303 EAI Implementation 138–39, 141, 143, 145, 147, 149, 151, 153, 304 EAI Integration 52–53, 55, 57, 59, 61, 63, 65, 67, 69 EDI (Electronic Data Interchange) 8–11, 32–34, 125, 128, 136, 157, 191–92, 215–16, 256–57 EDI data 10, 249, 252–53, 307 EDI disassembler component 156, 304 EDI documents 16, 34, 157, 181, 215, 305 EDI files 14, 156, 178–79, 181, 187, 199, 225, 262–63, 304 – received 186, 189 EDI format 14, 34, 189, 192 EDI parsers 156, 178–79, 183, 215, 304, 306

312 

 Index

EDI parsing 156, 304 EDI processing 179, 184, 215, 306 EDI processing and persistence 178–79, 181, 183, 185, 187, 189, 191, 193, 195 EDI standards 9, 11, 157 EDI transactions 15, 225, 252–53, 307 EDIFabric 156–58, 179, 183, 186, 192, 304 EDIFabric parser 178–79, 181, 216 EDIFACT 10–11, 14, 27, 117–18, 125, 128, 136–37, 157, 215–16 EDIFACT Messages 135–36, 138 EDIReader 156–57, 182, 186, 192, 304 EIP (Enterprise Integration Pack) 118, 216 Employees 78, 138, 141, 143, 148, 150, 152–54 Encode connector 133–34, 136 Encoding 117, 133–34 Encrypted column 21, 51–52, 77–78, 114, 301 Encrypted column in SQL Server 77–78, 301 Encrypted setting 83–84 Encrypted wizard 78–82 Endpoints 61, 107, 115, 207 End-to-End Transactional Support 211 Enterprise Integration Pack (EIP) 118, 216 Enterprises 25–26, 35, 37, 49, 113, 204, 227, 300, 306–7 ESB (Enterprise Service Bus) 25, 207, 237, 264–65, 269, 278–79, 281, 283, 307–8 ESB applications 278–79 ESB BizTalk Applications 272 ESB BizTalk Applications configuration 270 ESB BizTalk Applications option 270 ESB Configuration Tool window 268–71 ESB itinerary application 283 ESB Management Portal 265, 272–73, 277, 308 ESBSource.zip file 273 Event viewer, advanced 228, 230–31, 236–37, 307 Evolution of BizTalk 2, 4, 6, 8, 10, 12, 14, 16, 18 Export 98–101 Export Bindings 99, 101 Export Model option 293–94 Export process 114, 294–95

F Feature pack 21–22, 53, 107 FeaturePack.ConfigureServices.ps1, 107, 109 File – dropped 161–62, 171, 189, 195, 197, 304 – input.xml 295–96 – installation 253 – received 161, 186–88, 198 – selected 129–30 – source 158, 178, 202 File processing service 185–86 File share 90, 92–94, 97, 302 – connect to Azure 20, 90 File Transfer Protocol. See FTP Files option 92, 106 – application configuration 164 Filesharebackup 96 – named 94, 96 Filesharebackup storage location 97 FileSystemWatcher 163, 168, 171, 191 FileSystem-Watcher 171 FileSystemWatcher component 161, 163, 165, 304 Filters 29, 48, 214–15, 262, 281, 284, 286–87 Flat file 14, 27, 34, 117, 133, 143, 155, 176, 304 Flat file data 139, 141, 145–46, 178 – received 168 Flat file decoding connector 139, 143 Flat file message 137–39, 156 Flat file Orders schema 166–67 Flat file schema 141–43 Folder location, destination 96, 254 FTP (File Transfer Protocol) 9, 15, 20, 26–27, 102–3, 216, 305 FTP Server Type property 102–3 FTP Servers 102–3 G Gateway 62–64, 70, 73, 149 Graphical user interface (GUI) 98, 160 GS segment 183, 192–94 GUI (Graphical user interface) 98, 160 H Health insurance portability 33, 301

Index  HIPAA (Health Insurance Portability and Accountability Act) 32–33, 157, 181, 252, 301 HIPAA Database Toolkit 227, 252–53, 255, 257, 259, 261, 263–64, 307 HIPAA Database Toolkit Adapter 253 Host instances 86, 105, 236, 242, 248 HWS (Human Workflow Services) 12–13 Hybrid connections 116, 138, 146–47, 303 I IaaS 23, 113, 115, 154, 224–25, 303 IaaS services 23, 303 IETF (Internet Engineering Task Force) 16, 133 IIS (Internet Information Services) 7, 65–67, 107, 115, 213, 249, 272, 274, 276 Import 98, 100–101 Infrastructure setting section 242–43, 245 Install 3, 61–62, 66, 110, 139, 173, 253, 267, 274 Install Microsoft BizTalk ESB Toolkit option 266 Install OrderProcessingService 173 Installation 18, 53, 62, 66, 115, 140, 174–75, 266, 268 Integration 17–18, 26, 210, 220–21, 264, 299–300, 302, 305, 307 Integration account 116–21, 123, 129–33, 138–40, 142–43, 146, 214–15, 303, 306 – existing 121, 123, 129–30 Integration account connectors 117, 133, 138, 303 Integration patterns 25, 51, 114, 300 Integration services 1, 20–21, 297, 300 Integration solution 25, 117, 218–19, 225, 305 Interface 2–5, 29, 184, 231, 242 Internal Settings section 127–28 Internet Engineering Task Force (IETF) 16, 133 InvoiceFilter filter 286–87 Invoices 277–79, 281, 289, 295 Invoices type messages/documents 290–91 ISA segment 183, 192–94 Itinerary 207, 266, 278–79, 284, 292–93, 308 Itinerary Broker Service 284–85 Itinerary designer area 283, 285, 287, 292–93 Itinerary service 234, 288–92

 313

J, K JSON 73, 148, 158–59 L Languages 2, 5, 34, 218 – programming 2, 5 Leverage Microsoft Product Support Services 220 Limitations 15, 157–58 Line of Business (LOB) 17, 203, 216 Link, localhost/BizTalkOperationalDataService 109 List, drop-down 71, 73, 75, 79, 119, 121, 123–24, 132, 148–49 LOB (Line of Business) 17, 203, 216 Location, outbound 198, 289, 292 Location transparency 6 Logic App Designer 69–71, 134 Logic app workflow 134, 136–37, 144, 151, 153–54 Logic apps 66, 73–74, 117, 131–32, 136, 147 Logic Apps option 68, 75, 139 LogicApp Adapter 66–67 LogicAppAdapter.iso file 66 Long-running transactions 203–6, 306 M MABS (Microsoft Azure BizTalk Services) 115, 303 Management 8, 16, 27, 66–67, 299 Management and ReceiveService applications on IIS 65–66 Management IIS application 66–67 Management REST APIs 20, 104, 107, 109 Management service 66–67 Management service application pool 73 Map compilation 19, 51, 84–86, 114, 302 Map output 145–46, 148, 150–52, 158–59, 191, 304 Mapping 161–62, 178, 189, 191, 304 – object-to-object 159, 166, 168, 170, 172 Maps 13, 40–41, 85, 129, 138, 140–41, 143, 215, 217 MDAC (Microsoft Data Access Components) 26 MDNs (Message Disposition Notifications) 16, 133

314 

 Index

Medical House 36–39, 42–43, 45–47 Message Disposition Notifications (MDNs) 16, 133 Message formats 9, 28, 216 Message outbound location 278, 292 Message type 71, 125, 207, 277–78, 281, 284–85, 291, 295–97 MessageBox database 28–30, 87, 212, 305 Microsoft BizTalk Adapter 252 Microsoft BizTalk ESB Toolkit 266–67 Microsoft BizTalk Server 16, 22, 25–26, 34, 66–67, 77, 107, 266 Microsoft BizTalk Server Adapter for Logic Apps 66 Microsoft Business Rule Composer 208–9 Microsoft Commerce Server 8, 299 Microsoft Data Access Components (MDAC) 26 Microsoft Host Integration Server 26, 300 Microsoft Merchant Server 7, 299 Microsoft Partner Company 252, 264, 307 Microsoft Partners 307–8 Microsoft Report Viewer application 274 Microsoft Site Server 6–8, 299 Microsoft Site Server Commerce (MSSC) 7, 299 Microsoft XML web services 26 Microsoft’s BizTalk Server 1, 25 MSSC (Microsoft Site Server Commerce) 7, 299 MTS (Microsoft Transaction Server) 2–3 MTS Explorer 3–4 Multiple hosts/host instances 51–52, 104–5, 114 Multiple suspended messages 51–52, 104, 106, 114, 302 N Name, desired 64, 92, 98, 119, 121, 124, 129, 162, 164 Name text box 92, 119, 121, 124, 129–30, 162 Navigate 66–67, 94, 107, 109, 148, 173, 273, 302 NCo 51–52, 77, 114, 301 NET Connector 21, 51–52, 77, 114, 301 NET Framework 18–20

Neudesic 227, 264–65, 267, 269, 271, 273, 275, 297, 307 Next button 62, 79–80, 253–54, 266–67 O ODX file 39–40, 42 Off-Ramp service 289–90 On-premises data gateway 61–62, 64–65, 147 Operational data 20–21, 104, 108–9, 111–12, 302 Operations 1, 143, 161, 208, 230–31, 268, 304, 307 Orchestration 25–26, 29–32, 36, 39, 53, 56, 155, 159, 232 Orchestration Design Surface 31–32 Order Processing Service 173–76 Ordered delivery 19, 51, 87, 114, 302 Ordered delivery on dynamic Send Ports 21, 51 302 OrderProcessing.WindowsService.exe file 173 Orders 6, 8–9, 27, 165–67, 277–79, 281, 289, 291–92, 295–96 Orders schema 281–82 – processed 166–67 Orders type documents 289–90 Orders XML schema 166–67 OrdersFilter filter 286–87 OrderTransform class 170 Outbound message data 191–92, 195–96, 198–99 Outgoing EDI File 194 Output 108–9, 148, 162, 165, 178–79, 183, 191–92, 274, 294–95 Output data 142, 162, 169–70 Output file 177–78 Output location 176–77, 187, 192, 197, 296–97 P Parsing 156, 178, 183, 189, 224 Parties and Business Profiles 98–100 Partner Solutions for BizTalk Azure Applications 227–28, 230, 232, 234, 236, 238, 240, 242, 244

Index  Persistence 178–79, 181, 183, 185, 187, 189, 191, 193, 195 Pipeline components 155, 304 Port 28, 37–39, 42–46, 83–84, 94, 96–97, 259–60, 277–80, 287 Port surfaces 32 Portal 152, 209, 215 Power BI 20–21, 52, 104, 108–9, 302 Processing type 179, 184, 192 Properties 28–29, 34, 127–28, 136, 204, 207, 261–62 Properties dialog box 57, 286 Protocols 4–5, 9–10, 16, 26–27, 216 Q Qualifiers 121–23, 303 Queries 84, 138, 234–35, 237, 239, 255, 263–64 R Received file data 172, 180, 188, 197–98 ReceiveService Applications 67 ReceiveService applications on IIS 65–66 Remote Procedure Call (RPC) 4, 299 Request trigger 134, 136, 144 Resolver 278, 285, 288–89 Response data 145–46 REST APIs 107–8 RIA (Rich Internet Application) 227 Rich Internet Application (RIA) 227 Routing 28, 203, 207, 281, 295 RPC (Remote Procedure Call) 4, 299 S SAP, support for 51–52, 77, 114, 301 SAP systems 77, 116–17, 301 Scalability 25, 203, 218–19, 305 Schema files 130–31 Schema generator file 255–56 Schemas 34, 71, 73, 125–26, 130–31, 139, 141, 166–68, 281–82 Schemas section 126 Script 109, 253, 256–57, 274–76, 307 SDKs 215, 217, 221, 306 Search BizTalk Artifacts 232–33 Send Port 16, 29, 38, 42, 44–45, 48, 259–61, 280–81, 289

 315

– ordered delivery on dynamic 20, 51, 84, 87, 114, 302 Send Port Properties 87, 261, 281 Sequence 42, 87–88, 161, 182–83 Service configuration file 176 Service Oriented Architecture (SOA) 25, 265 Service Pack (SPs) 6, 8, 12 Settings 64–66, 72, 88, 94–96, 115, 124–25, 128–31, 300, 303 SFTP (Secure FTP) 20, 26–27, 102–3, 216, 305 SFTP adapters 51, 90, 102–3, 114, 302 SOA (Service Oriented Architecture) 25, 265 SPs (Service Pack) 6, 8, 12 SQL connectors 147–48 SQL script file 256 SQL Server 18, 20, 22–23, 51, 77–78, 114–16, 146–48, 255–56, 301 – connect to 21, 51–52, 77 – on-premises 146–48, 154 SQL Server database 51, 77, 146–47, 210, 252–53, 262, 264, 307 Standards 4, 9–11, 29, 32 Stateless End-to-End Process Implementation 161, 163, 165, 167, 169, 171, 173, 175, 177 Storage account 90–92, 94 Subscriptions 28–29, 62, 64, 119, 238 Support 15–16, 18–21, 23, 77, 113–14, 206–7, 300–301, 303, 306 Support for Visual Studio 18–19 Support forum, active 157 SWIFT (Society for Worldwide Interbank Financial Telecommunications) 10, 18, 215 System.Transactions namespace 212 T Tables 21–22, 51–52, 77–78, 98, 135, 148–49, 186–88, 257, 263 Target object 158–59, 161–62, 170, 304 TDCC (Transportation Data Coordinating Committee) 9–10 Tracking, artifacts analytics 51–52, 114, 301 Tracking data 22, 29, 52–53, 249, 301 Tracking system 203, 212–14, 305

316 

 Index

Trading partners 9–10, 20, 23, 32–33, 121, 123, 277, 299–301, 303 – existing 122–23, 303 Transaction data 183, 186, 189–90, 194, 198, 252, 307 Transaction rollback 205 Transaction sets 109, 136, 156, 182–83, 197, 304 Transactional support 211–12, 306 Transactions 32–34, 179–80, 182, 189–91, 202, 204–6, 210–11, 225, 256 Transformed data 147, 161, 165, 170 Trigger 61, 69–70, 76–77, 143, 152 Types, transaction 126, 205, 211 U UCC (Uniform Code Council) 10 Uniform Code Council (UCC) 10 URL 66–67, 72–73, 76, 108, 139, 144–45, 151 User interface 13, 108 V Validation 14–15, 125, 133, 156, 182, 282, 293, 304 Valuable BizTalk Azure Application Options 299, 302, 304, 306, 308 Valuable BizTalk Azure Application Options and automate business processes 300 Values 28, 74, 76, 102–3, 126, 138, 149, 178, 293–94 – desired 93, 107, 109, 121, 123 Visual Studio 12, 16–20, 51, 114–15, 139–40, 217–18, 274, 283, 306 Visual Studio Team Services (VSTS) 21–22 VM (Virtual Machines) 19, 52, 61, 113–16, 300, 302–3 VSTS (Visual Studio Team Services) 21–22

W WABS (Windows Azure BizTalk Services) 23, 300 Wexflow 160, 304 Window, file service 92 Windows NT 4, 6 Windows Service 161–65, 168, 173–75, 304 Windows Service name 163 Windows Services list 174–75 Workflow 25–26, 29, 69, 73–74, 143–47, 155, 159–60, 210, 304–5 Workflow engines 159–60, 204, 211, 304–6 Workflow support 203, 210 WPC (Washington Publishing Company) 227, 252, 254, 297, 307 WSDL (Web Services Description Language) 29 WWF (Windows Workflow Foundation) 18, 210, 305 X, Y, Z XDR schemas 15 XML 32–34, 36, 97–98, 148, 151–52, 155–58, 178, 191–92, 303–4 XML data 141, 146, 170, 192, 253 XML format 145–47, 161 XML schema 118, 141–42, 303 XML Schema Definition (XSDs) 12, 29, 34, 158 XMLSerializer class 168–70 XSDs (XML Schema Definition) 12, 29, 34, 158