Data Management at Scale: Best Practices for Enterprise Architecture [1 ed.] 149205478X, 9781492054788

Table of contents :
Cover
Copyright
Table of Contents
Foreword
Preface
Navigating Through This Book
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Chapter 1. The Disruption of Data Management
Data Management
Analytics Is Fragmenting the Data Landscape
Speed of Software Delivery Is Changing
Networks Are Getting Faster
Privacy and Security Concerns Are a Top Priority
Operational and Transactional Systems Need to Be Integrated
Data Monetization Requires an Ecosystem-to-Ecosystem Architecture
Enterprises Are Saddled with Outdated Data Architectures
Enterprise Data Warehouse and Business Intelligence
Data Lake
Centralized View
Summary
Scaled Architecture
Chapter 2. Introducing the Scaled Architecture: Organizing Data at Scale
Universally Acknowledged Starting Points
Each Application Has an Application Database
Applications Are Specific and Have Unique Context
Golden Source
There’s No Escape from the Data Integration Dilemma
Applications Play the Roles of Data Providers and Data Consumers
Key Theoretical Considerations
Object-Oriented Programming Principles
Domain-Driven Design
Business Architecture
Communication and Integration Patterns
Point-to-Point
Silos
Hub-Spoke Model
Scaled Architecture
Golden Sources and Domain Data Stores
Data Delivery Contracts and Data Sharing Agreements
Eliminating the Siloed Approach
Domain-Driven Design on an Enterprise Scale
Read-Optimized Data
Data Layer as a Holistic Picture
Metadata and the Target Operating Model
Summary
Chapter 3. Managing Vast Amounts of Data: The Read-Only Data Stores Architecture
Introducing the RDS Architecture
Command and Query Responsibility Segregation
What Is CQRS?
CQRS at Scale
Read-Only Data Store Components and Services
Metadata
Data Quality
RDS Tiers
Data Ingestion
Integrating Commercial Off-the-Shelf Solutions
Extracting Data from External APIs and SaaSs
Historical Data Service
Design Variations
Data Replication
Access Layer
File Manipulation Service
Delivery Notification Service
De-Identification Service
Distributed Orchestration
Intelligent Consumption Services
Populating RDSs on Demand
RDS Direct Usage Considerations
Summary
Chapter 4. Services and API Management: The API Architecture
Introducing the API Architecture
What Is Service-Oriented Architecture?
Enterprise Application Integration
Service Orchestration
Service Choreography
Public Services and Private Services
Service Models and Canonical Data Models
Similarities Between SOA and Enterprise Data Warehousing Architecture
Modern View on SOA
API Gateway
Responsibility Model
The New Role of the ESB
Service Contracts
Service Discovery
Microservices
The Role of the API Gateway Within Microservices
Functions
Service Mesh
Microservices Boundaries
Microservices Within the API Reference Architecture
Ecosystem Communication
API-Based Communication Channels
GraphQL
Backend for Frontend
Metadata
Using RDSs for Real-Time and Intensive Reads
Summary
Chapter 5. Event and Response Management: The Streaming Architecture
Introducing the Streaming Architecture
The Asynchronous Event Model Makes the Difference
What Do Event-Driven Architectures Look Like?
Mediator Topology
Broker Topology
Event Processing Styles
A Gentle Introduction to Apache Kafka
Distributed Event Data
Apache Kafka Features
The Streaming Architecture
Event Producers
Event Consumers
Event Platform
Event Sourcing and Command Sourcing
Governance Model
Business Streams
Streaming Consumption Patterns
Event-Carried State Transfer
Playing the Role of an RDS
Using Streaming to Populate RDSs
Controls and Policies for Guiding the Domains
Streaming as the Operational Backbone
Guarantees and Consistency
Consistency Level
“At Least Once, Exactly Once, and at Most Once” Processing
Message Order
Dead Letter Queue
Streaming Interoperability
Metadata for Governance and Self-Service Models
Summary
Chapter 6. Connecting the Dots
Recap of the Architectures
RDS Architecture
API Architecture
Streaming Architecture
Strengthening Patterns
Enterprise Interoperability Standards
Stable Data Endpoints
Data Delivery Contracts
Accessible and Addressable Data
Crossing Network Principles
Enterprise Data Standards
Consumption-Optimization Principles
Discoverability of Metadata
Semantic Consistency
Supplying the Corresponding Metadata
Data Origination and Movements
Reference Architecture
Summary
Chapter 7. Sustainable Data Governance and Data Security
Data Governance
Organization: Data Governance Roles
Processes: Data Governance Activities
People: Trust and Ethical, Social, and Economic Considerations
Technology: Golden Source, Ownership, and Application Administration
Data: Golden Sources, Golden Datasets, and Classifications
Data Security
Current Siloed Approach
Unified Data Security for Architectures
Identity Providers
Security Reference Architecture and Data Context Approach
Security Process Flow
Practical Guidance
RDS Architecture
API Architecture
Streaming Architecture
Intelligent Learning Engine
Summary
Chapter 8. Turning Data into Value
Consumption Patterns
Using Read-Only Data Stores Directly
Domain Data Stores
Target Operating Model
Data Professionals as a Target User Group
Business Requirements
Nonfunctional Requirements
Building the Data Pipeline and Data Model
Distributing Integrated Data
Business Intelligence Capabilities
Self-Service Capabilities
Analytical Capabilities
Standard Infrastructure for Automated Deployments
Stateless Models
Prescripted and Configured Workbenches
Standardize on Model Integration Patterns
Automation
Model Metadata
Advanced Analytics Reference Architecture
Summary
Chapter 9. Mastering Enterprise Data Assets
Demystifying Master Data Management
Master Data Management Styles
MDM Reference Architecture
Designing a Master Data Management Solution
MDM Distribution
Master Identification Numbers
Reference Data Versus Master Data
Determining the Scope of Your Enterprise Data
MDM and Data Quality as a Service
Curated Data
Metadata Exchange
Integrated Views
Reusable Components and Integration Logic
Data Republishing
Relation to Data Governance
Summary
Chapter 10. Democratizing Data with Metadata
Metadata Management
Enterprise Metadata Model
Enterprise Knowledge Graph
Architectural Approaches for Metadata Management
Metadata Interoperability
Metadata Repositories
Marketplace to Provide Rapid Access to Authorized Data
Summary
Chapter 11. Conclusion
Delivery Model
Fully Decentralized Approach
Partially Decentralized Approach
Structuring Teams
InnerSource Strategy
Culture
Technology Choices
The Decline of Traditional Enterprise Architecture
Blueprints and Diagrams
Modern Skills
Control and Governance
Last Words
Glossary
Index
About the Author
Colophon