Driving Data Quality with Data Contracts: A comprehensive guide to building reliable, trusted, and effective data platforms 9781837635009, 1837635005

Table of contents :
Cover
Title Page
Copyright and Credits
Dedication
Foreword
Contributors
Table of Contents
Preface
Part 1: Why Data Contracts?
Chapter 1: A Brief History of Data Platforms
The enterprise data warehouse
The big data platform
The modern data stack
The state of today’s data platforms
The lack of expectations
The lack of reliability
The lack of autonomy
The ever-increasing use of data in business-critical applications
Summary
Further reading
Chapter 2: Introducing Data Contracts
What is a data contract?
An agreed interface between the generators of data, and its consumers
Setting expectations around that data
Defining how the data should be governed
Facilitating the explicit generation of quality data
The four principles of data contracts
When to use data contracts
Data contracts and the data mesh
Domain ownership
Data as a product
Self-serve data platform
Federated computational governance
Data contracts enable a data mesh
Summary
Further reading
Part 2: Driving Data Culture Change with Data Contracts
Chapter 3: How to Get Adoption in Your Organization
Using data contracts to change an organization
Articulating the value of your data
Building data products
What is a data product?
Adopting a data product mindset
Designing a data product
Walking through an example of a data product
Summary
Further reading
Chapter 4: Bringing Data Consumers and Generators Closer Together
Who is a consumer, and who is a generator?
Data consumers
Data generators
Assigning responsibility and accountability
Feeding data back to the product teams
Managing the evolution of data
Summary
Further reading
Chapter 5: Embedding Data Governance
Why we need data governance
The requirements of data governance
How data governance programs are typically applied
Promoting data governance through data contracts
Assigning responsibility for data governance
Responsibilities of the data generators
Introducing the data architecture council
Working together to implement federated data governance
Summary
Further reading
Part 3: Designing and Implementing a Data Architecture Based on Data Contracts
Chapter 6: What Makes Up a Data Contract
The schema of a data contract
Defining a schema
Using a schema registry as the source of truth
Evolving your data over time
Evolving your schemas
Migrating your consumers
Defining the governance and controls
Summary
Further reading
Chapter 7: A Contract-Driven Data Architecture
A step-change in building data platforms
Building generic data tooling
Introducing a data infrastructure team
A case study from GoCardless in promoting autonomy
Promoting autonomy through decentralization
Introducing the principles of a contract-driven data architecture
Automation
Guidelines and guardrails
Consistency
Providing self-served data infrastructure
Summary
Further reading
Chapter 8: A Sample Implementation
Technical requirements
Creating a data contract
Providing the interfaces to the data
Introducing IaC
Creating the interfaces from the data contract
Creating libraries for data generators
Populating a central schema registry
Registering a schema with the Confluent schema registry
Managing schema evolution
Implementing contract-driven tooling
Summary
Further reading
Chapter 9: Implementing Data Contracts in Your Organization
Getting started with data contracts
The ability to define a data contract
The ability to provision an interface for the data for consumers to query
The ability of generators to write data to the interface
Migrating to data contracts
Discovering data contracts
What is a data catalog?
Why are data catalogs important for discovering data contracts?
What is data lineage?
Why is data lineage important for data contracts?
Building a mature data contracts-backed data culture
Summary
Further reading
Chapter 10: Data Contracts in Practice
Designing a data contract
Identifying the purpose
Considering the trade-offs
Defining the data contract
Deploying the data contract
Monitoring and enforcing data contracts
The data contract’s definition
The quality of the data
The performance and dependability of the data
Data contract publishing patterns
Writing directly to the interface
Materialized views on CDC
The transactional outbox pattern
The listen-to-yourself pattern
Summary
Further reading
Index
About Packt
Other Books You May Enjoy