دانلود کتاب Delta Lake: The Definitive Guide: Modern Data Lakehouse Architectures with Data Lakes

فهرست مطالب :

Copyright
Table of Contents
Foreword by Michael Armbrust
Foreword by Dominique Brezinski
Preface
Who This Book Is For
How This Book Is Organized
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments
Denny
Tristen
Scott
Prashanth
Chapter 1. Introduction to the Delta Lake Lakehouse Format
The Genesis of Delta Lake
Data Warehousing, Data Lakes, and Data Lakehouses
Project Tahoe to Delta Lake: The Early Years Months
What Is Delta Lake?
Common Use Cases
Key Features
Anatomy of a Delta Lake Table
Delta Transaction Protocol
Understanding the Delta Lake Transaction Log at the File Level
The Single Source of Truth
The Relationship Between Metadata and Data
Multiversion Concurrency Control (MVCC) File and Data Observations
Observing the Interaction Between the Metadata and Data
Table Features
Delta Kernel
Delta UniForm
Conclusion
Chapter 2. Installing Delta Lake
Delta Lake Docker Image
Delta Lake for Python
PySpark Shell
JupyterLab Notebook
Scala Shell
Delta Rust API
ROAPI
Native Delta Lake Libraries
Multiple Bindings Available
Installing the Delta Lake Python Package
Apache Spark with Delta Lake
Setting Up Delta Lake with Apache Spark
Prerequisite: Set Up Java
Setting Up an Interactive Shell
PySpark Declarative API
Databricks Community Edition
Create a Cluster with Databricks Runtime
Importing Notebooks
Attaching Notebooks
Conclusion
Chapter 3. Essential Delta Lake Operations
Create
Creating a Delta Lake Table
Loading Data into a Delta Lake Table
The Transaction Log
Read
Querying Data from a Delta Lake Table
Reading with Time Travel
Update
Delete
Deleting Data from a Delta Lake Table
Overwriting Data in a Delta Lake Table
Merge
Other Useful Actions
Parquet Conversions
Delta Lake Metadata and History
Conclusion
Chapter 4. Diving into the Delta Lake Ecosystem
Connectors
Apache Flink
Flink DataStream Connector
Installing the Connector
DeltaSource API
DeltaSink API
End-to-End Example
Kafka Delta Ingest
Install Rust
Build the Project
Run the Ingestion Flow
Trino
Getting Started
Configuring and Using the Trino Connector
Using Show Catalogs
Creating a Schema
Show Schemas
Working with Tables
Table Operations
Conclusion
Chapter 5. Maintaining Your Delta Lake
Using Delta Lake Table Properties
Delta Lake Table Properties Reference
Create an Empty Table with Properties
Populate the Table
Evolve the Table Schema
Add or Modify Table Properties
Remove Table Properties
Delta Lake Table Optimization
The Problem with Big Tables and Small Files
Using OPTIMIZE to Fix the Small File Problem
Table Tuning and Management
Partitioning Your Tables
Defining Partitions on Table Creation
Migrating from a Nonpartitioned to a Partitioned Table
Repairing, Restoring, and Replacing Table Data
Recovering and Replacing Tables
Deleting Data and Removing Partitions
The Life Cycle of a Delta Lake Table
Restoring Your Table
Cleaning Up
Conclusion
Chapter 6. Building Native Applications with Delta Lake
Getting Started
Python
Rust
Building a Lambda
What’s Next
Chapter 7. Streaming In and Out of Your Delta Lake
Streaming and Delta Lake
Streaming Versus Batch Processing
Delta as Source
Delta as Sink
Delta Streaming Options
Limit the Input Rate
Ignore Updates or Deletes
Initial Processing Position
Initial Snapshot with withEventTimeOrder
Advanced Usage with Apache Spark
Idempotent Stream Writes
Delta Lake Performance Metrics
Auto Loader and Delta Live Tables
Auto Loader
Delta Live Tables
Change Data Feed
Using Change Data Feed
Schema
Conclusion
Chapter 8. Advanced Features
Generated Columns, Keys, and IDs
Comments and Constraints
Comments
Delta Table Constraints
Deletion Vectors
Merge-on-Read
Stepping Through Deletion Vectors
Conclusion
Chapter 9. Architecting Your Lakehouse
The Lakehouse Architecture
What Is a Lakehouse?
Learning from Data Warehouses
Learning from Data Lakes
The Dual-Tier Data Architecture
Lakehouse Architecture
Foundations with Delta Lake
Open Source on Open Standards in an Open Ecosystem
Transaction Support
Schema Enforcement and Governance
The Medallion Architecture
Exploring the Bronze Layer
Exploring the Silver Layer
Exploring the Gold Layer
Streaming Medallion Architecture
Conclusion
Chapter 10. Performance Tuning: Optimizing Your Data Pipelines with Delta Lake
Performance Objectives
Maximizing Read Performance
Maximizing Write Performance
Performance Considerations
Partitioning
Table Utilities
Table Statistics
Cluster By
Bloom Filter Index
Conclusion
Chapter 11. Successful Design Patterns
Slashing Compute Costs
High-Speed Solutions
Smart Device Integration
Efficient Streaming Ingestion
Streaming Ingestion
The Inception of Delta Rust
The Evolution of Ingestion
Coordinating Complex Systems
Combining Operational Data Stores at DoorDash
Change Data Capture
Delta and Flink in Harmony
Conclusion
Chapter 12. Foundations of Lakehouse Governance and Security
Lakehouse Governance
The Emergence of Data Governance
Data Products and Their Relationship to Data Assets
Data Products in the Lakehouse
Maintaining High Trust
Data Assets and Access
The Data Asset Model
Unifying Governance Between Data Warehouses and Lakes
Permissions Management
Filesystem Permissions
Cloud Object Store Access Controls
Identity and Access Management
Data Security
Fine-Grained Access Controls for the Lakehouse
Conclusion
Chapter 13. Metadata Management, Data Flow, and Lineage
Metadata Management
What Is Metadata Management?
Data Catalogs
Data Reliability, Stewards, and Permissions Management
Why the Metastore Matters
Unity Catalog
Data Flow and Lineage
Data Lineage
Data Sharing
Automating Data Life Cycles
Audit Logging
Monitoring and Alerting
What Is Data Discovery?
Conclusion
Chapter 14. Data Sharing with the Delta Sharing Protocol
The Basics of Delta Sharing
Data Providers
Data Recipients
Delta Sharing Server
Using the REST APIs
Anatomy of the REST URI
List Shares
Get Share
List Schemas in Share
List All Tables in Share
Delta Sharing Clients
Delta Sharing with Apache Spark
Stream Processing with Delta Shares
Delta Sharing Community Connectors
Conclusion
Index
About the Authors
Colophon