دانلود کتاب Data Algorithms with Spark: Recipes and Design Patterns for Scaling Up using PySpark

فهرست مطالب :

Cover Copyright Table of Contents Foreword Preface Why I Wrote This Book Who This Book Is For How This Book Is Organized Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Part I. Fundamentals Chapter 1. Introduction to Spark and PySpark Why Spark for Data Analytics The Spark Ecosystem Spark Architecture The Power of PySpark PySpark Architecture Spark Data Abstractions RDD Examples Spark RDD Operations DataFrame Examples Using the PySpark Shell Launching the PySpark Shell Creating an RDD from a Collection Aggregating and Merging Values of Keys Filtering an RDD’s Elements Grouping Similar Keys Aggregating Values for Similar Keys ETL Example with DataFrames Extraction Transformation Loading Summary Chapter 2. Transformations in Action The DNA Base Count Example The DNA Base Count Problem FASTA Format Sample Data DNA Base Count Solution 1 Step 1: Create an RDD[String] from the Input Step 2: Define a Mapper Function Step 3: Find the Frequencies of DNA Letters Pros and Cons of Solution 1 DNA Base Count Solution 2 Step 1: Create an RDD[String] from the Input Step 2: Define a Mapper Function Step 3: Find the Frequencies of DNA Letters Pros and Cons of Solution 2 DNA Base Count Solution 3 The mapPartitions() Transformation Step 1: Create an RDD[String] from the Input Step 2: Define a Function to Handle a Partition Step 3: Apply the Custom Function to Each Partition Pros and Cons of Solution 3 Summary Chapter 3. Mapper Transformations Data Abstractions and Mappers What Are Transformations? Lazy Transformations The map() Transformation DataFrame Mapper The flatMap() Transformation map() Versus flatMap() Apply flatMap() to a DataFrame The mapValues() Transformation The flatMapValues() Transformation The mapPartitions() Transformation Handling Empty Partitions Benefits and Drawbacks DataFrames and mapPartitions() Transformation Summary Chapter 4. Reductions in Spark Creating Pair RDDs Reduction Transformations Spark’s Reductions Simple Warmup Example Solving with reduceByKey() Solving with groupByKey() Solving with aggregateByKey() Solving with combineByKey() What Is a Monoid? Monoid and Non-Monoid Examples The Movie Problem Input Dataset to Analyze The aggregateByKey() Transformation First Solution Using aggregateByKey() Second Solution Using aggregateByKey() Complete PySpark Solution Using groupByKey() Complete PySpark Solution Using reduceByKey() Complete PySpark Solution Using combineByKey() The Shuffle Step in Reductions Shuffle Step for groupByKey() Shuffle Step for reduceByKey() Summary Part II. Working with Data Chapter 5. Partitioning Data Introduction to Partitions Partitions in Spark Managing Partitions Default Partitioning Explicit Partitioning Physical Partitioning for SQL Queries Physical Partitioning of Data in Spark Partition as Text Format Partition as Parquet Format How to Query Partitioned Data Amazon Athena Example Summary Chapter 6. Graph Algorithms Introduction to Graphs The GraphFrames API How to Use GraphFrames GraphFrames Functions and Attributes GraphFrames Algorithms Finding Triangles Motif Finding Real-World Applications Gene Analysis Social Recommendations Facebook Circles Connected Components Analyzing Flight Data Summary Chapter 7. Interacting with External Data Sources Relational Databases Reading from a Database Writing a DataFrame to a Database Reading Text Files Reading and Writing CSV Files Reading CSV Files Writing CSV Files Reading and Writing JSON Files Reading JSON Files Writing JSON Files Reading from and Writing to Amazon S3 Reading from Amazon S3 Writing to Amazon S3 Reading and Writing Hadoop Files Reading Hadoop Text Files Writing Hadoop Text Files Reading and Writing HDFS SequenceFiles Reading and Writing Parquet Files Writing Parquet Files Reading Parquet Files Reading and Writing Avro Files Reading Avro Files Writing Avro Files Reading from and Writing to MS SQL Server Writing to MS SQL Server Reading from MS SQL Server Reading Image Files Creating a DataFrame from Images Summary Chapter 8. Ranking Algorithms Rank Product Calculation of the Rank Product Formalizing Rank Product Rank Product Example PySpark Solution PageRank PageRank’s Iterative Computation Custom PageRank in PySpark Using RDDs Custom PageRank in PySpark Using an Adjacency Matrix PageRank with GraphFrames Summary Part III. Data Design Patterns Chapter 9. Classic Data Design Patterns Input-Map-Output RDD Solution DataFrame Solution Flat Mapper functionality Input-Filter-Output RDD Solution DataFrame Solution DataFrame Filter Input-Map-Reduce-Output RDD Solution DataFrame Solution Input-Multiple-Maps-Reduce-Output RDD Solution DataFrame Solution Input-Map-Combiner-Reduce-Output Input-MapPartitions-Reduce-Output Inverted Index Problem Statement Input Output PySpark Solution Summary Chapter 10. Practical Data Design Patterns In-Mapper Combining Basic MapReduce Algorithm In-Mapper Combining per Record In-Mapper Combining per Partition Top-10 Top-N Formalized PySpark Solution Finding the Bottom 10 MinMax Solution 1: Classic MapReduce Solution 2: Sorting Solution 3: Spark’s mapPartitions() The Composite Pattern and Monoids Monoids Monoidal and Non-Monoidal Examples Non-Monoid MapReduce Example Monoid MapReduce Example PySpark Implementation of Monoidal Mean Functors and Monoids Conclusion on Using Monoids Binning Sorting Summary Chapter 11. Join Design Patterns Introduction to the Join Operation Join in MapReduce Map Phase Reducer Phase Implementation in PySpark Map-Side Join Using RDDs Map-Side Join Using DataFrames Step 1: Create Cache for Airports Step 2: Create Cache for Airlines Step 3: Create Facts Table Step 4: Apply Map-Side Join Efficient Joins Using Bloom Filters Introduction to Bloom Filters A Simple Bloom Filter Example Bloom Filters in Python Using Bloom Filters in PySpark Summary Chapter 12. Feature Engineering in PySpark Introduction to Feature Engineering Adding New Features Applying UDFs Creating Pipelines Binarizing Data Imputation Tokenization Tokenizer RegexTokenizer Tokenization with a Pipeline Standardization Normalization Scaling a Column Using a Pipeline Using MinMaxScaler on Multiple Columns Normalization Using Normalizer String Indexing Applying StringIndexer to a Single Column Applying StringIndexer to Several Columns Vector Assembly Bucketing Bucketizer QuantileDiscretizer Logarithm Transformation One-Hot Encoding TF-IDF FeatureHasher SQLTransformer Summary Index About the Author Colophon