Big Data Analytics: A Guide to Data Science Practitioners Making the Transition to Big Data (Chapman & Hall/CRC Data Science) 9781032457550, 2138085855, 9781032458144, 9781003378822

Table of contents :
Preface

I Setting the Scene: Analyzing Big Data

Introduction

1 What is Big in “Big Data”?

2 Approaches to Analyzing Big Data

3 The Two Domains of Big Data Analytics

3.1 A practical big P problem

3.1.1 Simple logistic regression (naive approach)

3.1.2 Regularization: the lasso estimator

3.2 A practical big N problem

3.2.1 OLS as a point of reference

3.2.2 The Uluru algorithm as an alternative to OLS

II Platform: Software and Computing Resources

Introduction

4 Software: Programming with (Big) Data

4.1 Domains of programming with (big) data

4.2 Measuring R performance

4.3 Writing efficient R code

4.3.1 Memory allocation and growing objects

4.3.2 Vectorization in basic R functions

4.3.3 apply-type functions and vectorization

4.3.4 Avoiding unnecessary copying

4.3.5 Releasing memory

4.3.6 Beyond R

4.4 SQL basics

4.4.1 First steps in SQL(ite)

4.4.2 Joins

4.5 With a little help from my friends: GPT and R/SQL coding

4.6 Wrapping up

5 Hardware: Computing Resources

5.1 Mass storage

5.1.1 Avoiding redundancies

5.1.2 Data compression

5.2 Random access memory (RAM)

5.3 Combining RAM and hard disk: Virtual memory

5.4 CPU and parallelization

5.4.1 Naive multi-session approach

5.4.2 Multi-session approach with futures

5.4.3 Multi-core and multi-node approach

5.5 GPUs for scientific computing

5.5.1 GPUs in R

5.6 The road ahead: Hardware made for machine learning

5.7 Wrapping up

5.8 Still have insufficient computing resources?

6 Distributed Systems

6.1 MapReduce

6.2 Apache Hadoop

6.2.1 Hadoop word count example

6.3 Apache Spark

6.4 Spark with R

6.4.1 Data import and summary statistics

6.5 Spark with SQL

6.6 Spark with R + SQL

6.7 Wrapping up

7 Cloud Computing

7.1 Cloud computing basics and platforms

7.2 Transitioning to the cloud

7.3 Scaling up in the cloud: Virtual servers

7.3.1 Parallelization with an EC2 instance

7.4 Scaling up with GPUs

7.4.1 GPUs on Google Colab

7.4.2 RStudio and EC2 with GPUs on AWS

7.5 Scaling out: MapReduce in the cloud

7.6 Wrapping up

III Components of Big Data Analytics

Introduction

8 Data Collection and Data Storage

8.1 Gathering and compilation of raw data

8.2 Stack/combine raw source files

8.3 Efficient local data storage

8.3.1 RDBMS basics

8.3.2 Efficient data access: Indices and joins in SQLite

8.4 Connecting R to an RDBMS

8.4.1 Creating a new database with RSQLite

8.4.2 Importing data

8.4.3 Issuing queries

8.5 Cloud solutions for (big) data storage

8.5.1 Easy-to-use RDBMS in the cloud: AWS RDS

8.6 Column-based analytics databases

8.6.1 Installation and start up

8.6.2 First steps via Druid's GUI

8.6.3 Query Druid from R

8.7 Data warehouses

8.7.1 Data warehouse for analytics: Google BigQuery example

8.8 Data lakes and simple storage service

8.8.1 AWS S3 with R: First steps

8.8.2 Uploading data to S3

8.8.3 More than just simple storage: S3 + Amazon Athena

8.9 Wrapping up

9 Big Data Cleaning and Transformation

9.1 Out-of-memory strategies and lazy evaluation: Practical basics

9.1.1 Chunking data with the ff package

9.1.2 Memory mapping with bigmemory

9.1.3 Connecting to Apache Arrow

9.2 Big Data preparation tutorial with ff

9.2.1 Set up

9.2.2 Data import

9.2.3 Inspect imported files

9.2.4 Data cleaning and transformation

9.2.5 Inspect difference in in-memory operation

9.2.6 Subsetting

9.2.7 Save/load/export ff files

9.3 Big Data preparation tutorial with arrow

9.4 Wrapping up

10 Descriptive Statistics and Aggregation

10.1 Data aggregation: The ‘split-apply-combine’ strategy

10.2 Data aggregation with chunked data files

10.3 High-speed in-memory data aggregation with arrow

10.4 High-speed in-memory data aggregation with data.table

10.5 Wrapping up

11 (Big) Data Visualization

11.1 Challenges of Big Data visualization

11.2 Data exploration with ggplot2

11.3 Visualizing time and space

11.3.1 Preparations

11.3.2 Pick-up and drop-off locations

11.4 Wrapping up

IV Application: Topics in Big Data Econometrics

Introduction

12 Bottlenecks in Everyday Data Analytics Tasks

12.1 Case study: Efficient fixed effects estimation

12.2 Case study: Loops, memory, and vectorization

12.2.1 Naïve approach (ignorant of R)

12.2.2 Improvement 1: Pre-allocation of memory

12.2.3 Improvement 2: Exploit vectorization

12.3 Case study: Bootstrapping and parallel processing

12.3.1 Parallelization with an EC2 instance

13 Econometrics with GPUs

13.1 OLS on GPUs

13.2 A word of caution

13.3 Higher-level interfaces for basic econometrics with GPUs

13.4 TensorFlow/Keras example: Predict housing prices

13.4.1 Data preparation

13.4.2 Model specification

13.4.3 Training and prediction

13.5 Wrapping up

14 Regression Analysis and Categorization with Spark and R

14.1 Simple linear regression analysis

14.2 Machine learning for classification

14.3 Building machine learning pipelines with R and Spark

14.3.1 Set up and data import

14.3.2 Building the pipeline

14.4 Wrapping up

15 Large-scale Text Analysis with sparklyr

15.1 Getting started: Import, pre-processing, and word count

15.2 Tutorial: political slant

15.2.1 Data download and import

15.2.2 Cleaning speeches data

15.2.3 Create a bigrams count per party

15.2.4 Find “partisan” phrases

15.2.5 Results: Most partisan phrases by congress

15.3 Natural Language Processing at Scale

15.3.1 Preparatory steps

15.3.2 Sentiment annotation

15.4 Aggregation and visualization

15.5 sparklyr and lazy evaluation

V Appendices

Appendix A: GitHub

Appendix B: R Basics

Appendix C: Install Hadoop

VI References and Index

Bibliography

Index