Big Data

38.1 Databases with DBI & dbplyr

Connection pooling for apps: pool. Credentials: keyring, environment variables (.Renviron), config package — never hard-code passwords.

library(DBI)
con <- dbConnect(RSQLite::SQLite(), ":memory:")
# Others: RPostgres::Postgres(), RMariaDB::MariaDB(), odbc::odbc(), bigrquery::bigquery(), duckdb::duckdb()

dbWriteTable(con, "flights", nycflights13::flights)
dbListTables(con); dbListFields(con, "flights")
dbGetQuery(con, "SELECT carrier, AVG(dep_delay) AS delay FROM flights GROUP BY carrier")
dbExecute(con, "DELETE FROM flights WHERE year < 2013")
dbGetQuery(con, "SELECT * FROM flights WHERE origin = ?", params = list("JFK"))   # parameterized
dbBegin(con); dbCommit(con); dbRollback(con)

library(dplyr)
flights_db <- tbl(con, "flights")             # lazy table
q <- flights_db |> group_by(carrier) |> summarise(delay = mean(dep_delay, na.rm = TRUE)) |> arrange(desc(delay))
show_query(q)                                  # see the generated SQL
collect(q)                                     # run and bring results into R
dbDisconnect(con)

38.2 Arrow — larger-than-memory datasets

library(arrow)
write_dataset(big_df, "data/parquet", partitioning = c("year", "month"))
ds <- open_dataset("data/parquet")
ds |> filter(year == 2024, amount > 100) |>
  group_by(region) |> summarise(total = sum(amount)) |>
  collect()                                    # only the result is loaded into memory
open_dataset("s3://bucket/path")               # cloud storage

38.3 DuckDB — in-process analytical SQL

library(duckdb)
con <- dbConnect(duckdb())
dbGetQuery(con, "SELECT region, SUM(amount) FROM 'data/*.parquet' GROUP BY region")
duckplyr::read_parquet_duckdb("data/*.parquet") |> summarise(n = n(), .by = region)
to_duckdb(ds)                                 # arrow ↔ duckdb zero-copy

38.4 Spark with sparklyr

library(sparklyr)
sc <- spark_connect(master = "local")         # or a cluster / Databricks
flights_tbl <- copy_to(sc, nycflights13::flights, "flights")
flights_tbl |> filter(dep_delay > 60) |> count(carrier) |> collect()
ml_linear_regression(flights_tbl, dep_delay ~ distance)
spark_disconnect(sc)

38.5 Other approaches

bigmemory, ff, disk.frame (legacy), polars (R bindings to Polars), collapse (fast in-memory), cloud warehouses (BigQuery, Snowflake, Databricks via odbc/dbplyr), targets for orchestrating large pipelines.

🧪 Part III Exercises

Perform a full EDA of palmerpenguins and summarize findings in five bullet points with supporting plots.

Simulate the sampling distribution of the median for skewed data and compare bootstrap CIs to theory.

Test whether body mass differs among penguin species (ANOVA + Tukey + assumption checks + non-parametric alternative).

Fit and compare three regression models for mpg; check diagnostics; report a tidy table with broom/gtsummary.

Fit a logistic regression predicting am in mtcars; compute odds ratios, ROC curve and AUC.

Build a tidymodels workflow comparing logistic regression, random forest, and XGBoost with cross-validation.

Forecast AirPassengers with ARIMA and ETS; compare accuracy on a holdout set.

Implement a money S3 class with print, format, + methods; then an R6 Wallet class holding money objects.

Write a dplyr helper using {{ }} that computes grouped summaries for any column, and a ggplot helper taking column names as strings.

Speed up a slow loop-based function by vectorization, then by Rcpp, then by parallelization; benchmark each with bench::mark().