Big Data with Apache Spark

Spark (written in Scala) is the most-used Scala application framework.

Concepts: lazy transformations vs actions, partitions, shuffles, caching (.cache()), broadcast variables, Catalyst optimizer, Structured Streaming, MLlib, GraphX. Spark 4 supports Scala 2.13 (Scala 3 usage works via 2.13 cross-compat libraries).

Other data tools: Apache Flink (Java/Scala APIs via flink-scala-api), Kafka Streams (Scala DSL), fs2-kafka, zio-kafka, Scio (Apache Beam / Google Dataflow), Delta Lake, Breeze (numerical), Smile (ML).

import org.apache.spark.sql.{SparkSession, functions as F}

val spark = SparkSession.builder.appName("Demo").master("local[*]").getOrCreate()
import spark.implicits.*

// DataFrame API
val df = spark.read.option("header", "true").option("inferSchema", "true").csv("sales.csv")
df.groupBy("region")
  .agg(F.sum("amount").as("total"), F.avg("amount").as("avg"))
  .orderBy(F.desc("total"))
  .show()

// Typed Dataset API
case class Sale(region: String, amount: Double)
val ds = df.as[Sale]
ds.filter(_.amount > 100).map(s => s.region.toUpperCase).distinct().show()

// SQL
df.createOrReplaceTempView("sales")
spark.sql("SELECT region, COUNT(*) FROM sales GROUP BY region").show()

// RDD (low level)
val counts = spark.sparkContext.textFile("book.txt")
  .flatMap(_.split("\\W+")).map(w => (w.toLowerCase, 1)).reduceByKey(_ + _)

spark.stop()