← back to the tree

data-eng-lab

Nineteen lakehouse scenarios — batch, streaming, and federated SQL — on live Iceberg tables, on Atlas.

19 scenarios5 datasetspaired Scala & PySpark

make up && make datasets

The medallion lakehouse as a layered pyramid — bronze, silver, and gold beneath an Iceberg apex, fed by flowing streams.
The medallion lakehouse beneath its Iceberg apex. Vector hero, 2026 — data-eng-lab repository.

Most lakehouse patterns are things you read about. This is a lab you run them in. Nineteen end-to-end scenarios execute against a working Iceberg lakehouse on the Atlas platform — fourteen batch, four streaming, one hybrid — and every scenario ships fully wired: a matched Scala (Zeppelin) and PySpark (Jupyter) notebook pair implementing identical logic, its own Airflow DAG, an architecture diagram, and a generated README documenting the data model down to the column. The folder names carry the catalog: scd2-online_retail-spark-iceberg, sessionization-gh_archive-spark-iceberg, time_travel-nyc_taxi-spark-iceberg.

Raw Parquet lands in object storage and moves through bronze, silver, and gold layers, every table an Apache Iceberg table cataloged by Atlas’s Iceberg REST catalog. Spark does the compute, Trino answers ad-hoc and federated SQL, Redpanda drives the streaming work, and Airflow and Jenkins run the production path.

Highlights

  • Iceberg’s table format worked hands-on — schema evolution, time travel, table maintenance and compaction, SCD2 dimension tracking, and incremental upserts, each exercised against live tables rather than described
  • Warehouse modeling on TPC-H — a star schema build, a join-optimization scenario, and BI-style analytical queries answered by Trino over the gold layer, plus federated queries reading the same tables Spark writes
  • Streaming on Redpanda — Kafka-compatible ingest of both a synthetic event stream and GH Archive data, windowed aggregations, and change-data-capture over the Online Retail dataset
  • Semi-structured and behavioral pipelines — JSON flattening and sessionization over GH Archive’s event firehose, data-quality gates on NYC Taxi trips, and feature engineering on MovieLens
  • The full medallion pattern, twice — once as paired notebooks (bronze ingest, silver cleaning and enrichment, gold aggregation) and once as a CI-built Spark application, so the notebook and production versions of the same pipeline sit side by side
  • A real CI path — two Maven Scala Spark apps, built and ScalaTest-tested by Jenkins, published as jars to MinIO, and launched by Airflow, the way a production job actually reaches a cluster
  • Declarative datasets — five public datasets (NYC Taxi, GH Archive, MovieLens, Online Retail, TPC-H) defined in a registry with tiered scales and fetched with a single make datasets
  • Atlas enablement runbook — the A1–A9 checklist and go-live guide for standing the whole stack up from cold

Stack & Architecture

The shape is a medallion lakehouse — landing, then bronze, silver, and gold — of Apache Iceberg tables on MinIO. A single REST catalog is the source of truth, so Spark, Trino, and the streaming jobs all see the same tables instead of their own copies. Scenarios are written as paired Scala and PySpark notebooks so each technique reads idiomatically in both languages, each scenario carries its own Airflow DAG, and the whole lab runs on Atlas rather than a bespoke Compose file it would have to maintain by hand. Per-scenario documentation is generated from the source tree and projected to both the site and the wiki.

Status

Atlas-enabled and go-live tested, with the full scenario catalog documented on a rendered site mirrored to a GitHub wiki.