The Medallion Architecture — Structuring Your Delta Lake for Scale

Most data lakes start the same way: someone dumps raw files into blob storage, writes a notebook to read them, and calls it a pipeline. Six months later you have three teams fighting over which version of a table is "the real one." The medallion architecture solves this — not by adding complexity, but by making the implicit explicit.

The three zones

The idea is simple. Data flows through three layers, each with a clear contract.

Bronze is raw ingestion. You land exactly what the source sent you — schema preserved, no transformations, append-only. Think of it as your audit trail. When something breaks downstream six months from now, bronze is how you replay from truth.

Silver is cleaned and conformed. You apply type casting, deduplication, and business rules. Rows that fail validation don't get silently dropped — they go to a quarantine table. Silver is where you create the single source of truth for each domain entity.

Gold is purpose-built for consumers. Aggregations, wide tables, feature stores. Gold tables are optimised for the query patterns of the team that owns them — analytics, ML, or a product feature.

Setting it up in Delta Lake

Delta Lake makes this durable. Every layer is a Delta table, which means you get transaction logs, time travel, and schema enforcement for free.

python
from delta.tables import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Bronze: land raw with full audit columns
def write_bronze(df, table_path: str):
    (
        df
        .withColumn("_ingested_at", current_timestamp())
        .withColumn("_source_file", input_file_name())
        .write
        .format("delta")
        .mode("append")
        .option("mergeSchema", "true")
        .save(table_path)
    )

# Silver: upsert with deduplication
def write_silver(df, table_path: str, merge_key: str):
    if DeltaTable.isDeltaTable(spark, table_path):
        target = DeltaTable.forPath(spark, table_path)
        (
            target.alias("t")
            .merge(df.alias("s"), f"t.{merge_key} = s.{merge_key}")
            .whenMatchedUpdateAll()
            .whenNotMatchedInsertAll()
            .execute()
        )
    else:
        df.write.format("delta").save(table_path)

Schema enforcement is not optional

The single biggest mistake I see is treating silver like a permissive landing zone. Silver should reject schema drift aggressively.

python
from pyspark.sql.types import StructType, StructField, StringType, TimestampType

SILVER_SCHEMA = StructType([
    StructField("user_id", StringType(), nullable=False),
    StructField("event_type", StringType(), nullable=False),
    StructField("occurred_at", TimestampType(), nullable=False),
])

def enforce_schema(df, schema: StructType):
    try:
        return spark.createDataFrame(df.rdd, schema)
    except Exception as e:
        raise ValueError(f"Schema validation failed: {e}")

If a source adds a new column, that should be a deliberate decision — not something that silently propagates into your gold layer and breaks downstream models.

Naming conventions that scale

The architecture only works if the layers are discoverable. I use a consistent path pattern:

/data/bronze/{source_system}/{entity}/
/data/silver/{domain}/{entity}/
/data/gold/{consumer}/{table_name}/

And in Unity Catalog:

sql
-- Bronze: raw namespace per source
CREATE CATALOG IF NOT EXISTS raw;
CREATE SCHEMA IF NOT EXISTS raw.salesforce;

-- Silver: domain namespaces
CREATE CATALOG IF NOT EXISTS silver;
CREATE SCHEMA IF NOT EXISTS silver.crm;

-- Gold: consumer namespaces
CREATE CATALOG IF NOT EXISTS gold;
CREATE SCHEMA IF NOT EXISTS gold.analytics;

When to skip a layer

Gold tables don't always need silver parents, and silver doesn't always need a bronze step. A lookup table loaded from a trusted internal API can go straight to silver. A one-off export for a single dashboard can live in gold without a corresponding silver model.

The layers are a mental model, not a bureaucratic requirement. The point is that every table has a clear tier — and everyone on your team knows what that tier means.

#The three zones

#Setting it up in Delta Lake

#Schema enforcement is not optional

#Naming conventions that scale

#When to skip a layer

The three zones

Setting it up in Delta Lake

Schema enforcement is not optional

Naming conventions that scale

When to skip a layer