Spark Partitioning — The Difference Between a 3-Minute and 3-Hour Job

Partitioning is the single most common source of Spark performance problems I've encountered. Jobs that run in two minutes on a small dataset grind to a halt at scale, often for one of three reasons: too many small partitions, too few large ones, or skew that concentrates all the work on a single task.

Understanding how Spark partitions data — and when you're fighting against it — turns debugging from guesswork into diagnosis.

#The two types of partitions

Spark has two distinct partitioning contexts that people frequently conflate.

Input partitions are determined by how you read data. When you read a Parquet file, Spark creates one partition per file (or per block for large files). This is mostly outside your control once the data is written.

Shuffle partitions are created during wide transformations — groupBy, join, distinct. These default to 200, which is almost always wrong.

pythonCopy
# Default: 200 shuffle partitions regardless of data size
spark.conf.set("spark.sql.shuffle.partitions", "200")

# Better: size to your data
# Rule of thumb: aim for partition sizes between 100MB and 1GB
# For a 10GB dataset post-shuffle, use ~50-100 partitions
spark.conf.set("spark.sql.shuffle.partitions", "50")

On Databricks, Adaptive Query Execution (AQE) handles this automatically in most cases — but it's still worth understanding what's happening underneath.

#Diagnosing skew

A job where one task takes 45 minutes while the other 199 finish in 30 seconds is a skew problem. You'll see it in the Spark UI as a massive outlier in the task duration distribution.

The cause is almost always a join or group-by on a key with a highly uneven distribution — a null value in 80% of rows, or a single high-volume customer dominating an orders table.

pythonCopy
# Identify skew before the join
df.groupBy("customer_id").count().orderBy("count", ascending=False).show(20)

# If one key dominates, salting distributes the load
import pyspark.sql.functions as F

SALT_BUCKETS = 50

# Salt the skewed side
skewed_df = (
    large_df
    .withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int"))
    .withColumn("customer_id_salted", F.concat_ws("_", "customer_id", "salt"))
)

# Explode the small side to match
small_df_exploded = (
    small_df
    .withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)])))
    .withColumn("customer_id_salted", F.concat_ws("_", "customer_id", "salt"))
)

result = skewed_df.join(small_df_exploded, "customer_id_salted")

Salting is ugly. Use it as a last resort after confirming skew — don't reach for it preemptively.

#repartition vs coalesce

Both reduce the number of partitions, but they behave very differently.

coalesce(n) merges partitions without a full shuffle. It's fast and cheap, but it can only reduce partitions — not redistribute data across them. Use it when writing output files and you want fewer, larger files.

repartition(n) triggers a full shuffle. Every row moves across the network to land in the right partition. It's expensive but necessary when you need evenly distributed data — for example, before a join on a skewed column.

pythonCopy
# Writing: coalesce is almost always right
(
    result_df
    .coalesce(10)  # Merge to 10 output files without reshuffling
    .write
    .format("delta")
    .save("/data/silver/orders/")
)

# Before a join on a known-skewed column: repartition to even the load
df_repartitioned = df.repartition(200, "region_code")

#Partition pruning in Delta Lake

Delta Lake stores min/max statistics for each file at write time. If you filter on a partitioned column, Spark can skip entire files without reading them — but only if you partition your table on columns you actually filter on.

pythonCopy
(
    df
    .write
    .format("delta")
    .partitionBy("event_date", "region")  # Enables pruning on these columns
    .save("/data/silver/events/")
)

# This query reads only the files for 2024-01, EU — skips everything else
spark.read.format("delta").load("/data/silver/events/").filter(
    (F.col("event_date") >= "2024-01-01") & (F.col("region") == "EU")
)

The rule: partition on columns with low cardinality that appear frequently in your WHERE clauses. Partitioning on user_id creates millions of tiny files and does more harm than good.

#A quick checklist

Before pushing a Spark job to production, I run through:

• What is spark.sql.shuffle.partitions set to, and does it match the expected post-shuffle data volume?
• Are there any keys with extreme cardinality in joins or group-bys?
• Is the output partitioned on a column that consumers actually filter on?
• Are we using coalesce before writes to avoid small file accumulation?
• Is AQE enabled? (It is by default on Databricks — don't turn it off.)

Most performance problems in Spark come down to moving more data than necessary, or moving it at the wrong time. Getting partitioning right is usually the highest-leverage fix available.