Migrating big data workloads between major versions requires careful planning. If you're comparing Spark 2 vs Spark 3, the fundamental shift isn't just a minor version increment—it represents a complete architectural overhaul of how distributed queries execute, how Python integrates with the JVM, and how SQL compliance is handled.
The primary difference lies in the introduction of Adaptive Query Execution (AQE), major enhancements to the Catalyst optimizer, and native support for Pandas UDFs. While Spark 2 relied entirely on a static execution plan formulated before a job started, Spark 3 adapts dynamically at runtime based on precise shuffle statistics. This structural leap changes performance baselines, memory management, and debugging workflows for data engineers.
Core Architectural Changes
The transition from the Spark 2.x ecosystem to Spark 3.x introduced deep engine-level modifications. Understanding these changes helps explain why identical queries can behave differently across versions.
Adaptive Query Execution (AQE)
In Spark 2, the execution plan was frozen the moment a job kicked off. If dataset cardinality estimations were wrong, the query could suffer from severe data skew or inefficient join strategies. Spark 3 solves this with AQE, which re-optimizes the query plan between stages using runtime statistics.
- Dynamic Join Strategy Switching: Spark 3 can automatically convert a sort-merge join into a broadcast hash join at runtime if a table turns out to be small enough after filtering.
- Automatic Skew Handling: If a partition receives disproportionate data volumes, AQE splits it into smaller sub-partitions to prevent executor bottlenecks.
- Dynamic Partition Coalescing: Post-shuffle partitions are automatically merged if they are too small, reducing task scheduling overhead.
Catalyst Optimizer Enhancements
The Catalyst optimizer received major rewrites in Spark 3. It features better subquery rewriting, improved rule-based optimization, and tighter ANSI SQL compliance. Writing standard SQL functions across different database platforms became much simpler because Spark 3 aligned closer to strict ANSI standards.
Performance and Memory Management
Performance benchmarks show that Spark 3 can execute standard workloads significantly faster than Spark 2, largely due to runtime optimizations and memory allocation improvements.
Memory Allocation and Garbage Collection
Spark 2 relied heavily on custom memory management, but Spark 3 leans into modernized project execution models. Improvements in off-heap memory handling and row-format layouts minimize Java Virtual Machine (JVM) garbage collection pauses. However, these changes mean that memory configurations (like spark.sql.autoBroadcastJoinThreshold) require careful tuning during migration to avoid unexpected out-of-memory errors.
Python and Pandas Integration
Data scientists using PySpark notice a massive shift between versions. Spark 2 executed user-defined functions (UDFs) row-by-row, introducing heavy serialization overhead. Spark 3 introduced vectorized Pandas UDFs (enabled by Apache Arrow), allowing columnar data transfers between JVM workers and Python processes. This reduces serialization bottlenecks by orders of magnitude for machine learning and data transformation pipelines.
Breaking Changes and Migration Challenges
Upgrading isn't always plug-and-play. Moving workloads from Spark 2 to Spark 3 exposes several breaking changes that teams must address.
- API Behavior Shifts: Certain configuration defaults changed between versions. For instance, implicit type castings in SQL are stricter in Spark 3, meaning queries that silently coerced data types in Spark 2 might throw explicit runtime errors.
- Hive Metadata Handling: Spark 3 updates support for external metastores, requiring validation of Hive catalog interactions and table serde properties.
- Dependency Upgrades: Scala, Parquet, and Avro libraries received major version bumps. Custom plugins or older third-party libraries built for Spark 2 must be recompiled for Spark 3.
To explore the official technical specifications and upgrade pathways, consult the Apache Spark Documentation. For hands-on enterprise migration strategies, review the Databricks Migration Guide.
Frequently Asked Questions
Is Spark 3 completely backward compatible with Spark 2 code?
Most standard DataFrame API code runs without modification, but low-level RDD operations, custom configurations, and strict SQL queries may require code refactoring due to underlying optimizer and parser updates.
Why does my query throw an Out-of-Memory error after upgrading to Spark 3?
This usually happens because Spark 3 changes default thresholds, such as automatic broadcast joins. If a dataset size sits near the threshold boundary, Spark 3 might attempt to broadcast it into memory where Spark 2 chose a safer sort-merge join. Adjusting memory settings or tuning thresholds resolves this.
Do I need to rewrite my PySpark UDFs?
You don't have to rewrite them, but refactoring standard UDFs into vectorized Pandas UDFs is strongly recommended to capture the performance multipliers available in Spark 3.
How do I check if Adaptive Query Execution is working?
You can verify AQE behavior by inspecting the Spark UI or running the EXPLAIN command on your queries. When AQE is enabled, physical plans show dynamic stages that adapt based on completed upstream task metrics.
