Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• Overview of the Big Data landscape and Spark's role in contemporary data platforms
• Comprehending Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate approach for each use case
• Setting up and configuring SparkSession, and grasping the basics of application configuration

Module 2: PySpark DataFrames

• Ingesting and outputting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Utilising PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data manipulation
• Applying data quality checks and developing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Understanding the relationship between Docker images, Kubernetes pods, and Spark driver/executor processes — a conceptual perspective for cluster consumers rather than administrators
• Insight into the execution of Spark jobs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings
• Differentiating controls via Spark session parameters versus cluster provisioning by platform teams — and the significance of this boundary for tuning
• Practical exercise: Examine a live Spark-on-Kubernetes job within the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — In-Depth Analysis

• Evaluating executor count versus executor size: memory-to-core trade-offs and logical reasoning for sizing
• Allocating memory for drivers and executors, managing memory overhead, and translating these into pod resources
• Dynamic allocation: behaviour on Kubernetes, scenarios for resource savings, and limitations
• Practical exercise: Execute identical jobs with varying executor and core configurations to analyse runtime and resource consumption

Module 5: Scaling Dynamics & Cost Optimisation

• Analysing how node addition or removal affects job behaviour, completion time, and resource usage
• Weighing the performance and cost implications of using numerous small executors versus fewer large ones
• Shuffle mechanics and partition tuning, including the estimation of cost impacts from configuration decisions
• Practical exercise: Expand the sandbox cluster from five to ten nodes to observe changes in job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• The small-files problem: why sources fragmented into many 1–5 MB Parquet files hinder performance and disrupt partitioning
• Strategies for repartitioning and coalescing data
• Regulating partition sizes during read and write operations
• Writing Parquet files efficiently to prevent the recurrence of small-files issues downstream
• Practical exercise: Ingest a dataset comprising numerous small Parquet partitions, apply varied repartitioning strategies, and assess performance improvements

Module 7: Pandas Memory Management & Failure Analysis

• Identifying root causes of out-of-memory errors in Pandas and recognising warning signs
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when exporting large datasets to CSV
• Utilising chunked processing and dtype optimisation for resource-constrained environments
• Practical exercise: Simulate a typical Pandas out-of-memory scenario and resolve it via chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars against Pandas: performance profiles, lazy evaluation, and memory behaviour
• Integrating Polars with PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS contexts
• Practical exercise: Refactor a Pandas-centric transformation into Polars to compare memory usage and processing velocity

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory-management principles to a realistic ETL pipeline
• Understanding optimisation specifics for machine-learning workloads co-located on the same cluster
• Employing a systematic tuning workflow to diagnose cost and performance issues
• Practical exercise: Complete an end-to-end mini-project involving data loading, transformation, and basic model training, with participants independently tuning Spark configurations