Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 21 hours
Course Outline
Module 1: Foundations of Big Data & Spark
- Survey of the Big Data landscape and Spark’s position in modern data architectures
- Deep dive into Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
- Distinguishing between RDD and DataFrame APIs and determining when to apply each
- Establishing SparkSessions and mastering the fundamentals of application configuration
Module 2: PySpark DataFrames
- Ingesting and exporting data from enterprise sources and formats, including CSV, JSON, Parquet, and Delta
- Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
- Executing advanced operations like window functions, timestamp management, and handling nested data structures
- Integrating data quality checks and crafting reusable, maintainable PySpark code
Module 3: Kubernetes & Docker Essentials for Spark Users
- Conceptual link between Docker images/Kubernetes pods and Spark driver/executor processes—focused on cluster consumers rather than administrators
- Exploring the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings to these resources
- Clarifying the control boundaries: Spark session parameters versus cluster provisioning by platform teams, and why this distinction is critical for tuning
- Practical Exercise: Examine a live Spark-on-Kubernetes job within a sandbox cluster to identify driver and executor pods
Module 4: Advanced Spark Session Configuration on Kubernetes
- Navigating the trade-offs between executor count and size: memory versus cores, and logical reasoning for resource allocation
- Allocating memory for drivers and executors, managing overhead, and translating these into Kubernetes pod resources
- Dynamic allocation: behavior on Kubernetes, scenarios where it conserves resources, and limitations
- Practical Exercise: Execute identical jobs with varying executor and core configurations to analyze runtime differences and resource utilization
Module 5: Scaling Dynamics & Cost Efficiency
- Impact analysis of node addition/removal on job behavior, completion latency, and total resource consumption
- Evaluating the performance and cost implications of many small executors versus fewer large ones
- Shuffle mechanics, partition tuning, and estimating the financial impact of configuration decisions
- Practical Exercise: Expand the sandbox cluster from five to ten nodes to observe the effect on job completion time and resource usage
Module 6: Optimal Data Ingestion & Partitioning
- The small-files problem: why splitting sources into numerous 1–5 MB Parquet files hurts performance and disrupts partitioning
- Strategies for repartitioning and coalescing data streams
- Regulating partition sizes during both read and write operations
- Efficient Parquet writing techniques to prevent downstream small-file proliferation
- Practical Exercise: Ingest a dataset composed of many small Parquet partitions, apply varied repartitioning methods, and benchmark performance before and after optimization
Module 7: Pandas Memory Control & Failure Analysis
- Identifying root causes of out-of-memory errors in Pandas and recognizing warning signs
- Implementing memory-efficient conversion patterns between Spark and Pandas
- Preventing memory spikes when exporting large datasets to CSV
- Leveraging chunked processing and dtype optimization for memory-constrained environments
- Practical Exercise: Simulate a typical Pandas out-of-memory failure and resolve it using chunking and dtype optimizations
Module 8: Polars as a Strategic Complement
- Positioning Polars alongside Pandas: performance profiles, lazy evaluation, and memory behavior
- Determining where Polars integrates with PySpark and Pandas in modern cloud data stacks and migration strategies, including AWS contexts
- Practical Exercise: Refactor a Pandas-centric transformation into Polars and benchmark memory usage and processing velocity
Module 9: Optimization in ETL & ML Workloads
- Applying configuration, partitioning, and memory management principles across a realistic ETL pipeline
- Optimization considerations specific to machine learning workloads sharing the same cluster resources
- Implementing a systematic tuning workflow to diagnose cost and performance bottlenecks
- Practical Exercise: Complete an end-to-end mini-project involving data loading, transformation, and a basic model-training step, with participants manually tuning the Spark configuration
Requirements
Candidates are expected to possess the following skills:
- Hands-on proficiency in Python programming, covering functions, modules, and fundamental object-oriented principles.
- Entry to intermediate-level experience with Pandas and standard tabular data workflows.
- Foundational knowledge of PySpark and Spark DataFrames, including data ingestion, transformations, actions, joins, and aggregations.
- A solid grasp of SQL and general data processing logic, such as filtering, grouping, and joining datasets.
- Basic awareness of Docker and Kubernetes principles, including containers, images, and pods. Experience with Kubernetes administration is not a prerequisite.
- Familiarity with standard data formats, including CSV, JSON, and Parquet.
Candidates are not required to be Kubernetes administrators or infrastructure specialists. This course is tailored for data engineers, developers, and data scientists who need to understand and optimize Spark workloads on Kubernetes from an application and configuration standpoint.
Intended Audience
This program is ideal for professionals who build, maintain, or enhance data processing and machine learning pipelines using Python and Spark in cloud or containerized settings.
It is specifically beneficial for:
- Data Engineers involved in PySpark development, distributed processing, and ETL pipeline management.
- Data Scientists handling large-scale datasets or executing machine learning tasks using Spark, Pandas, or Polars.
- Python Developers working on data-intensive applications aiming to enhance memory efficiency and processing throughput.
- Machine Learning Engineers overseeing data preparation and model training workloads on shared Kubernetes or cloud infrastructure.
- Analytics Engineers managing large datasets who seek to improve processing efficiency.
- DevOps, Platform, and Cloud Engineers supporting Spark on Kubernetes who need to comprehend how application-level settings influence resource consumption and performance.
- Technical Leads and Solution Architects involved in the design or optimization of contemporary data processing platforms.
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
