Data Analysis

Big Data — Hadoop, Spark & Kafka

Distributed processing at scale. The 5 Vs of Big Data, MapReduce origins (Google paper, Hadoop), HDFS (NameNode, DataNode, replication, block size), YARN resource manager, Hadoop ecosystem (Hive, Pig, HBase, ZooKeeper, Oozie, Sqoop, Flume), Apache Spark — RDDs & DataFrames & Datasets, Spark SQL, Catalyst optimizer, Tungsten execution engine, Spark on YARN / Kubernetes / standalone, Spark Streaming & Structured Streaming, Spark MLlib, GraphX, Delta Lake (ACID on data lakes), Apache Iceberg & Apache Hudi, Apache Kafka — topics, partitions, brokers, consumer groups, Kafka Connect, Kafka Streams, ksqlDB, Schema Registry, Avro / Protobuf / JSON serialization, Apache Flink (true streaming with event-time), Apache Beam (unified batch + stream), data lake vs data warehouse vs lakehouse, modern stacks (Databricks, Snowflake, BigQuery, Athena), Parquet & ORC columnar formats, file partitioning & bucketing, performance tuning & AQE, cluster autoscaling, and observability for big-data jobs. 30 units, 450 lessons.