Apache Spark 3 & Big Data Essentials in Scala

Create a spark session, read a json file into a data frame with infer schema, print the data frame schema, and build data frames from tuples or implicits for testing.

Create and inspect data frames through two exercises: build a smartphone dataset and load movies.json to printSchema and count the rows using Spark show and the count method.

Understand data frames as distributed tables with schemas on a Spark cluster, emphasizing immutability, partitioning, and the distinction between transformations and actions, plus lazy evaluation and shuffles.

Learn to read and write data frames from multiple data sources with Spark, using Json formats, schemas (infer or explicit), path options, and option maps with overwrite save mode.

Learn to read and write data with Spark across json, csv, parquet, text, and jdbc sources. Configure date formats, schemas, headers, separators, and compression, plus practical exercises with movies data.

Learn to manipulate data frames in Spark using columns and expressions, performing projections, selects, adding and renaming columns, and filtering, union, and distinct operations.

Practice columns and expressions in Spark by selecting two columns, computing total profit, and filtering comedies with IMDb ratings above threshold in a movies data frame.

Master aggregations and grouping on Spark DataFrames with count, count distinct, approx count distinct, min, max, sum, avg, mean, and std dev for a movies data set.

Learn how spark performs data frame joins, including inner, left, right, full outer, semi and anti joins, with join conditions, column ambiguity handling, and working with arrays.

practice data frame joins in spark by reading Postgres tables, computing max salaries per employee, identifying non managers with anti joins, and retrieving latest titles for top earners.

Explore common spark data types and dataframe operations. Learn how to add plain values with lit, apply boolean filters, perform numeric calculations, and use string and regex functions.

Learn to work with complex Spark data types by parsing dates in multiple formats, and using structs, arrays, and functions like to_date, date_diff, current_date, current_timestamp, and split.

Learn to handle nulls in Spark dataframes with coalesce to pick the first non-null rating, perform null checks, and fill or drop missing values.

Explore type-safe data processing with datasets by converting data frames to typed datasets, using encoders and implicits, and applying map, filter, and show on cars and numbers data.

Explore joining and grouping datasets in Spark, using datasets and data frames, including inner and outer joins, and group by key with practical examples involving guitar players, bands, and guitars.

Learn to use the spark sql shell to treat spark as a big data database, creating databases and tables, inserting data, and understanding managed versus external tables.

Learn to use spark sql in scala: build a spark session, read json data, create temp views, run sql, and transfer Postgres tables to a spark warehouse.

Explore Spark SQL exercises that read a movies data frame, save it as a table in Rock the JVM, and perform joins to analyze employee hires and department salaries.

Master the low level spark APIs with rdds, learning to read and write rdds and convert to and from data frames and data sets.

Explore RDDs through transformations, lazy and eager actions, and partitioning control, including repartitioning, coalescing, and grouping with shuffling—through movie json exercises.

Celebrate your progress as you reflect on the thousands of lines of code written from scratch. Believe in yourself—you're awesome and you can do anything.