A Crash Course In PySpark

This course uses Google Colab as a development environment. Sign in with Gmail, run Python code in notebooks, and upload datasets that are deleted when the session ends.

Learn how data frames in Spark resemble tables, how to load data, filter and pivot to extract insights, and explore salary patterns by gender, job title, and city using PySpark.

Configure Google Colab to run PySpark by updating apt-get, installing the Java JDK, downloading and unpacking the latest Spark, setting environment variables, installing findspark, and creating a SparkSession.

Ingest data from csv into a spark dataframe, clean nulls (city, jobtitle), and cleanse salary and latitude by mean and median imputation with numpy for ready analysis.

Use PySpark to group by gender and city, compute average salaries, identify the highest-paying city, and compare male and female pay with deltas and aliases.

Learn how to bring CSV data into a Spark dataframe, define a custom schema with struct fields to enforce correct types, and compare inferred types with the explicit schema.

Explore inspecting dataframes in PySpark by applying a schema to a csv read, viewing column types, the first rows, and descriptive statistics to identify nulls, duplicates, and data ranges.

Learn to handle null and duplicate values in PySpark with practical techniques. Drop rows with nulls, filter by specific columns, and create a clean city column using when.

Select and rename columns, then filter dataframes by exact matches, pattern like with wildcards, ranges, and membership, using substring and aliasing to streamline PySpark data queries.

Learn to apply multiple filters in PySpark data frames using and and or logic, creating between-like conditions and combining criteria for precise data selection.

Learn to run sql on data frames using Spark sql, register a temporary table, and write queries to create data frames like full name of females from the original data.

Add columns in PySpark by removing dollar sign from salary with substring and casting to float to enable min, max, and average, then derive monthly salary and yes/no female indicator.

Group by and aggregation techniques in PySpark demonstrate cleaning salary data, casting to float, and computing total, average, min, and max salaries by gender and by city.

Learn to output a grouped dataframe to files by writing to csv, json, or parquet with a single line, and explore optional partitioning and compression.

Tackle a PySpark challenge using web log data to flag Mexico, sum bytes by yes/no, then count distinct IP addresses per country with group by and count distinct.

Learn a PySpark workflow to read a csv with header inference and add a home column, then group by home and country to sum bytes and count distinct ips, respectively.

Finish the course confident in PySpark and start seeing its value in your work. Ask questions in the Q&A, and I’ll be happy to help you out.