World Cup Big Data Analysis

01 Summary

Academic Big Data project that completes an assessment guide on the FIFA World Cup using Apache Spark (PySpark). It integrates five data sources (teams, stadiums, players, matches in CSV, and tournaments in multiline JSON) into a relational model, exercising Spark's three main APIs: RDDs, DataFrames, and Spark SQL. The notebook covers everything from data cleaning to joins, repartitioning, calculated columns with business rules, and ten analytical aggregations using declarative SQL.

02 Highlights

• ✦End-to-end PySpark pipeline integrating five data sources into a relational model with foreign keys.
• ✦Ten analytical aggregations in Spark SQL over temporary views, plus joins with column ambiguity resolution.
• ✦Calculated columns applying business rules (player BMI, age and score labeling).
• ✦Explicit partitioning control to leverage parallelism in RDDs and DataFrames.

03 Screenshots