World Cup Big Data Analysis

01 Problem and context

Distributed processing fundamentally changes how analyses are written: operations are planned and distributed across partitions prior to execution, and decisions like join ordering or partition counts carry more weight than the query logic itself. A testbed with enough interconnected entities was needed to put this properly into practice.

Distributed processing changes how an analysis is written: operations are planned and spread across partitions before anything executes, and choices such as join order or partition count weigh more than the query itself. This work uses a relational sports model as a bench for exercising all three Apache Spark APIs over the same data: RDDs, DataFrames and Spark SQL. The object of study is the pipeline and how each API behaves, not the sporting conclusions.

02 Solution

A relational model across five sources in two formats (four CSV files and one multiline JSON) linked by foreign keys, used to practice with all three Spark APIs: RDDs, DataFrames, and Spark SQL. It features explicit type casting over schema inference, manual partitioning control to observe its impact on the execution plan, and ten aggregations comparing the declarative syntax against its DataFrame equivalent.

1. 01Relational modelling of five sources in two formats: four CSV files plus a multiline JSON, linked by foreign keys.
2. 02Ingestion and cleaning: header normalisation and explicit type casting rather than relying on automatic schema inference.
3. 03Integration: joins across the five entities resolving same-name column ambiguity, and reading the nested JSON with multiline enabled.
4. 04Explicit partition control to exercise parallelism and observe how it changes the execution plan.
5. 05Derived columns from conditional rules.
6. 06Ten analytical aggregations in Spark SQL over temporary views, set against their DataFrame API equivalents.
7. 07Environment work to get the pipeline running end to end locally, rewriting the RDD operations that failed on serialisation.

03 Results

Verifiable figures in the repository and its documentation:

04 How it's built

Big Data project on the FIFA World Cup using Apache Spark (PySpark). It integrates five data sources (teams, stadiums, players, matches in CSV, and tournaments in multiline JSON) into a relational model, exercising Spark's three main APIs: RDDs, DataFrames, and Spark SQL. The notebook covers everything from data cleaning to joins, repartitioning, calculated columns with business rules, and ten analytical aggregations using declarative SQL.

05 Ethical decisions

• ✦Player records are synthetic, with names, ages and measurements generated for the exercise. That avoided handling real people's biometric data for a task that never required it.
• ✦The labels derived from body-mass index are computed over fictional profiles and pass no judgement on any real person.

06 Highlights

• ✦End-to-end PySpark pipeline integrating five data sources into a relational model with foreign keys.
• ✦Ten analytical aggregations in Spark SQL over temporary views, plus joins with column ambiguity resolution.
• ✦Calculated columns applying business rules (player BMI, age and score labeling).
• ✦Explicit partitioning control to leverage parallelism in RDDs and DataFrames.

07 Limitations

The dataset is synthetic and small: fifteen teams, eighty players, a hundred matches, ten stadiums and three tournaments, with generated rankings and scores. Two limits follow, and it is better to state them first. None of the sporting conclusions describes real competitions; they are correct queries over invented data. And at this volume Spark buys no performance at all, since coordination overhead dominates the computation, so partitioning is exercised as engine mechanics rather than as measured optimisation.