Introduction to PySpark — PySpark

Learning objective

In this lesson, you will understand the purpose and architecture of PySpark and set up a working PySpark environment. This is the foundation for the rest of the course: before you can transform data, build pipelines, or run distributed jobs, you need a clear mental model of what PySpark is and how it starts.

Why PySpark matters

PySpark is the Python interface to Apache Spark, a distributed computing engine designed for large-scale data processing. If pandas works well on a single machine, Spark is built for datasets that are too large, too slow, or too operationally complex for one computer.

You use PySpark when you need to:

process data across a cluster of machines
work with files in systems like HDFS, S3, or cloud data lakes
run the same logic on 10 MB today and 10 TB later
combine Python productivity with Spark's distributed execution engine

In short, PySpark is about scaling familiar data workflows beyond the limits of local memory and CPU.

The big idea: Python on top of a distributed engine

PySpark is not a separate engine from Spark. Instead, it is a Python API that talks to the Spark engine running on the JVM. That means your Python code describes work, while Spark plans and executes that work efficiently across resources.

Core architecture

A basic Spark application includes:

Driver: the main program that coordinates the job
SparkSession: the entry point you use in PySpark code
Executors: worker processes that actually perform tasks
Cluster manager: software such as Standalone, YARN, or Kubernetes that allocates resources

A useful mental model is this:

you write df.filter(...) in Python
Spark builds an execution plan
tasks are sent to executors
results are combined and returned

This separation is why Spark can scale. Your code stays compact, but the engine can split the work across many cores or many machines.

A concrete example: reading data with Spark

Suppose an online retailer has a sales.csv file with 50 million rows. On one machine, loading and cleaning it may be slow or impossible if memory is limited. In PySpark, you can express the task in a few lines:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("IntroExample") \
    .getOrCreate()

sales_df = spark.read.csv("sales.csv", header=True, inferSchema=True)

filtered_df = sales_df.filter(sales_df.amount > 100)
filtered_df.select("customer_id", "amount").show(5)

What matters here is not just the syntax. Spark can divide the file into partitions, process those partitions in parallel, and optimize the plan before execution.

This is also a primary workflow pattern you will use throughout the course:

create a SparkSession
read data into a DataFrame
transform it
inspect or write results

Setting up a PySpark environment

For learning and local development, the simplest setup is a local Spark session on your own machine. A common path is:

install Python 3.10+ or similar
install Java because Spark runs on the JVM
install PySpark with pip
test that Spark starts successfully

Example commands:

python -m pip install pyspark

Then test it:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .master("local[*]") \
    .appName("SetupTest") \
    .getOrCreate()

print(spark.range(5).collect())

If this works, Spark is running in local mode, where local[*] means "use all available CPU cores on this machine." That is perfect for learning the API before moving to a real cluster.

In professional environments, the same PySpark code may later run on Databricks, EMR, Google Dataproc, or a Kubernetes-based Spark platform.

Introduction to PySpark