Data Preprocessing

Why preprocessing matters

Raw data is rarely ready for machine learning. A customer table might contain ages like 29, 31, and unknown, incomes written as 50000 and $50,000, and cities entered as both NYC and New York. A model sees this mess literally.

Data preprocessing is the step where we clean and reshape data so patterns become learnable. Think of it like washing, chopping, and measuring ingredients before cooking: the recipe matters, but prep determines whether the dish works.

Common preprocessing tasks include:

• handling missing values
• fixing inconsistent formats
• encoding categories like red, blue, green
• scaling numeric features
• removing duplicates or obvious errors

Without preprocessing, your model may learn noise, overweight the wrong features, or fail entirely.

Cleaning: fix what is broken first

Start by finding problems that would confuse a model. Suppose an online store has these rows:

• Age = 34, Country = US, Spent = 120.5
• Age = ?, Country = U.S., Spent = 120.50
• Age = 340, Country = United States, Spent = 0

Here, ? is a missing value, 340 is likely an outlier or typo, and the country labels are inconsistent. Basic cleaning often includes:

• replacing or imputing missing values, such as using the median age
• standardizing labels, like mapping US, U.S., and United States to US
• removing duplicates
• flagging impossible values, like age 340

A simple rule: clean for meaning, not perfection. You are trying to preserve real signal while removing accidental mess.

Transforming data so models can use it

After cleaning, we often transform features into forms models handle better. Two common steps are encoding and scaling.

If Color is red, blue, green, many models need numbers, so we use one-hot encoding:

• red → [1,0,0]
• blue → [0,1,0]
• green → [0,0,1]

For numeric features, scaling prevents large units from dominating. A common method is min-max scaling:

x_scaled = (x - min) / (max - min)

Example: if house sizes range from 800 to 2800 square feet, then a house of 1800 becomes:

(1800 - 800) / (2800 - 800) = 1000 / 2000 = 0.5

Now size sits neatly between 0 and 1, making it easier for many algorithms to compare features fairly.

A simple preprocessing workflow

In practice, preprocessing is a repeatable pipeline, not a random list of fixes. A common order is: inspect, clean, transform, then train.

import pandas as pd
from sklearn.preprocessing import MinMaxScaler

# fill missing ages with median
 df['age'] = df['age'].fillna(df['age'].median())

# standardize country labels
 df['country'] = df['country'].replace({'U.S.': 'US', 'United States': 'US'})

# one-hot encode a category
 df = pd.get_dummies(df, columns=['color'])

# scale a numeric column
 scaler = MinMaxScaler()
 df[['income']] = scaler.fit_transform(df[['income']])

One crucial habit: fit preprocessing steps using training data only, then apply the same rules to validation or test data. Otherwise, you accidentally leak future information into training and get overly optimistic results.