Training and Evaluation

Training vs. Evaluation

Think of machine learning like training for a race: practice builds skill, but only a fair timed trial tells you how good you really are. In ML, the model learns patterns from a training set, then we judge it on separate evaluation data it has never seen before.

If you train and test on the same examples, the model can simply memorize. That may look impressive, but it won't generalize. A common split is:

• 80% training
• 20% test

For example, with 1,000 house listings, you might train on 800 and evaluate on 200. During training, the model adjusts its internal parameters to reduce error. During evaluation, you freeze the model and ask: How well does it perform on new data?

That separation is the heart of trustworthy machine learning.

What Happens During Training?

Training is an iterative search for better parameters. Imagine adjusting a recipe after every taste test: less salt, more heat, repeat. A model does something similar by making predictions, measuring error, and updating itself to reduce that error.

In supervised learning, each training example has an input and a correct answer. The model predicts, compares, and learns.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)   # training
preds = model.predict(X_test) # evaluation

A danger here is overfitting: the model becomes excellent at the training set but poor on fresh data. It's like memorizing past exam questions instead of understanding the subject. Good training aims for patterns that transfer, not just patterns that repeat.

Measuring Performance: Accuracy, Precision, Recall

Different tasks need different scorecards. For a classifier, accuracy is the fraction of correct predictions:

accuracy = correct predictions / total predictions

If a model gets 90 out of 100 examples right, accuracy = 90/100 = 0.90 = 90%.

But accuracy can mislead. Suppose only 5 out of 100 patients have a disease. A lazy model that predicts "no disease" for everyone gets 95% accuracy—yet it misses every sick patient.

That's why we also use:

• Precision = among predicted positives, how many were truly positive?
• Recall = among actual positives, how many did we find?

Worked example: model predicts 10 fraud cases, and 8 are truly fraud. There are actually 20 fraud cases total.

• precision = 8/10 = 0.80
• recall = 8/20 = 0.40

So the model is usually right when it flags fraud, but it misses many fraud cases.

Validation, Test Sets, and Fair Comparison

If the test set is the final exam, the validation set is your practice quiz. You use validation data to compare models, tune settings, or choose features. Only after those decisions are finished should you touch the test set.

A common split looks like:

• 70% training
• 15% validation
• 15% test

With 10,000 customer records, that means 7,000 for learning, 1,500 for tuning, and 1,500 for final evaluation. Why not keep checking the test set? Because then you start indirectly optimizing for it, and it stops being a fair measure.

The test set should feel like tomorrow's data.

Good evaluation is not just about getting a high number. It's about getting a number you can trust.