Model Selection

What does model selection really mean?

Imagine you're predicting apartment prices in New York City. A linear regression is like drawing the best straight trend line through the data; a decision tree is like asking a sequence of yes/no questions; a neural network is like building a flexible maze of patterns. All can work—but not equally well.

Model selection means choosing the model that best fits your task, data, and constraints.

You usually match the model to the problem type:

• Regression: predict a number, like $2,800 rent
• Classification: predict a category, like spam vs not spam
• Clustering: group unlabeled data, like customer segments

A good choice balances:

• Accuracy
• Interpretability
• Training speed
• Amount of data available
• Risk of overfitting

The "best" model is not the fanciest one. It's the one that solves the real problem reliably.

Start with the problem, not the algorithm

A common beginner mistake is asking, "Should I use XGBoost or a neural net?" before asking, "What am I optimizing for?" That's like choosing hiking boots before knowing whether you're climbing Everest or walking to school.

Ask these questions first:

• What is the target? Number, category, ranking, or group?
• How much data do I have? 500 rows or 5 million?
• Do I need explanations? Banks and hospitals often do.
• How fast must predictions be? Ad bidding may need answers in milliseconds.
• What mistakes are costly? Missing fraud may be worse than flagging a few extra transactions.

For example:

• With small tabular data (say 2,000 customer records), try logistic regression, decision trees, or random forests.
• With images or audio, deep learning often shines.
• With strict interpretability, simpler models often win even if they lose a tiny bit of accuracy.

How do we compare models fairly?

You don't judge a student by the homework they copied from the answer key. Similarly, you shouldn't judge a model only on the data it trained on. Instead, split data into training and validation/test sets.

A common metric for classification is accuracy:

accuracy = correct predictions / total predictions

Worked example:

• 90 correct predictions out of 120 patients
• accuracy = 90 / 120 = 0.75 = 75%

But metrics depend on the task:

• Regression: MAE, RMSE, R²
• Classification: accuracy, precision, recall, F1

Here's a simple Python example:

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = DecisionTreeClassifier(max_depth=3)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(accuracy_score(y_test, preds))

The key idea: compare models on unseen data, not just training performance.

Trade-offs: accuracy vs simplicity vs speed

Model selection is often a negotiation, not a contest. A random forest may beat logistic regression by 2 percentage points, but if the simpler model trains in 1 second, explains its decisions, and is easier to maintain, it may be the better business choice.

Think of common trade-offs:

• Simple models: faster, easier to explain, may miss complex patterns
• Complex models: powerful, but harder to tune and easier to overfit
• Large models: may need more data and computing power

A practical workflow is:

1. Start with a simple baseline
2. Measure on validation data
3. Try more complex models only if needed
4. Prefer the simplest model that meets the goal

If a sales team needs a churn model by Friday, a well-tested logistic regression may be more valuable than a half-finished neural network.