A model's quality is measured by comparing its predictions with the true values on test data: for classification use the confusion matrix with accuracy, precision, recall and F1-score; for regression use MAE, MSE, RMSE and \(R^2\).
1. Classification: confusion matrix
| Predicted positive | Predicted negative |
|---|
| Actual positive | TP | FN |
| Actual negative | FP | TN |
- Accuracy \(= \dfrac{TP + TN}{TP + TN + FP + FN}\)
- Precision \(= \dfrac{TP}{TP + FP}\) (how many predicted positives are correct)
- Recall / sensitivity \(= \dfrac{TP}{TP + FN}\) (how many actual positives are found)
- Specificity \(= \dfrac{TN}{TN + FP}\)
- F1-score \(= \dfrac{2PR}{P + R}\) (balance of precision P and recall R)
- ROC curve and AUC: plot TPR against FPR for different thresholds; AUC near 1 is good.
Example: TP = 40, FN = 10, FP = 5, TN = 45. Accuracy = 85/100 = 0.85; precision = 40/45 = 0.89; recall = 40/50 = 0.80; F1 = 2(0.89)(0.80)/(1.69) ≈ 0.84.
2. Regression (\(y_i\) actual, \(\hat{y}_i\) predicted, n points)
- MAE \(= \frac{1}{n}\sum |y_i - \hat{y}_i|\)
- MSE \(= \frac{1}{n}\sum (y_i - \hat{y}_i)^2\); RMSE \(= \sqrt{\text{MSE}}\)
- \(R^2 = 1 - \dfrac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}\) (fraction of variance explained; 1 is perfect)
3. How to measure reliably
- Evaluate on data not used for training (train–test split, e.g. 80:20).
- Use k-fold cross-validation and average the score over the folds.
- Compare training and test scores to detect overfitting (high train, low test) or underfitting (both low).
- For clustering, use measures such as the silhouette coefficient.
