Overfitting is when a model learns the training data too closely, including its noise and quirks, instead of the general pattern. It scores very well on the training set but poorly on new, unseen data.
Typical sign: training accuracy keeps rising while validation accuracy stalls or falls.
Ways to reduce it
- More (or augmented) training data, so noise averages out.
- A simpler model: fewer layers, parameters or tree depth; prune decision trees.
- Feature selection: drop irrelevant or redundant features.
- Regularization: L1 (Lasso) / L2 (Ridge) penalties on large weights.
- Cross-validation (e.g. k-fold) to choose the model and its hyperparameters on held-out data.
- Early stopping: stop training when validation loss starts rising.
- Dropout in neural networks.
- Ensembles such as bagging / random forests, which average out individual models’ variance.
(The printed question says “reducing training dataset”; the usual intent is reducing overfitting, answered above. Note that shrinking the training set generally makes overfitting worse, not better.)
— rhs