Overfitting

Overfitting occurs when a model learns peculiarities or noise in its training data so closely that performance deteriorates on new data.

Training versus generalisation

A very flexible model can have extremely small training error while having larger test error. Therefore training fit alone is not a sufficient measure of model quality.

Reducing overfitting

Approaches include collecting more informative data, reducing model complexity, regularisation, appropriate feature selection, early stopping and cross-validation.

Underfitting

The opposite problem occurs when a model is too simple to represent important structure, leading to poor performance even on training data.