First, the key terms.

Validation set
Examples used to compare model choices during development.
Regularization
A constraint or penalty intended to limit undesirable fitting behavior.

The idea, explained

Training error usually becomes easier to reduce as a model becomes more flexible. The important question is whether performance transfers to representative unseen cases. Regularization can penalize large weights, stop training early or otherwise limit complexity. Validation data helps compare choices; a final test set should remain separate from repeated tuning. The right balance depends on data quantity, noise, architecture and deployment conditions, rather than a universal rule that simpler is always better.

STEP 1

Compare separate curves

Measure training and validation error as training progresses. Falling training error with worsening validation error is evidence of overfitting under that validation setup. First ensure the split is meaningful and the two measurements are comparable.

STEP 2

Change one cause at a time

Try better data coverage, a more appropriate model, a regularization setting or early stopping. Keep the final test set for a later assessment. A better validation score after many experiments can still reflect selection overfitting.

Follow a worked example.

A model memorizes which teacher wrote each practice question and scores well on practice exams. When a new teacher writes the exam, accuracy collapses. Testing on questions from unseen teachers would have exposed this problem earlier.

Walk it through, one step at a time

  1. At epoch 5, training error is 10 and validation error 12. At epoch 20, they are 2 and 18.
  2. Later training fits the development examples more closely but transfers worse to this validation set.
  3. Consider the earlier checkpoint and investigate the mismatch. Do not automatically declare it the universally best model; confirm on a suitable independent test.

Where people get stuck

Repeatedly choosing a model based on the same test set turns that set into part of the tuning process.

Now try it yourself.

Pause before opening each answer. Explain your reasoning in your own words.

1. Check the core idea

Can very low training loss coexist with poor future performance?

Reveal the answer & reasoning

Yes. Training loss measures fit to the training examples, not transfer to different examples or changing conditions.

2. Apply what you learned

Why is checking a final test set after every tuning change problematic?

Reveal the answer & reasoning

The test results influence development choices. The set then becomes part of the tuning process, making its apparent independence misleading.