First, the key terms.

Entity leakage
Information shared across splits through repeated people, devices or organizations.
Temporal leakage
Future information entering a prediction about an earlier time.

The idea, explained

Leakage can be obvious, such as including an eventual outcome as an input, or subtle, such as preprocessing the full dataset before splitting. Duplicate users across splits can let a model recognize people rather than solve the intended task. Time-dependent problems often require a chronological split. A valid evaluation reproduces what the system would have known at the moment it had to act, including data collection delays.

STEP 1

Write the availability timeline

For each feature, record when it is collected and when the deployed system can access it. A column’s presence in a historical database does not prove that it was available when the original decision was made.

STEP 2

Split the real unit of independence

If one patient has many records, row-wise random splitting may place the same patient in training and testing. Split by the appropriate entity or time boundary, then fit preprocessing using training data only.

Follow a worked example.

A model forecasts loan default but includes a collection-agency flag added after default. That flag makes historical performance look excellent while being unavailable when the lending decision is actually made.

Walk it through, one step at a time

  1. A model predicts whether equipment will fail tomorrow. The dataset includes a repair code entered after a failure.
  2. Remove the future repair information and test on a later time period. The score may drop sharply, but the result is more representative of deployment.
  3. Also check whether duplicate sensor sequences appear in both periods, which could create another shortcut.

Where people get stuck

A random split alone does not prevent leakage through repeated entities, future information or shared preprocessing.

Now try it yourself.

Pause before opening each answer. Explain your reasoning in your own words.

1. Check the core idea

Why should a scaler usually be fitted on the training partition?

Reveal the answer & reasoning

Fitting on the full dataset lets information from held-out examples influence preprocessing. Fit the transformation on training data and apply it unchanged to the held-out data.

2. Apply what you learned

Is a lower score after removing leakage evidence that the correction was a mistake?

Reveal the answer & reasoning

No. The old score may have measured access to information the deployed model would never possess. A realistic lower score is more useful than a misleading high one.