Unit ID: DLPY-M01-U02 Estimated active time: 25-35 minutes
More flexible means more ways to fail
Neural networks are flexible. That is their strength.
It is also their danger.
A flexible model can learn useful structure. It can also learn noise, shortcuts, leakage, spurious correlations, and patterns that do not survive outside the training data.
This is why deep learning needs care.
Data scale
Deep learning often needs more data than simpler models because it has many parameters. More parameters mean more ways to fit the training examples.
This does not mean every deep learning project needs millions of examples. Transfer learning can reduce the amount of task-specific data needed. But small data still requires caution.
Ask:
- How many examples do we have?
- Are the labels reliable?
- Are examples independent?
- Are important groups or cases missing?
- Is the validation set truly separate?
- Could the model see future information by mistake?
Compute
Deep learning can be expensive because training repeats many calculations.
The important terms are:
- CPU: general processor; good enough for small course examples.
- GPU: processor designed for many parallel calculations; useful for larger models.
- Device: where tensors and model parameters live, such as CPU or GPU.
- Batch: a small group of examples processed together.
- Epoch: one pass through the training data.
In this course, the required path is CPU-first. We will keep examples small.
Care
Care means the model is not trusted just because it trained.
Care includes:
- comparing with a baseline;
- plotting training and validation curves;
- inspecting errors;
- checking slices or groups where relevant;
- documenting data limits;
- avoiding private or sensitive data; and
- recording what the model must not be used for.
The dangerous sentence
Be careful with this sentence:
> The model learned it.
That sentence can hide too much.
What did it learn? From which data? With which labels? Under which split? Does it work on new examples? Where does it fail? What happens if the input changes?
Use more careful language:
> On this validation set, the model reduced loss compared with the baseline, but error examples show it still fails on short noisy messages.
That sentence is less flashy. It is much more useful.
Mini practice
A learner trains a neural network on 500 rows. The training score is excellent. The validation score is weak.
What are three possible explanations?
Good answers include:
- the model overfit;
- the dataset is too small;
- the training and validation examples differ;
- labels are noisy;
- features are not suitable;
- the learning rate or model size is wrong; or
- the split exposed a data problem.
The answer is not simply use a bigger network.