Module 08 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. Two words in a freshly initialised nn.Embedding have cosine similarity 0.7. What does that mean?
- A. They are related
- B. Nothing — the rows are random until trained
- C. The embedding dimension is too small
- D. They will be confused by the model
2. Why does mean-pooling over a padded sequence need a mask?
- A. To speed it up
- B. Otherwise padding positions are averaged in and short sequences are diluted
- C. To avoid a shape error
- D. Masks are optional
3. What does bag-of-words discard?
- A. Word identity
- B. Word order
- C. Document length
- D. Vocabulary size
4. What is the structural limitation of an RNN's hidden state?
- A. It cannot handle numbers
- B. It is fixed-size, so signal from early positions fades over long sequences
- C. It requires a GPU
- D. It cannot be trained with backpropagation
5. In self-attention, what do each row's weights sum to?
- A. The sequence length
- B. 1
- C. 0
- D. The embedding dimension
---
Answer Key and Explanations
Check these only after attempting every question.
1. B — Nothing — the rows are random until trained
The geometry people describe is a result of training on a task, not a property the table has from the start.
2. B — Otherwise padding positions are averaged in and short sequences are diluted
Padding carries no information. Without the mask, a short sentence's representation is pulled towards zero by however much padding it needed.
3. B — Word order
'good service' and 'service good' produce identical vectors. It still works when meaning depends on word presence — including negation, when 'not' is in the vocabulary.
4. B — It is fixed-size, so signal from early positions fades over long sequences
That is the vanishing-gradient problem again, now along time. Attention removed both this and the sequential bottleneck.
5. B — 1
Softmax guarantees it. Each token's new representation is a weighted average of all tokens, with weights from similarity.
Practical Check
Apply this module to your own work: complete the module activity for *Embeddings, Sequences, and Attention Intuition*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.