Unit 06.01: Why transfer learning works, and when it does not

Transfer learning works because edges are edges in every domain. It works less well the deeper you go.

Early layers transfer; late layers may mislead

Four layer groups sorted by what they learn and how well it transfers.

The code lists them.

LAYERS = [
    ("early layers",  "edges, colours, simple textures", "transfer almost always"),
    ("middle layers", "corners, repeated patterns, parts", "transfer usually"),
    ("late layers",   "whole-object concepts from the source task",
     "transfer only if the tasks are similar"),
    ("the classifier", "the source task's exact classes", "always replaced"),
]
print(f"{'layer group':16} {'learns':38} transfers?")
for group, learns, transfers in LAYERS:
    print(f"{group:16} {learns:38} {transfers}")

print("""
Transfer learning works because edges are edges in every domain. It works less
well as you go deeper, and the further your task is from the source, the more
layers you need to retrain.

Photographs to photographs: excellent. Photographs to X-rays or microscopy:
the early layers still help and the late ones may actively mislead.
""")

Photographs to photographs transfers excellently. Photographs to X-rays or microscopy is the interesting case: the early layers still help, and the late layers encode whole-object concepts from natural images that have no counterpart in a radiograph.

The further the domain, the more layers you need to retrain - which increases the amount of data you need, sometimes past what you have.

The mistake this prevents

The mistake is assuming transfer learning removes the data requirement. It reduces it in proportion to how similar your task is to the source, and for a genuinely different domain the reduction can be small.

Takeaway

Early layers transfer almost always; late ones transfer only between similar domains. Distance from the source task determines how many layers you must retrain and how much data that needs.

Mark complete