Safety
These were some magical results from distillation by @geoffreyhinton that really shocked me when I first saw them, and TBH I still don’t ful…
These were some magical results from distillation by @geoffreyhinton that really shocked me when I first saw them, and TBH I still don’t fully understand it even to this date https://www.ttic.edu/dl/d
These were some magical results from distillation by @geoffreyhinton that really shocked me when I first saw them, and TBH I still don’t fully understand it even to this date https://www.ttic.edu/dl/dark14.pdf. The TLDR: distillation has incredible robustness wrt the training distribution, even when it has little to zero overlap with the target distribution. This also provides an interesting perspective to the now trendy topic of on vs off policy distillation — in particular why both are valid options despite their (big) difference.
Source: Jeremy Howard (X) | 2026-06-07