Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Xin Li, Hao Jiang, Xin Gao +6 authors
Multi-teacher on-policy distillation merges specialist language models but suffers from unbalanced feedback across domains, which domain-normalized rescaling fixes to recover lost skills.