Shaping capabilities with token-level data filtering
Neil Rathi, Alec Radford
Token filtering during pretraining effectively reduces unwanted language model capabilities while maintaining alignment, becoming more effective at larger scales and tolerating noisy labels with sufficient compute.