T-POP: Test-Time Personalization with Online Preference Feedback
The Chinese University of Hong Kong, ShenZhen · College of Computer Science and Technology, Zhejiang University · Tianjin University · The Chinese University of Hong Kong, Shenzhen · Harbin Institute of Technology （Shenzhen） · Beihang University · City University of Hong Kong · Hong Kong University of Science and Technology (Guangzhou)
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions.