InstEmb: Instruction-Following Embeddings through Glimpses of the Future

摘要

Recent advances have empowered large language models (LLMs) with remarkable fine-grained instruction-following capabilities in text generation tasks. However, embedding methods typically rely solely on the hidden state of the input's last token, limiting their ability to capture complete semantic signals distributed across the full output tokens. Moreover, existing discrete-to-continuous re-encoding approaches introduce semantic discontinuity. To address these limitations, we propose $\textbf{InstEmb}$, a novel instruction following embedding framework. InstEmb jointly optimizes two key aspects: (1) Input-Intrinsic semantic information, achieved by employing contrastive learning focused on the representation of the last input token, and (2) Output-Aware semantic information, captured through representation self-distillation leveraging learnable look-ahead tokens without introducing additional decoding latency. Additionally, we introduce $\textbf{Dual-Anchor Alignment Pooling (DAAP)}$, explicitly aligned with our dual training objectives. Extensive experiments demonstrate that InstEmb achieves state-of-the-art performance across multiple instruction following benchmarks without benchmark-specific supervised data.