FFN Lens: How Transformers Divide Labor for Multilingual Tasks

摘要

Large Language Models (LLMs) demonstrate strong performance in multilingual tasks, yet the process of constructing predictions in the target language remains under-explored. In this work, we introduce the FFN Lens, a novel interpretability method focusing on the Transformer’s core computational module, the Feed-Forward Network (FFN). By directly leveraging model parameters, the FFN Lens identifies both the critical units responsible for constructing specific information and the input features that drive them, which is essential for understanding Large Language Models. Applying FFN Lens to multilingual tasks, we demonstrate the prediction construction process and reveal the distinct division of labor across model layers. We identify a three-stage functional pipeline for constructing multilingual predictions: Latent Translation, Semantic Mapping, and Self Emphasis. We further introduce subspace analysis to validate this three-stage mechanism from a complementary perspective, and leverage these mechanistic insights to propose a training-free uncertainty estimation method.