Weight Updates as Activation Shifts: A Principled Framework for Steering

摘要

Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices---such as intervention location and parameterization---that currently rely on empirical heuristics rather than a principled foundation. We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior. This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site. We further explain why certain intervention locations outperform others and show that weight updates and activation updates play distinct, complementary functional roles. This analysis motivates a new approach---joint adaptation---that trains in both spaces simultaneously. Our post-block steering method achieves accuracy within 0.4%-1.5% of full-parameter tuning while training only $0.04% of model parameters, consistently outperforming prior activation steering methods such as ReFT and PEFT approaches including LoRA while using significantly fewer parameters. Finally, we show that joint adaptation often surpasses the performance ceilings of weight and activation updates in isolation, introducing a new paradigm for efficient model adaptation.