Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment

摘要

Despite their strong general capabilities, large language models (LLMs) often remain unreliable when outputs must be numerically precise. A key reason is the training objective: standard cross-entropy treats numeric tokens as unstructured categories and ignores the metric structure of their values. We address this mismatch by proposing **S**mooth **M**aximum **M**ean **D**iscrepancy (**SMMD**), which builds on the classic MMD by incorporating value-distance kernels over numeric tokens and graph-based smoothness. With this kernel defined over a numeric sub-vocabulary, SMMD aligns the predicted numeric distribution to the target via kernel matching and smooths the prediction--target residual over the induced kernel graph to encourage local consistency. We evaluate SMMD on four numeric-target tasks---mathematical reasoning, arithmetic calculation, clock-time recognition, and chart question answering---across multiple open-weight LLM and VLM backbones. SMMD consistently improves accuracy over both cross-entropy and recent numeric-target losses; analyses show complementary effects between MMD and smoothness and underscore the importance of distance-based kernel design.