On Code-Induced Reasoning in LLMs

摘要

Code data has been shown to enhance the reasoning capabilities of large language models (LLMs), but it remains unclear which aspects of code are most responsible. We investigate this question with a systematic, data-centric framework. We construct parallel instruction datasets across ten programming languages and introduce controlled perturbations that selectively disrupt structural and semantic properties of code. We then fine-tune LLMs from five model families and eight scales on each variant and evaluate their performance on natural language, math, and code tasks. Across 3,331 experiments, our results show that LLMs are more vulnerable to structural perturbations than semantic ones, particularly on math and code tasks. Appropriate abstractions like pseudocode and flowcharts can be as effective as code, while encoding the same information with fewer tokens without adhering to original syntax can often retain or even improve performance. Notably, even corrupted code with misleading signals remains competitive when surface-level regularities persist. Finally, syntactic styles also shape task-specific gains, with Python favoring natural language reasoning and lower-level languages such as Java and Rust favoring math. Through our systematic framework, we provide a fine-grained analysis of how different aspects of code influence reasoning and inform the design of training data for enhancing LLM reasoning capabilities.