Arithmetic Without Algorithms

Abstract

Do large language models (LLMs) solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data? To investigate this question, we use arithmetic reasoning as a representative task. Using causal analysis, we identify a subset of the model (a circuit) that explains most of the model's behavior for basic arithmetic logic and examine its functionality. By zooming in on the level of individual circuit neurons, we discover a sparse set of important neurons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons is the mechanism used to produce correct arithmetic answers. To test this, we categorize each neuron into several heuristic types — such as neurons that activate when an operand falls within a certain range — and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a bag of heuristics.

Which model components participate in answering arithmetic prompts?

We employ activation patching to identify model components (MLP layers and attention heads) that make up the arithmetic circuit, responsible for completion of arithmetic prompts such as "226 - 68 =". We find an arithmetic circuit comprised of a spare set of attention heads and most MLPs.
We use linear probing to pinpoint the circuit components that promote the correct answer token. We find that the later MLPs are responsible for generating the correct answer only in the last position, utilizing earlier representations.

The effect of each model component on predicting the answer. The arithmetic circuit is comprised of a few attention heads that move operand and operator information to the last position, and MLPs that process this information to promote the correct answer.

Linear probes are only successful in extracting the answer after the later MLPs of the circuit.

How is the correct answer generated?

We dive into neuron-level resolution, and find that a small set of MLP neurons (roughly 1% in each layer) is enough to explain the model's arithmetic behavior.
Each such neuron acts as a "heuristic", identifying a specific input pattern and promoting a set of corresponding answer tokens.

LLMs answer arithmetic prompts by combining several unrelated heuristics, implemented by late-layer MLP neurons.
Each heuristic activates according to rules based on the input values of operands, and boosts the logits of corresponding result tokens.

We hypothesize that the combination of these unrelated heuristics is the mechanism used by LLMs to produce correct arithmetic answers.
To prove this, we develop an automated method to categorize each neuron into several heuristic types, such as neurons that activate when an operand falls within a certain range. We use this method to show a causal link between heuristic neurons and the model's accuracy on arithmetic prompts associated with these heuristics.

Knocking out neurons that implement heuristics associated with each prompt (full lines) leads to a greater decrease in accuracy than knocking out the same number of neurons whose heuristics are not associated with each prompt (dashed lines). This effect occurs across model sizes.

How do arithmetic heuristics develop over training?

We aim to check if the "bag of heuristics" emerges as the model's primary arithmetic mechanism early in training, or does it override some other mechanism.
We analyze the arithmetic heuristics across training checkpoints of Pythia-6.9B.
We discover that the model develops the "bag of heuristics" mechanism gradually across training, starting from an early checkpoint.
We show the heuristics mutual to the last checkpoint explain most of the model's accuracy on arithmetic prompts at each stage of training. This shows that, while other heuristics also exist in some form in early checkpoints, they are less important and become vestigial in later stages.
We show a causal link between the model's accuracy on arithmetic prompts and the heuristics associated with these prompts across training. This is evidence that across training, the model relies on the bag of heuristics to complete arithmetic prompts, making the existence of another algorithm-based mechanism unlikely.
The current lack of an arithmetic algorithm within LLMs suggests that improving their mathematical abilities may require fundamental changes to training and architectures, rather than post-hoc techniques like activation steering.

The arithmetic heuristics slowly develop over training.

Across training, ablating a few heuristics associated with each prompt, leads to a large decrease in accuracy.

How to cite

bibliography

Yaniv Nikankin, Anja Reusch, Aaron Mueller, Yonatan Belinkov, “Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics”.

bibtex

	@article{nikankin2024arithmetic,
	  title={Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics},
	  author={Nikankin, Yaniv and Reusch, Anja and Mueller, Aaron and Belinkov, Yonatan},
	  journal={arXiv preprint arXiv:2410.21272},
	  year={2024}
	}