Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses

arXiv:2609.09902v1 Announce Type: new Abstract: Reading a transformer's internal states in token space is easy to do and hard to trust: a logit lens on a single hidden state is dominated, at intermediate layers, by the generic tokens the model would predict for almost any input. We read the difference instead. Subtracting two closely matched prompts' hidden states and projecting through the unembedding cancels the shared component and surfaces what separates them, an operation equivalent to reading a RepE/ActAdd steering vector through a logit lens. Built into a training-free tracer that reads at every position, sub-layer, and head and averages over designed baselines, it traces a compound- noun MLP->attention chain in Phi-2, confirmed there by activation patching, with the same distinction recovered across three architectures by readout and probe rather than by patching; it reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a set of domain-to-domain mappings rather than a single figurativity feature. A cross-seed control marks the boundary: across five networks differing only in initialization, the same distinction surfaces as almost entirely different tokens (top-10 overlap 0.08). What a computation looks like in token space is network-specific; the distinction it draws is not

Related

• The First Token Is a Clue: Verbalizing Multi-Token Concepts from the J-lens
• The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models
• Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model

Source: arXiv cs.CL | 2026-09-10