From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

arXiv:2610.21149v1 Announce Type: new Abstract: Self-generated In-Context Learning (ICL) enables LLMs to synthesize their own few-shot demonstrations, often outperforming manual prompts. However, existing methods filter demonstrations by observation, assuming a rationale is high-quality if it leads to the correct output. We show such filtering suffers from extit{observational equivalence}, whereby both faithful causal rationales and spurious post-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone. Inspired by independent causal mechanisms, we argue that faithful rationales imply extit{transfer invariance}, in which causal rationales should remain effective across logically similar samples. We instantiate this criterion as extbf{E}xecutable extbf{C}ausal extbf{I}ntervention for extbf{D}emonstrations extbf{D}istillation (ECI2D), which abstracts language rationales into executable programs and applies the causal interventions (do-operator) on logically similar samples, retaining only those demonstrations that imply transfer invariance. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts.

Source: arXiv cs.CL | 2026-10-16