What Does a System Modify When It Modifies Itself?

arXiv:2603.27611v2 Announce Type: replace Abstract: When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates its revisions? Cognitive science describes executive control, metacognition, and hierarchical learning, while artificial intelligence modifies policies, parameters, and learning mechanisms, but the two fields lack common criteria for comparing these transformations. We propose a minimal analytical model distinguishing functional rules Phi_t = {R0, ..., Rk}, modification mechanisms Mt, and evaluation criteria Nt. It separates two independent dimensions: the depth of the modified target and whether the modification is blind or reflexive. Internal representational access AtR and endogenous causal control AtC are further distinguished from external inspectability and modifiability by a designer. Four regimes are defined by the target of modification: action without organizational modification, modification of low-level rules, modification of control rules or mechanisms, and revision of the evaluation criterion. Each is related to cognitive phenomena and artificial systems while making explicit the system boundary on which attribution of endogenous self-modification depends. The comparison yields a conditional crossed-opacities hypothesis: humans often have richer endogenous self-description at abstract and strategic levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The framework also identifies three difficulties - viability, evaluation of revisions to evaluation criteria, and continuity of identity - and proposes two empirical discriminants for identifying the target of a modification and the causal role of self-representation.

Related

• Can a Dynamic Internal Field Govern a Transformer's Cognition? Certifiability, not Superiority, in Homeostatic Compute Control
• Does Your Reasoning Model Implicitly Know When to Stop Thinking?
• Consciousness as Uncommon Self-Knowledge: A Synergistic Information Framework
• What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say 'I Don't Know'

Source: arXiv cs.AI | 2026-09-10