From Found to Designed: Concepts as a Design Axis for Large Language Models

arXiv:2607.26825v1 Announce Type: new Abstract: Large language models (LLMs) encode rich concept-like information, but represent it implicitly through distributed statistical associations rather than as explicit, structured, compositional concepts. Consequently, concept-level structure is typically found rather than designed: it is recovered after training through probing or dictionary learning, with no architectural guarantee of stability, compositionality, controllability, or alignment with human conceptual organization. We argue that concepts should instead be treated as a design axis for LLMs, and map the design space along two dimensions: the pipeline stage at which concept structure is introduced (training objective, core architecture, inference, or post-hoc interpretation), and whether that structure is internally derived from the model's own representations or grounded in external resources. This taxonomy reveals three broad patterns: inference-time approaches remain comparatively underexplored, related ideas have developed largely in isolation across pipeline stages, and externally grounded methods span the entire pipeline despite often being described under different terminology. Together, these observations motivate moving beyond recovering concept-like structure from trained models toward designing LLMs with explicit conceptual representations.

Related

• Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning
• How Large Language Models Balance Internal Knowledge with User and Document Assertions
• Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict

Source: arXiv cs.CL | 2026-07-30