Research
Interpreting Object-Dependent Concept Brittleness in Text-to-Image Diffusion Models
arXiv:2609.09909v1 Announce Type: new Abstract: Although text-to-image diffusion models generally exhibit strong prompt-following ability, we identify a persistent and previously underexplored failure
arXiv:2609.09909v1 Announce Type: new Abstract: Although text-to-image diffusion models generally exhibit strong prompt-following ability, we identify a persistent and previously underexplored failure pattern in which a small subset of prompts differing only in the object consistently fails to realize the same target concept under identical generation settings. We term this phenomenon object-dependent concept brittleness. Such cases suggest systematic internal blind spots rather than random sampling noise. In this paper, we present an interpretability-oriented framework to audit and minimally correct these failures. Our key idea is to analyze denoising trajectories in a step-wise sparse autoencoder (SAE) space, where abstract style and attribute concepts become more separable than in the raw denoising representation. This sparse space enables us to compare successful and failed generations, identify concept dimensions whose evidence is missing, weakened, or temporally delayed, and construct class-level concept prototypes from reliable class-consistent samples. Based on this audit process, we introduce a lightweight inference-time correction strategy that interpolates denoising features toward the corresponding prototype in SAE space. Rather than serving as a task-specific retraining method, this intervention acts as a validation of the diagnosed concept deficiency. We evaluate the proposed framework on style and attribute failure cases across multiple diffusion backbones, with significant improvements in concept consistency, text fidelity, and repair success. Further analyses show that deeper denoising representations provide clearer concept structure, while early-stage intervention offers the strongest correction leverage. Code is available at https://github.com/Metecade/Object-Dependent-Concept-Brittleness.
Related
- Concept Unlearning by Modeling Key Steps of Diffusion Process
- Dynamic Eraser for Guided Concept Erasure in Diffusion Models
- Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks
- To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion
- A Unified Framework for Diffusion Model Unlearning with f-Divergence
Source: arXiv cs.CV | 2026-09-10