From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs

arXiv:2505.14368v3 Announce Type: replace-cross Abstract: Recent studies demonstrate that Large Language Models (LLMs) are vulnerable to attacks that generate harmful or sensitive outputs. As open-source LLMs are increasingly adopted in high-impact applications such as finance, law, and healthcare, systematically investigating their security risks is becoming increasingly important towards a trustworthy LLM era. This paper comprehensively studies effective prompt injection attacks against 14 widely used open-source and three closed-source LLMs on five attack benchmarks. Moreover, existing evaluation metrics mostly only consider the attack success rate, overlooking uncertainty in model responses. Our proposed Attack Success Probability (ASP) additionally captures uncertain behaviors for evaluation, where the model may initially refuse a harmful request but subsequently provide harmful guidance or vice versa, reflecting inconsistency and ambiguity in attack feasibility. By systematically analyzing the effectiveness of prompt injection attacks, we propose a straightforward and effective hypnotism attack; results show that this attack causes aligned language models, including StableLM2, Mistral, Openchat, and Vicuna, to generate objectionable behaviors, achieving around 90% ASP. We also find that moderately well-known LLMs exhibit higher vulnerability to prompt injection attacks, highlighting the need to raise public awareness and prioritize efficient mitigation strategies.

Related

• Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysis
• A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks
• Mind the Gap: Evaluating Model- and Agentic-Level Vulnerabilities in LLMs with Action Graphs
• ContextualJailbreak: Evolutionary Red-Teaming via Simulated Conversational Priming

Source: arXiv cs.CL | 2026-09-28