PromptResponse: Optimizing Prompts for LLM Coding Tasks

arXiv:2608.21074v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to input prompt variations. This paper presents nicode{x00AB}PromptResponsenicode{x00BB}, a controlled study examining how formatting and LLM-based tuning of coding task prompts affect the resulting code's performance, efficiency, and stability. Using five semantically identical yet syntactically distinct variants of the HumanEval datasetnicode{x2014}baseline, JSON, Markdown, YAML, and an LLM-tuned versionnicode{x2014}we had GPT-4o solve its coding problems over 8200nicode{x00A0}executions. Our results show that consistent formattingnicode{x2014}especially JSONnicode{x2014}improves generation efficiency and syntactic stability, with minor gains in task performance. Conversely, the LLM-tuned prompts resulted in significantly degraded task performance without significant improvements in any other dimension. These findings suggest that low-effort reformatting alone can yield measurable improvements, while tuning must account for model alignment. We conclude our work with providing a set of practical recommendations informed by our results as well as releasing our dataset variants and evaluation pipeline for future work.

Related

• Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review
• CAPO: Constraint-Aware Prompt Optimization for LLM Agents
• A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

Source: arXiv cs.AI | 2026-08-24