Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches

arXiv:2512.12677v2 Announce Type: replace-cross Abstract: We explore efficient strategies to fine-tune decoder-only Large Language Models (LLMs) for downstream text classification under resource constraints. Two approaches are investigated: (1) attaching a classification head to a pre-trained causal LLM and fine-tuning on the task using the LLM's final-token embedding as a sequence representation, and (2) instruction-tuning the LLM in a prompt-to-response format for classification. To enable single-GPU fine-tuning of models up to 8B parameters, we combine 4-bit model quantization with Low-Rank Adaptation (LoRA) for parameter-efficient training. Experiments on two patent benchmarks, a proprietary 5-class single-label corpus and the public WIPO-Alpha multi-label dataset with 14 categories, show that the embedding-based method matches or exceeds the instruction-tuned method on single-label classification while training 10 to 30 times fewer parameters. Instruction-tuning is competitive only in the multi-label regime, and only with substantially larger trainable budgets of at least 100M parameters. Both methods are very competitive with fine-tuned domain-specific BERT models, and on the single-label task they surpass them. Paired McNemar tests and bootstrap Delta F1 95 percent confidence intervals confirm that the numerical advantage of the embedding-head approach is consistent in direction but not statistically certified at p < 0.05. We further validate single-label generalization on AG News and report ablations on pooling, verbalizer choice, and calibration, together with a distillation recipe that recovers BERT-class throughput. We discuss the advantages of each approach while outlining practical guidelines and future directions for optimizing LLM fine-tuning in classification scenarios.

Source: arXiv cs.AI | 2026-05-25