Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment

This NeurIPS 2026 workshop paper examines how well LLM judges evaluate personalized writing tasks, testing how the writing task, generator-judge pairing, evaluation objective, and general reasoning ability affect judge performance across four LLMs and three long-form writing tasks with human pairwise labels. The authors find judge quality correlates weakly with MMLU-measured ability (Pearson's ρ ≤ 0.2), varies by task and criteria, and that paired comparisons yield more consistent results than independent scoring, while the strongest LLMs do not most reliably reproduce human evaluations.