New research published on ArXiv introduces RENDER, a benchmark control demonstrating that the way an AI system "reads" its conversation history is not merely an implementation detail, but a critical factor in performance. While memory and RAG (Retrieval-Augmented Generation) evaluations often treat input formatting as secondary, RENDER shows that the presentation of the same history—as a summary, a record, or a raw excerpt—drastically changes outcomes.
The "Reader-Facing" Artifact
The study utilizes a five-level packet ladder to localize exactly when answer-bearing content enters the model's input. By employing deterministic templates that approximate popular systems like ChatGPT-style entries, LangChain summaries, and MemGPT-style typed records, the researchers isolated the impact of formatting on nine different LLMs using 500 LongMemEval questions.
Key Performance Metrics
The results highlight a significant sensitivity to how evidence is rendered:
- Matched-budget resolved packets outperformed recency-truncated raw dialogue by a margin of 42.4 to 72.6 points.
- The spread between the best and worst performing templates ranged from 24.6 to 48.8 points per model.
- ChatGPT-style entries yielded higher point estimates than raw conversation for 7 out of the 9 models tested.
The research also uncovered a stark contrast in how models handle formal versus natural language. Three models that scored 0% on formal ledger packets were able to answer the same factual questions at a rate of 45.4% to 53.4% when the information was presented as natural-language entries. This effect persists even under retrieval noise and transfers to other benchmarks like HotpotQA, suggesting that future AI evaluations must explicitly control for the reader-facing artifact.
