ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models

In this work, we propose \textbf{ForgetBench}, a benchmark for measuring forgetting dynamics in the long-term parametric memory of LLMs under continual knowledge editing. It comes from a paper posted to arXiv on 15 October 2026. Large language models (LLMs) have demonstrated strong capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquired knowledge under repeated updates remains insufficiently understood. Existing evaluation paradigms primarily focus on single-step reasoning or static knowledge editing, which fail to capture the temporal dynamics of knowledge retention and degradation during continual model modification. ForgetBench introduces two complementary evaluation paradigms, namely concept-based QA and scenario-based QA, to probe the retention of isolated facts and structured relational knowledge. Building upon a sequential editing framework, we construct temporally ordered knowledge streams and evaluate model behavior across multiple editing stages. To quantify these dynamics, we introduce a unified evaluation framework that indexes memory age by the number of subsequent edits and measures temporal decay, retention strength, and cross-instance stability. Extensive experiments across diverse models and editing methods reveal distinct forgetting trajectories and a recurring tension between long-term retention and generalization quality. Our findings highlight the need for more robust memory mechanisms that can effectively acquire, update, and preserve knowledge over time in future LLMs.

Why it counts

In this work, we propose \textbf{ForgetBench}, a benchmark for measuring forgetting dynamics in the long-term parametric memory of LLMs under continual knowledge editing.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.