LLMpedia – Materializing an LLM’s Encyclopedic Knowledge at Scale

Beschreibung

Large language models have become everyday knowledge interfaces for millions of users. However, their real factual reliability in open-ended writing is still not well understood. Existing benchmarks rely on small, pre-selected sets of questions, which can create an availability bias: they may overestimate factuality by only measuring what evaluators expected to ask. LLMpedia addresses this problem by letting models write full encyclopedic articles from memory only, without retrieval, and then systematically verifying the generated claims.
As a large-scale study, LLMpedia generated around 1 million articles across three model families. The results show that the benchmark picture is incomplete: for gpt-5-mini, the verifiable true rate on Wikipedia-covered subjects is 74.7%, more than 15 percentage points below the 90%+ impression suggested by benchmark-style evaluation. For frontier subjects that can only be verified through curated web evidence, the true rate drops further to 63.2%.
LLMpedia makes two main contributions. First, it offers a new way to study what LLMs know, how they express that knowledge in free text, and where their factual limits lie. Second, it provides the first fully open parametric encyclopedia framework, with all prompts, artifacts, and evaluation verdicts publicly released, enabling transparent and reproducible research at scale.

LLMpedia is available online at llmpedia.net.

Please note that the presentation will be held in English.