A consultant asks Claude about a public company's Q4 2024 earnings. The model produces a detailed response with specific revenue numbers, growth percentages, and even quotes from the earnings call. The consultant cross-checks against the actual Q4 2024 press release and finds the numbers are wrong — they appear to be from Q4 2023, presented as if they were current.
What's the most likely cause?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Training cutoffs create a specific failure mode: the model knows old data well, doesn't know new data at all, and has no native mechanism to tell you which is which. When asked about recent events, it often substitutes the most recent thing it does know — confidently, without flagging the substitution. The architectural fix is to never ask LLMs about recent specific facts unaided: provide current data via tool use (get_earnings_release(ticker, quarter)) or RAG over current sources, and have the agent surface its data sources so the user can verify currency.
Re-running won't produce randomly different wrong numbers — it'll likely produce the same wrong numbers, because the model is consistently substituting Q4 2023 data. The cause is the cutoff, not randomness.
Press releases are the authoritative source for earnings; the model is the derivative source. When they conflict on facts the cutoff predicts the model wouldn't know, the press release wins.
Recent-data use cases are exactly where RAG and tool use earn their keep. The fix exists; abandoning the use case is over-correction.