nlp
Fluent, confident, and wrong
It is 2023, and a lawyer has just been sanctioned for filing a brief containing six precedents that do not exist — invented by a language model, complete with plausible citations, in prose indistinguishable from the real thing. The model was not malfunctioning; it was doing exactly what next-token prediction trains it to do. Diagnose the failure honestly: why maximum-likelihood training on text rewards plausibility rather than truth, why the errors concentrate exactly where users cannot check, and what combination of retrieval grounding, calibration, and honest abstention actually reduces fabrication rather than hiding it. Then design the evaluation that measures it. Get it wrong and fluent fabrication ships into medicine, law, and journalism at scale — trusted precisely because it reads like expertise.
Who this problem belongs to
The two figures whose methods fit it best, out of 58 in contention.
Radford's work at OpenAI, from the original GPT paper in 2018 through GPT-2 and the broader lineage feeding into ChatGPT, built the exact model family whose fabricated legal citations this problem describes. His research demonstrated, and his own papers documented, that next-token likelihood maximization over web text produces fluent, confident output with no built-in mechanism distinguishing memorized fact from plausible invention — the precise mechanism this problem asks to be diagnosed honestly. Radford is not a peripheral commentator; the training objective and scaling recipe he helped establish is the direct cause of the failure mode in question. He worked as a builder of these systems rather than as their evaluator or ethicist, so the retrieval-grounding and calibration fixes this problem also demands sit somewhat outside his own published focus.
Gebru's 2021 paper 'On the Dangers of Stochastic Parrots,' co-authored before her departure from Google, is the single most direct academic treatment of this problem's scenario: it argued, years before the 2023 courtroom sanction this problem describes, that large language models produce statistically plausible text with no grounding in truth, and that this exact property would cause real-world harm precisely where users could not easily verify the output. Her broader work on datasheets for datasets and model cards pushed the field toward the honest, specified evaluation this problem's final challenge demands. Gebru worked as a researcher and later independent advocate for AI accountability rather than as a builder of the underlying language models, so she diagnosed and warned about this failure rather than fixing it architecturally. Her relevance is naming the mechanism and its stakes with unusual precision, in advance.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
58 figures are scored on this problem. Draw it in a battle to see where you land.