share the AI response that was so wrong it almost looped back to impressive. I’ll start. asked for a brief biography of a colleague before a meeting. got a confident, detailed biography. of a completely different person with the same name who died in 1987.
asked it to find three peer-reviewed sources on a topic. got three sources. real journal names. plausible DOIs. author names that looked right. none of them existed. when I asked where the citations came from it said ‘I may have generated these based on typical publication patterns in the field.’ you may have.
asked it to summarize a specific paper by title and author. got a detailed, structured summary with methodology, findings, limitations. the paper I named was a working paper that had never been published. the AI had written an entire research paper and summarized it.
asked it whether a specific law existed in a specific country. it said yes, gave me the year of passage, the key provisions, and a brief legislative history. the law did not exist. the country had debated a similar proposal and rejected it. when I pushed back it apologized and then told me the same false information again with slightly different phrasing.
asked it to help me respond to an email. it helped me respond to what it thought the email said. I had pasted the email. the model had either ignored it or substituted a different email it thought was more likely based on my description. the response was perfect for an email I had not received.
the pattern I notice across all of these: the confidence is perfectly calibrated to the wrongness. the more fabricated the information, the more certain the delivery. there’s no hesitation signal, no ‘I’m less sure about this part.’ it presents invented biographies, nonexistent papers, and false legislation with the same tone as correct information.
the useful takeaway: the model’s confidence is not a signal of accuracy. it’s a feature of how the model generates text. confident presentation is a stylistic property, not an epistemic one.