For the first time, an AI has passed the classic Turing test. A model was even more often mistaken for a human than real people. Success for AI, but what does that mean for our use of technology?
Can a machine think like a human? The British mathematician Alan Turing wanted to answer this question back in 1950 with his “imitation game”. This experiment later went down in history as the Turing test.
The idea: If a person can no longer reliably tell during a conversation whether they are communicating with a machine or another person, the machine has passed the test. Now that’s exactly what happened.
What is the Turing Test?
Classically, the Turing test works in such a way that a test subject communicates with two conversation partners at the same time. One of them is a human, the other a machine. Both try to pass as human. In the end, the examiner has to decide who is who. If it cannot do this reliably, the machine has passed the Turing test.
Cognitive scientists Cameron Jones and Benjamin Bergen from the University of California San Diego used exactly this principle in a new study. Almost 500 participants each had five-minute chats, and in an additional study also 15 minutes, and then had to judge.
The result: GPT-4.5 from OpenAI was mistaken for humans 73 percent of the time, i.e. more often than real human conversation partners. Meta’s model LLaMa-3.1-405B was also statistically indistinguishable from humans at 56 percent.
The chatbot Eliza from the 1960s also took part in the study. However, Eliza was exposed in most cases because the rule-based program cannot react flexibly to conversations.
“We found that when given the right prompts, advanced LLMs can demonstrate the same tone of voice, directness, humor and fallibility as humans,” explains study author Cameron Jones in a press release.
While we know that LLMs can easily produce knowledge on almost any topic, this test showed that they can also convincingly represent social behaviors, which has significant implications for how we think about AI.
AI needs role instruction to pass test
The success of AI in the Turing Test initially seems as if the technology is superior to humans. According to the researchers, one detail is still crucial: the models only passed the test if they first received a so-called persona instruction.
The brief was to play a young, introverted person who knows internet culture and uses slang. Without this instruction, GPT-4.5 dropped to a success rate of 36 percent, LLaMa to 38 percent.
“They have the ability to appear human-like, but perhaps not so much the ability to figure out what it would take to appear human-like,” explains co-author Benjamin Bergen.
This raises a fundamental question: What good is a test whose result can be changed so massively by the right instructions? If people first have to tell the machine how to act human, did the machine really pass the test or rather the people who wrote the prompt?
AI Turing Test: Why passing doesn’t mean intelligence
The researchers also question the test: The Turing test was originally developed to investigate whether machines can keep up with human intelligence, explains Bergen. “But today we know that AI can answer many questions faster and more accurately than humans.”
The real problem is not just brain power. Seeing that machines can pass the test, and seeing how they pass it, forces us to reconsider what it actually measures. Increasingly, he measures human resemblance.
What is interesting is that the test subjects hardly asked any classic intelligence questions. Instead, they paid attention to language style and personality. Anyone who answered too perfectly looked suspicious. The test subjects did not examine knowledge or logic, but rather looked for errors.
In the end, that probably says more about us than about the machines. Apparently we try to recognize humanity not by cleverness, but by imperfection. However, our recognition features appear to be so predictable that an AI can be specifically trained to recognize them.
AI passes the Turing test: Importance for our online communication
The researchers clearly warn of the consequences. “We need to be more vigilant; when interacting with strangers online, people should be much less confident that they know whether they are talking to a human or an LLM,” says Jones.
Because: Even if you are attentive and can compare directly, you may no longer be able to tell the difference. Bergen also refers to actors who want to use bots specifically to get people to provide sensitive data, make voting decisions or make purchases.
The researchers say they hope to use this work to increase public understanding of what these systems can now achieve and what kind of protective measures society may need.
Also interesting:
