According to OpenAI, AI detectors are all junk

Since the beginning of 2023, a number of tools have been released that promised to be able to recognize text written by ChatGPT and other language models. Programs like GPT-Zero Schools and universities in particular should ensure that pupils and students write their homework or master’s theses themselves.

After just a few months, however, it became clear that the tools have a very low hit rate when it comes to recognizing texts written by AI. This even led to students being falsely accused of using ChatGPT to do their homework.

The tech news site Ars Technica, for example, had AI detectors like GPT-Zero a few weeks ago examined more closely and explains why they do not provide satisfactory results. Some experts sometimes refer to these detectors as “snake oil,” which points to their ineffectiveness despite high – stoked – expectations.

Now OpenAI has officially admitted this for the first time. The company, which caused real hype with the release of ChatGPT in autumn 2022, also had its own detector in its portfolio, AI Classifier.

The tool, which was admittedly in the experimental phase, was discontinued in July. The hit rate when recognizing texts written by AI was just 26 percent.

The official admission can be found in the FAQ of a back-to-school marketing post from OpenAI, like Ars Technica discovered has. The answer to the question: “Do AI detectors work?” is: “In short, no”. None of the tools published so far, including those from OpenAI, have been proven to be able to reliably distinguish between AI-generated and human-written texts.

Interestingly, OpenAI also addresses ChatGPT in the FAQ. The chatbot is also not able to recognize AI-generated texts. According to OpenAI, corresponding inquiries, such as whether users had written the essay themselves, were random and without ChatGPT discovering anything here.

The fact that AI detectors are not yet able to recognize AI-generated texts does not mean that this could be possible in the future. It could, but it doesn’t have to, as AI analyst Ethan Mollick from Wharton University thinks. AI texts are not recognizable and will probably remain so, says Mollick.