How We Determine Human Work
It has been well established that using text analysis alone is unreliable to test if work is done by a human or by AI. In fact, no method is completely perfect.
The OurWork app determines if a human wrote something by looking at how text was created. That is, we analyze typing patterns, document history, and edits made to the text. This is based on the principle that Writing is Thinking.
AI, or more precisely generative large language models (LLMs), guess the next token in a text (a letter or a word) based on the text immediately before it. In other words, LLMs generate text in one go from the beginning to the end. Humans don't work that way.
A person makes mistakes, changes their mind, and thinks in non-linear ways. We move text around, rewrite in other words, edit for clarity and so on. We look at many more indicators of human behavior, but the main idea is the same: the way humans write text mirrors the way humans think.
Some may argue that LLMs have those capabilities as well, but that is merely an illusion. In reality LLMs don't think at all. Their process is fundamentally different to human thinking which makes it hard to make and LLM think and act like a human.
How Reliable is This?
If people don't actively try to circumvent detection, this method nearly never classifies text as human when it is not. The most common way to get AI text into the OurWork app is to paste text in from an external source and this external work is never flagged as human.
That leaves two problems:
- The true origin of externally pasted text cannot be determined and therefore external text cannot be classified. Some human-made material cannot be certified.
- People can attempt to create tools that mimic human editing.
We'll leave an in-depth discussion of the first problem for later. The second issue is something we continuously work on. But the important part it that it takes effort to build, tune, and use circumvention tools.
The effort required to actively circumvent detection is a sufficient deterrent fort AI slop to proliferate in massive quantities. The OurWork platform ensures that the vast majority of content is truly made with love by humans. That means that you as a reader and a writer have a better experience and that's what matters.