Regardless of your position on AI Content, if you are a web publisher, you are going to be dealing with it in 2023.
It’s too tempting for your freelancers, content services, or even in-house content creators not to test the water with it the next time you give them a boring SEO assignment because the keyword is showing low difficulty in Ahrefs’ keyword explorer.
But if you’re being served AI content, you probably want to know if it is AI content.
So I tested the AI Detection tool Originality.ai in many difficult situations to see if it catches content generated by GPT-3 and ChatGPT.
Can AI help you catch AI?
As you will see, the results were… surprising 🤨.
Test #1: Straight Up Copy & Paste AI Content
For the first test, I wanted to give the tool an easy task to gauge its capabilities.
So I went to Jasper.ai and straight up prompted it with:
“write a short article about 5 affiliate marketing mistakes”
Here’s the output I got. It was… generic & bland, to say the least.
And unsurprisingly, the tool caught this right away.
I repeated the experiment several times with similar prompts in different niches to ensure it wasn’t a fluke.
It wasn’t. Every single time, the tool caught it and flagged the content as AI-generated.
It’s safe to say the tool will catch the most basic AI prompts, and I think it may spell bad news for programmatic SEO disciples.
Test #2: Edited GPT-3 Content.
For the next test, I wanted to give Originality.ai something a little more challenging 😥.
I won’t share the content I used because it was taken from one of our sites that ranks quite well, and I’m not exactly interested in outing it today, sorry 😜.
But to give you some context, the content I pasted in was generated by Jasper (GPT-3) and edited by one of our in-house writers to fix its inaccuracies, improve it and make it more engaging.
So it’s part human, part AI, and I wanted to know how the tool would handle that.
And I couldn’t fault it again. The tool caught the more “human” part of the content while understanding it is mostly AI-generated.
Test #3: An Article Submitted By A Freelance Writer For Authority Hacker.
The next trial was the blog post about mattress affiliate programs written by one of our freelancers for Authority Hacker.
In principle, this was fully human-written, but I was curious to see what the tool had to say.
So I took a snippet of it and submitted it to Originality.ai.
And the tool did find the content to be original/human-written.
I must admit I had a small sigh of relief when seeing this.
I suspect the small percentage of AI content may be due to the usage of Grammarly.
Test #4: Emulating Brian Dean’s Writing
It aced all the tests so far, so I decided to make things a little more difficult.
I pasted a Backlinko article in ChatGPT and told it to write an article about another SEO tactic in the same style.
The result was surprisingly good in style, but I’m wondering if it’s not just a rewrite of something Brian Dean already talked about.
Anyway, here’s the text I submitted to Originality.ai:
And it fooled Originality.ai.
Then I wanted to know if the result was due to using ChatGPT instead of Jasper or because of the method.
So I prompted ChatGPT to “write me an article about 5 affiliate marketing tips” and submitted it to Originality.ai.
And it got caught.
Conclusion / TLDR
- Most of the time, the tool tells the right story and detects AI content.
- However, it can be fooled (see test #4)
- You probably only should question writers if several pieces have been flagged by the tool.
- Several users on Twitter are already boasting about specific settings you can use to fool the tool.
- However, let’s be honest. Most AI-generated content doesn’t have much thought behind it and will probably get caught.
- The fact that this is viable at a small scale as a SAAS tool doesn’t mean it’s viable for Google in terms of costs and value, especially considering the fair rate of false positives.
- Nonetheless, if you add this to the fact that OpenAI is working on a watermark system for AI content, this can lead us to question the long-term viability of the new “programmatic SEO” trend.
If you want to discuss this topic, join us on the Twitter thread.