Putting ChatGPT to the test
There’s a whole lot of fuss about the rise of chatbots at the moment. ChatGPT has launched, and Google’s Bard is coming soon.
What is ChatGPT?
Similar to a search engine, but the user “chats” with it, asking questions in a conversational style.
Let’s try it out
ChatGPT has released a free research preview. It is not connected to the internet so cannot retrieve real time or even recent information. It refers to an information database from September 2021.
I thought it would be interesting to ask ChatGPT questions about some of our clients. I know the answers to the questions already but does ChatGPT?
To make it easy for ChatGPT I picked questions that are answered clearly on our clients’ websites. And as this version of ChatGPT uses old information (September 2021) I chose questions whose correct answers would have been the same then as now.
Melior Clinics
Melior Clinics owns a number of high-end aesthetic treatment clinics in London, Kent, Berkshire and Cambridgeshire.
Where are Melior Clinics branches? The correct answer is London Harley Street, Richmond, Sevenoaks, Reading, Huntingdon and Peterborough.
Oh dear. Hopeless! London Harley Street and Huntingdon are on the list but the other locations are incorrect. Melior Clinics has never had clinics in those cities!
The correct information is easily found here, and would have been in September 2021 too: https://www.meliorclinics.co.uk/find-a-clinic/ so why did ChatGPT fail so miserably? Surely that should have been the first thing it looked at?
Let’s try something else. What are the names of the practitioners at Melior Clinics?
Oh my goodness! Another terrible answer. Dr Timothy Beazleigh is indeed the clinical director at Melior Clinics but the others do not work there, nor have they ever done and were not on the website.
This information is easily found here: https://www.meliorclinics.co.uk/about-melior/practitioners/.
Let’s ask one more question to see if it can redeem itself. How much does botox cost at Melior Clinics?
Wrong again! It’s £199 not £195. Again, this is clearly shown on the Melior Clinics website and was £199 in 2021 too (https://web.archive.org/web/20210725122344/https://www.meliorclinics.co.uk/price-list/).
The extra information about other areas costing more and them offering a free consultation are correct and useful.
The Orchard
The Orchard is a beautiful wedding venue near Maidstone, Kent. We recently made them a new website (https://www.orchardsuite.co.uk) and run their Google Ads campaign.
What are the wedding packages available at The Orchard, Maidstone?
This information was correct in September 2021 – so it looks like we have a correct result! (The wedding packages have since changed by the way).
What is their phone number? Note that due to the way you can “converse” with ChatGPT I don’t have to specify “What is The Orchard, Maidstone’s phone number?” It already knows we’re chatting about The Orchard and sees this as a related, follow up question. Well, it should do, but …
That’s not their phone number it’s that of a competitor! It’s The Orangery’s phone number. Of all the wrong numbers to give out, it has chosen the worst possible – a competitor’s!
Dwight Turner Counselling
Dr Dwight Turner is a psychotherapist, author and course leader at the University of Brighton. We made him a new website (https://www.dwightturnercounselling.co.uk) a couple of years ago.
So – let’s see if ChatGPT knows Dwight. Who is Dr Dwight Turner?
ChatGPT only has access to information about “public figures”. Does that mean celebrities then? Dwight is an author and renowned psychotherapist. A Google Search for “Dr Dwight Turner” returns a results page dominated by him and no one else.
OK, let’s help ChatGPT out, let’s mention his most recent book.
Almost! One bad error. He lectures at the University of Brighton, not Edinburgh! According to Dwight’s LinkedIn profile he has never worked at the University of Edinburgh.
Open Door Digital
What’s the best digital marketing company in Crowborough? The correct answer is Open Door Digital of course but what does ChatGPT think?
OK – so ChatGPT won’t have opinions on who is “best” but it’s kindly listed some companies to try. But! None of them are in Crowborough.
- Redboat – Tunbridge Wells – close, but not Crowborough.
- Three Little Birds – cannot find such a company, as far as I’m concerned this is a song by Bob Marley & The Wailers.
- 3Sixty Digital – Bristol.
- Green Gecko Digital – Leeds.
Conclusion
Even allowing for ChatGPT’s preview data being 18 months old, its answers have been hopeless in this test.
Just as I finished this post, I received an invite from Google to test its Bard chatbot. It gave much better replies to the same questions – I shall post a full comparison with ChatGPT as soon as possible.