I've been a fan of PerplexityAI as an "answer engine", impressed by its custom summaries and list of cited sources. It has been so good, it's reduced my use of Google (search engine) quite substantially,
However, an experience yesterday has caused me to question just how authentic these lists are, due to a set of irrelevant sources that could only be defined as "citation decoration".
I'll reproduce my two searches, prompts and all, and you can judge for yourself.
The key lesson here is that just because it looks like a citation list, it doesn't mean it is one, so please always "read the label", especially if you are going to use results to help you in life or business.
So, come on this recreation of my journey into the depths of AI research, which happened when I conducted an experiment yesterday, to see if PerplexityAI could generate a bio for one of my alter egos, Professor Sebastian Longsword, of the MBA School Of MBA Credentials.
If you want to see the full search and follow the links: Professor Longsword Bio PerplexityAI
The Experiment
For those unfamiliar, Professor Sebastian Longsword is an eccentric MBA professor, a character I often portray in my stand-up comedy routines.
Seeking a fresh bio for him for his introduction to last night's gathering of the Australian Marketing Institute SA, I turned to PerplexityAI.
Initially, the AI seemed to do a commendable job, crafting a narrative that captured the essence of Longsword’s quirky personality. Here's the prompt I quickly cobbled together:
Can you craft a short bio about Professor Sebastian Longsword from Adelaide, relating to his pioneering work with short form MBA degrees?
Here is the first version of the bio generated by PerplexityAI:
Professor Sebastian Longsword is a renowned academic from Adelaide, known for his innovative approach to MBA education. He pioneered the concept of short-form MBA degrees, recognising the need for condensed yet comprehensive programs catering to busy professionals. Longsword's groundbreaking curriculum emphasises practical skills and real-world applications over traditional theoretical frameworks, equipping students with the tools to navigate the ever-evolving business landscape efficiently. His unique teaching methods, often infused with wit and eccentricity, have garnered widespread acclaim for making complex concepts accessible and engaging. Longsword's contributions have revolutionised MBA education, making it more accessible and relevant to the modern workforce.
At first glance, that looks passable, however, upon closer inspection, I found that not a single source it cited was pertinent. It was as if the AI was merely decorating the bio with citations to give an illusion of credibility.
- Robtrt Johnson - Orange County California Genealogical Society
- wiki-pages-nodenames.txt
- Full text of Medieval And Early Modern Fight Books
- https://osf.io/mepkc?action=download
- Diff chromium/deps/hunspell_dictionaries
As you can see, there's nothing that fed into that summary.
The Second Search - This Time With Pro Setting Enabled
Determined to get more accurate results, I provided PerplexityAI with a few hints and upgraded to the pro version.
The outcome improved, but only slightly—about 30% of the cited sources were now relevant. This improvement, while notable, still left a significant proportion of citations that were irrelevant or misleading.
Here's my second prompt:
That is a good start, please also include reference to his pride about the MBA School Of MBA Credentials wine list and his unique series of Wine and Textbook Pairing electives.
Here is what resulted:
Professor Sebastian Longsword is a renowned academic from Adelaide, celebrated for his pioneering work in short-form MBA degrees. He is the mastermind behind the MBA School Of MBA Credentials, where he has revolutionised business education by offering condensed yet comprehensive programs tailored for busy professionals. Longsword's innovative curriculum emphasises practical skills and real-world applications, making complex concepts accessible and engaging. In addition to his academic achievements, Professor Longsword takes great pride in the MBA School Of MBA Credentials' wine list, which he personally curates. This unique feature reflects his belief in the importance of a well-rounded educational experience. Furthermore, he has introduced a distinctive series of Wine and Textbook Pairing electives, blending the art of wine appreciation with rigorous academic study. These electives have become a hallmark of his teaching philosophy, offering students a delightful and enriching way to enhance their learning journey. Longsword's eccentric and witty teaching methods, combined with his dedication to making education both enjoyable and practical, have earned him widespread acclaim. His contributions have significantly impacted MBA education, making it more accessible and relevant to the modern workforce.
The Implications
This experience serves as a stark reminder of the importance of verifying sources, especially when using AI-generated content.
In the digital age, where information is abundant and easily accessible, the temptation to take citations at face value is strong.
However, as my experiment with PerplexityAI demonstrates, this can lead to the dissemination of inaccurate or irrelevant information.
What Would The Professor Do?
The simple message from me and the Professor is: Always check your sources.
Do not merely glance at them to see if they "look about right."
Dive deeper to ensure their relevance and accuracy.
This practice is crucial not just for maintaining the integrity of your work but also for fostering a more informed and critical audience.
In conclusion, while AI tools like PerplexityAI can be incredibly useful, they are not infallible.
It is up to us, as users, to scrutinise the information they provide and ensure its validity. So, the next time you generate content with AI, remember the lesson of Professor Sebastian Longsword and beware of Citation Decoration.
Image: SA branch of the Australian Marketing Institute.