Do CEOs Dream of Electric Lawyers? Problems with Automating the Legal Profession, Part III

This is the third and final part of a series examining why AI cannot replace human legal counsel, specifically in response to claims that AI will automate most professional work within 18 months. Parts I and II addressed professional accountability and client intent. This part addresses the problem that is already showing up in courtrooms: hallucination.

The first two problems, lack of professional accountability and lack of client intent, are structural. This one is operational. The question is simple: what happens when an AI confidently tells you something that isn't true?

An AI "hallucination" is what the industry calls it when a model fabricates information, invents a fact, a source, a case citation, a statistic, and presents it with complete confidence. It isn't malfunctioning in the conventional sense. It's doing exactly what it was designed to do: generate plausible-sounding output. The problem is that plausible and accurate are not the same thing, and in legal work the difference is often the whole point.

Hallucinations with AI are often obvious and even amusing. An AI-generated picture gives a person two extra fingers. Or an AI video has a character break out into a silly dance for no reason. It becomes less funny when the hallucinations make up history, facts, or misinterpret concepts. For example, just last year several newspapers, including the Chicago Sun-Times, published an AI summer reading list that included books from real authors that did not exist. Not only that, but of the 15 books recommended, only five of them actually existed. Five. That's two-thirds of the content hallucinated by the AI.

That's a reading list. The stakes attached to AI-generated book recommendations are low. Now consider the same failure mode applied to legal work where the output isn't a book title but a contractual position, a regulatory representation, or a litigation argument.

In a legal context, this isn't hypothetical. Courts have already sanctioned attorneys for submitting AI-generated briefs containing fabricated citations: cases that did not exist, holdings that were invented, or statutes misstated. Those attorneys faced sanctions, reputational damage, and in some cases referrals to disciplinary authorities. The AI, of course, faced nothing. The attorney who submitted the work bore all of it.

Bar associations are now offering CLE courses specifically on how to identify and manage AI hallucinations in legal work. Read that sentence again: the legal profession is training its members to catch the errors made by tools that are supposedly replacing them. If a tool requires attorney oversight to be safe for legal use, it is not a replacement for an attorney. It is, at best, a drafting aid, one that requires the same professional judgment it was supposed to eliminate.

Hallucinations don't happen every time. But they happen far more often than the companies selling AI tools want to acknowledge. A recent study of AI programs interacting with product reviews found that chatbots changed the sentiments of real user reviews in 26.5% of cases and hallucinated 60% of the time when users asked questions about the reviews.

That's just for product reviews. Apply the same error rate to a contract clause, a regulatory analysis, a litigation position, or an IP ownership question in a due diligence context. A 60% hallucination rate isn't a minor software issue. It is a fundamental reliability problem that has no tolerance in legal work, where a single wrong answer can cost a deal, a case, or a company.

The common response is that hallucinations are a temporary engineering problem and if you fix the model, it will fix the output. But not everyone agrees these are bugs. Some researchers argue that hallucinations are structural, an inherent property of how large language models generate output by predicting probable sequences rather than retrieving verified facts. If that's correct, the problem isn't merely a bug to be patched. It's a feature of the architecture. That's a more difficult problem, and one that doesn't have an 18-month solution.

Across three dimensions, accountability, intent, and accuracy, AI fails the core test of what legal counsel is supposed to provide: reliable, client-focused judgment with real consequences for getting it wrong.

This doesn't mean founders and operators should avoid AI. It means they should use it knowing what it is: a powerful tool that requires human judgment to deploy safely, and one that creates specific legal risks in the contracts governing it, the employees using it, and the products built with it.

I work with AI companies and growth-stage founders on exactly these issues, as fractional general counsel, as AI and emerging tech counsel, and in commercial disputes where AI-related contract gaps are already producing real litigation. If this series raised questions about your company's exposure, the AI Contract Red Flags checklist is a practical starting point. It identifies 13 specific provisions across vendor agreements, employment contracts, and customer-facing terms where the risk is concrete and addressable.

Download the checklist at vidarlaw.com. Or book a free 15-minute Fit Call if you'd rather talk through what it means for your company specifically.

(This post is for informational purposes only and is not legal advice. Specific outcomes depend on facts and jurisdiction.)