U.S. Attorney Sanctioned After Submitting ChatGPT-Generated Fake Case Law

 Judge Warns: “ChatGPT Can Write a Story—But It Can’t Practice Law”

In a cautionary tale about artificial intelligence and its growing role in the legal system, a Texas-based attorney has found himself in hot water after relying on ChatGPT to draft a legal brief—only to discover that the AI-generated citations were completely fictitious.

The attorney, Robert Lee, was representing a creditor in a routine bankruptcy matter in the Southern District of Texas when he filed a motion referencing four legal decisions. The only problem? None of those cases existed. All four had been fabricated by ChatGPT, OpenAI’s popular language model.

The filing immediately raised red flags for Judge Kenneth M. Hoyt, a veteran of the federal bench who quickly discovered that the cited cases could not be located in any database. Upon further investigation, it became evident that the attorney had used ChatGPT to assist in preparing the brief without verifying the authenticity of the citations.

A Mistake That Carried Consequences

Judge Hoyt issued a monetary sanction against Lee and used the ruling to send a broader message to the legal community.

“ChatGPT can write a story,” Hoyt remarked from the bench. “But it doesn’t practice law, and it certainly doesn’t know the law.”

Hoyt called the inclusion of fake citations “an affront to the legal profession,” emphasizing that attorneys have a responsibility to ensure the accuracy of any submission they make to the court.

While the fine imposed was relatively modest, the public rebuke serves as a sharp warning to other legal professionals who may be tempted to cut corners by outsourcing research to artificial intelligence tools.

Not an Isolated Incident

This isn’t the first time an American court has been forced to address the pitfalls of AI-generated legal work. Earlier this year, two attorneys in New York were sanctioned for submitting a legal brief peppered with nonexistent case law also generated by ChatGPT. In that instance, the judge went so far as to say the AI had created “an illusion of legal research.”

The recent Texas case only underscores a growing concern among judges, ethics boards, and legal scholars: while AI may be an incredibly powerful tool, its use in legal practice demands rigorous human oversight.

 ⚠️ The Risk of Over-Reliance on AI

Legal technology has advanced rapidly over the past decade. Tools powered by artificial intelligence—including natural language models like ChatGPT and Claude—have been embraced across the industry for tasks ranging from contract drafting to document review and even trial preparation.

But as many experts are now warning, these tools come with real risks.

“AI models like ChatGPT are designed to predict what words should come next in a sentence—not to tell the truth,” said Dana Franks, a legal tech analyst based in Washington, D.C. “They’re language generators, not fact-checkers. That’s why it’s so dangerous when legal professionals use them without independent verification.”

Franks notes that AI can be useful in early research stages or for generating ideas, but not for producing final work product.

“The tool doesn’t replace legal reasoning or fact-checking,” she emphasized. “That responsibility lies squarely with the attorney.”

Courts Taking Proactive Measures

In response to the increasing misuse of AI in court filings, some judges have begun requiring attorneys to disclose whether they’ve used AI tools in the preparation of any legal documents.

For example, a judge in Delaware now mandates that lawyers affirm, in writing, that any use of AI has been fact-checked and verified by a human. Other courts, including in California and Florida, are reportedly considering similar policies.

Bar associations are also stepping in. The American Bar Association (ABA) has launched a task force to examine the ethical implications of AI in legal practice, including how attorneys can responsibly integrate emerging technologies while still upholding professional standards.

The Bigger Picture: Ethics and Accountability

The broader concern isn’t just about fake case law. At stake is the reputation and reliability of the legal system itself. The public relies on attorneys to uphold accuracy, diligence, and ethical conduct. Introducing unvetted AI content into legal filings could jeopardize those values.

“Even one false citation can damage a case, waste judicial resources, and erode trust,” said Emily Carroll, a law professor at Stanford University. “And the worst part is that it’s preventable.”

Carroll argues that while legal education should embrace AI literacy, it must also instill skepticism. “Teaching students how to use these tools is important—but teaching them how to verify what those tools produce is absolutely essential.”

A Tool, Not a Replacement

The incident in Texas is a textbook example of what happens when legal professionals treat AI as a shortcut rather than a support tool. ChatGPT can help brainstorm arguments, summarize complex ideas, and even suggest legal strategies. But when it comes to producing accurate, court-ready filings, nothing replaces the judgment and responsibility of a licensed attorney.

AI doesn’t understand law—it mimics it. And while it can string together convincing-sounding sentences, that doesn’t make those sentences legally valid.

As courts become more vigilant and technology continues to evolve, attorneys will need to strike a careful balance: embracing innovation while upholding the timeless values of their profession.

Final Takeaway

AI can be an ally in the practice of law, but it’s not a substitute for human judgment, research, or ethical integrity. As the Texas courtroom drama shows, relying too heavily on ChatGPT—or any AI tool—without doing the necessary legwork can lead not only to professional embarrassment but also to disciplinary action.

In the end, technology is only as responsible as the people who use it. And in the courtroom, that responsibility still rests with the lawyer—not the machine.