Artificial intelligence continues to shape the way professionals work, but a recent case has highlighted the potential pitfalls of using AI in high-stakes environments like the courtroom. Jeff Hancock, a Stanford University expert on misinformation, is at the center of controversy after admitting to using an AI tool, ChatGPT-4o, to draft a court document that included fabricated citations.

The Case in Question

The controversy arose during a case challenging a Minnesota law that criminalizes the use of AI to mislead voters before elections. Hancock, who charged the state $600 per hour for his services, submitted a declaration that included fake citations generated by ChatGPT-4o. The fabricated references were identified by opposing lawyers, who subsequently petitioned the court to dismiss Hancock’s statement.

Hancock’s Defense

Hancock explained that the errors were unintentional, referring to the fabricated citations as “AI-hallucinated.” He stated that ChatGPT-4o mistakenly interpreted his notes as commands to insert citations, leading to the inclusion of non-existent references. In a separate filing, Hancock defended the use of generative AI tools in academic and professional settings, arguing that such tools are increasingly integrated into platforms like Microsoft Word and Gmail.

Despite his defense, the incident underscores a growing need for ethical guidelines and accountability in the use of AI.

The Legal Implications

The Attorney General’s Office filed a motion allowing Hancock to submit a revised declaration, but the incident has already sparked debates about the ethics of using AI in legal and expert submissions. This case follows a New York court ruling earlier this year, requiring professionals to disclose the use of AI in expert opinions after rejecting a submission containing AI-generated content.

AI in Research and Beyond

Hancock, a published expert on misinformation and technology, argued that generative AI tools like ChatGPT are invaluable for tasks such as literature surveys and document drafting. However, the incident calls into question whether sufficient safeguards exist to prevent unintentional misinformation.