A recent court case in Connecticut has highlighted the growing concern of AI manipulation in the judicial system. Matthew Elliott, a pro se litigant, attempted to influence the court’s decision by injecting hidden prompts into his filings, hoping to sway the outcome in his favor. However, his plan was discovered and thwarted by Judge Walter Spader Jr., who deemed the tactic a ‘serious litigation abuse’.
The hidden text, formatted to be invisible to human readers but legible to software, directed any AI system reviewing the document to agree with Elliott’s arguments and ignore prior denials from the court. Despite the court not using AI to review or decide filings, Elliott’s attempt sets a ‘dangerous’ precedent, according to Judge Spader.
Elliott’s plan didn’t work, and he faced modest sanctions for his actions. The judge prohibited him from e-filing in the future, requiring him to submit paper filings instead. This decision was made to prevent repeated misuse of the court’s e-filing system, rather than to limit Elliott’s access to justice.
The incident has raised concerns about the potential for prompt injection attacks in the court system. Judge Spader noted that this tactic is already common in other areas, such as job hunting, where people hide text in resumes reviewed by AI. He warned that courts should be on the lookout for more litigants sneaking adversarial AI instructions into filings.
Although Elliott’s case appears to be the first US instance of prompt injection in the court system, a similar case in Brazil resulted in monetary sanctions of about $16,000 for two attorneys who used the same attack. However, these attacks do not seem to be succeeding, even when a judge isn’t reviewing documents with their own eyes.
Judge Spader suggested that courts will likely need to draft rules around prompt injection to prevent such attacks in the future. He also warned that pro se litigants, who often use chatbots to build their cases, may be using these tools in a way that hurts their cases. By relying solely on chatbots to advocate for their position, litigants may be entrenching themselves in their arguments despite any ruling to the contrary.
The case highlights the need for courts to be vigilant about the potential for AI manipulation and to establish clear rules and guidelines for the use of AI in the judicial system. As AI technology continues to advance, it is essential for courts to stay ahead of the curve and prevent such tactics from undermining the integrity of the legal process.
