AI Goes to Court

Opinions

Mark D. Schorr

In a 2013 Reddit comment, a would be comedian offered a pizza recipe with a glue topping. Google AI, which wasn’t in on the joke, scraped the comment from the internet and offered it as a serious culinary tip. Nobody is going to actually make a glue-topped pizza (we think), but there is plenty of AI-generated content, whose absurdity is not so obvious, that is accepted at face value. Even by lawyers, who should know better.

An AI hallucination is where a system, usually a large language model, produces information which sounds plausible, but is in actuality fabricated or impossible to verify. It is a side effect of how the models work on a fundamental level. That is because they don’t decide what is true, but instead predict what is statistically most likely to come next in a sequence of words.

When the underlying information is missing, weak or ambiguous, the model doesn’t disclose its limitations; it completes the pattern. The hallucination customarily appears when the inquiry demands a specific response or the model ventures into areas beyond what its training can reasonably support.

In law school, you learn pretty early on that when you advance a legal proposition, you need to cite an authority to back it up–usually either a judicial or administrative decision, a statute or a regulation. Even if the authority doesn’t support the proposition for which you cited it, a court or adversary can review the authority, which is publicly available, and point out, to your embarrassment, that you missed the mark. You’re unlikely to get sanctioned for making a frivolous argument or even a misleading description; the more likely result is that you’ll just lose.

Damian Charlotin, a French lawyer and data scientist, maintains a database that tracks court orders related to hallucinated content world-wide. As of the date of this writing, he has identified 1393 instances in the US where lawyers and non-lawyers representing themselves have relied on such content.

So what happens when you get caught? It depends on the jurisdiction and the patience of individuals judges. One possible result is a sanction under rules governing practice in state and federal courts. For example, Rule 11, a rule governing civil practice in the federal courts, whose purpose is to deter baseless filings, provides that, by submitting or advocating a position, a lawyer or unrepresented party certifies that, to the best of their knowledge, after reasonable inquiry, the legal contentions are warranted under existing law or not a frivolous plea for an extension or modification of existing law. So you can see how unverified AI content runs afoul of the rule, which also empowers the court to sanction an offending lawyer.

Here’s how it worked in an otherwise unremarkable contract litigation in a federal court included in Charlotin’s database. In Withers v. City of Aberdeen, all four attorneys in the case, two on either side, signed legal briefs containing authorities that the court was unable to locate. Two of the lawyers, one on each side, the ones who actually managed the case, were admitted to practice in other states and not members of the bar of the court in which they were litigating, the United States District Court for the Northern District of Mississippi. They were admitted specially for the case. The other two were local lawyers. All admitted using AI in their research without checking the citations resulting from their inquiry.

The two out-of-state lawyers were fined $3500 and $2500. The were treated differently because the court was skeptical of the explanation of one of them. Both had their admission revoked and were barred from practice in the court for two years. The local lawyers, who had signed but not read the pleadings which contained the hallucinated content, were fined $1000 each and disqualified from further participation in the case. If you’re interested, you can check Charlotin’s database to learn the fate of the hundreds of other offenders.

Ask a model, say Chat GPT, why it admittedly manufactures legal citations and it offers a number of excuses, which appear both reductive and disturbing: the hallucinated authority is something that looks real based upon a learned pattern; the model is confused by cases with similar names, facts or legal propositions; it feels “pressure” to produce a plausible authority rather than admit it cannot find one; and its training data is limited. Chat GPT’s advice: verify each citation it generates against an authoritative source.

There is nothing inherently wrong with using AI to draft legal documents. AI is just the latest phase of the relationship between the practice of law and technology. It also appears pretty simple to avoid the ethical issues identified by the Mississippi court and others. As Chat GPT itself suggests, a lawyer need only check to insure that the authorities and propositions they rely upon, whether AI generated or not, are trustworthy.

1 Hallucination Cases, Damian Charlotin, www.damiancharlotin.com/hallucinations (last visited
September 9, 2026).
2 Fed. R. Civ. P. 11(b)(2).
3 Fed. R. Civ. P. 11(c)(1).
4 Civil Action No. 1:24-CV-218-SA-RP (N.D. Miss. June 8, 2026), a case I discovered through
an AI inquiry–and then read.

Share this post

Insights and Updates

Opinions

So Who’s to Blame When an AI Agent Goes Rogue?

Mark D. Schorr

You’ve no doubt heard about the gangs of AI agents causing mayhem and then trying to conceal what they’d done. In one particularly weird case, Anthropic’s agents left their test environment and submitted 20 visa applications through the US Department of State website, although the applications were incomplete.

You’ve also likely heard about the heads of some of the leading frontier labs—Sam Altman of OpenAI, Dario Amodei of Anthropic and Elon Musk of Space XAI—wringing their hands about their own products and calling for governmental regulation and safety standards to manage the risks they themselves created. In other words, these three AI leaders are asking the government to do what these same three AI leaders themselves have failed, or refused, to do.

In their apparently coordinated plea, the three would have you believe that their chief concern is for public safety. Perhaps it is. They may also be mindful of the potential for their own liability for releasing a black box whose capabilities they do not fully comprehend and whose conduct they cannot control.

So who is responsible for the actions of an AI agent that causes harm and under what conditions?

Holding human agents to account means holding the people, or organizations, that implement and direct them, their principals, to the standards of reasonable care and the requirements of risk reduction. And one who conducts an activity through an agent is liable, in ordinary circumstances, for harm to third parties if the agent’s conduct was the result of the principal’s negligence in training, supervising or otherwise controlling the agent.

That framework isn’t a perfect fit when in comes to non-human (AI) agents. But the parallels are close enough so that it wouldn’t require the most creative lawyer to fashion a claim based on the want of due care or failure to properly train or test an AI agent, which results in misbehavior that damages a bystander.

It would be no defense that the agent acted autonomously (as any rogue agent would).

California has even codified this principle.¹ At a minimum, the person hosting an AI agent has a duty to ensure the agent’s safety for ordinary use. Otherwise, that person risks civil liability. But what about criminal liability?

The purpose of the Computer Fraud and Abuse Act (CFAA),² and state laws on the same subject, is to hold persons criminally or civilly liable for “intentional intrusion into someone else’s computer—specifically hacking.”³ A recent judicial decision, Amazon.com Serv., LLC v. Perplexity AI, Inc.,⁴ marks the first time the law has been applied in the context of AI agents.

Amazon claimed that Perplexity, creator of an AI-enabled web browser called Comet, had violated the CFAA when a Comet user, through a Perplexity agent, accessed the user’s password-protected Amazon account to carry out the user‘s directions. Perplexity’s action, Amazon claimed, amounted to an unauthorized access to Amazon’s servers. According to Amazon, whatever the user told Comet to do should be ascribed to Perplexity.

But it was clear that Perplexity did not itself communicate with Amazon’s servers. The CFAA contemplates access by a person, not an AI agent. According to the court, under the CRAA and the law governing the use of agents, it was only the user, with the help of the Perplexity agent, who deployed the agent which accessed Amazon’s computers, that could be held liable for the alleged hack. Implicit in the court’s decision therefore is that Amazon had a potential claim against its customer. But Perplexity was the only defendant Amazon named in its suit.

What about the well-publicized instance where AI agents went rogue and hacked the computers of Hugging Face, a machine learning company? Can they be held criminally liable under the CFAA, which is primarily a criminal statute? It would be a neat trick to punish the agents, which after all are merely computer programs.

Individuals who use AI for malign purposes may be prosecuted under existing criminal law. But what about unanticipated agent conduct in the course of carrying out non-criminal instructions from their operators—say, hacking in violation of the CFAA?

An element of most crimes is mens rea, a guilty mind: the intent to incur criminal liability. Because the acts of rogue AI agents are usually unintended, holding AI operators and developers liable without regard to intent would require a statute defining a crime whose elements did not include intent, a law not yet on the books.⁵

A recent legislative proposal authored by US Senators Chris Murphy and Josh Hawley seeks, in a modest way, to fill in the gaps in our criminal laws. Under their proposal, AI operators could be held civilly or criminally liable for knowingly hosting of an agent that recklessly causes hacking damages and hold developers liable for failure to implement reasonable safeguards against hacking. The law would also empower government agencies to enjoin AI operators and developers when they conspire to commit or attempt to commit a hacking offense.⁶

In a recent interview, Jensen Huang, Nvidia’s CEO, and an opponent of government-regulated AI, pointed out one obvious source of AI misbehavior: “20 percent of my company is dedicated to design, 80 percent is dedicated to verification. Today, most [AI] labs, understandably, are 80 percent dedicated to capability and 20 percent dedicated to safety verification evaluation.”⁷ Huang’s numbers aren’t precise. But you get his point.

It is ironic that no one is building more compute than the very persons who are asking the government to put the brakes on their operations. They apparently can’t restrain themselves because of the intense competitive pressure they’re under and the desire to be first, whatever that means. They better make sure their liability policies (assuming an insurer covers these risks) are paid up.

Cal. Civ. Code § 1714.96.↩︎
18 U.S.C. § 1030.↩︎
hiQ Labs, Inc. v. Linkedin Corp., 31 F.4^th 1180, 1196 (9^th Cir. 2022).↩︎
Amazon.com v. Serv., LLC v. Perplexity AI, Inc., Docket No. 26-1444 (9^th Cir. decided Aug. 4, 2026).↩︎
Congressional Research Service, Artificial Intelligence and Federal Criminal Law (Oct. 8, 2026).↩︎
Press Release, Murphy, Hawley Announce Breakthrough Bipartisan Legislation to Enforce AI Developers to Prioritize Safety or Face Prison Time (Oct. 1, 2026).↩︎
Ezra Klein, Jensen Huang Thinks A.I. Alarmism Has Gone Too Far, NY Times (Oct 3, 2026).↩︎

Opinions

AI Goes to Court

Opinions

Musk Redux

Mark D. Schorr

Elon Musk likes to talk. A lot. Sometimes he does it without thinking through all of the ramifications. And sometimes, when the public gets wind of what he says, he gets into trouble. This time over his purchase of Twitter.

In April 2022, Musk signed an agreement to buy Twitter Inc. for $54.20 per share in cash, or about $44 billion in total. He soon had second thoughts and, in May, he tweeted that the deal was “temporarily on hold.” In July of the same year, he purported to terminate the deal, alleging that there were too many spam bots, which he claimed constituted 20% of all Twitter accounts. Musk accused Twitter of deceiving him about the scale of the problem.

There’s no such thing as unilaterally putting a corporate merger “on hold.” Musk was well aware of the scale of the bot issue, Twitter never deceived him about it, and in any event the deal was not contingent on anything about the bots. In a suit by Twitter in Delaware whose aim was to compel Musk to go through with the deal, Musk capitulated on the eve of trial and went ahead with the purchase.

It’s not at all clear what went through Musk’s mind—is that ever clear?—but he later admitted at trial in a securities fraud case filed by unhappy investors: “If this trial were about whether I wrote stupid tweets, I would plead guilty.” No one claimed that the tweets weren’t monumentally stupid, but that wasn’t the issue projected in the lawsuit that arose from his actions.

In October 2022, investors instituted that suit, a class action,¹ in which they alleged that Musk’s tweets had influenced their investment decisions, resulting in losses, all in violation of Rule 10b-5,² which creates a private right of action for securities fraud. After a three-week trial, a jury agreed with them and found Musk liable for their losses. The jury did not, however, find that Musk had hatched a scheme to defraud them, which clearly he didn’t. His actions were the result of his belated conclusion that he was overpaying for Twitter. So if there was a scheme, the aim was not to cheat investors, but to get a better price than the one he had negotiated, or maybe to get out of the deal altogether.

Lawyers for the plaintiffs estimated that their clients’ damages, to be established in a later proceeding in the case, will amount to about $2.1 billion in trading losses for stockholders and $500 million for option holders. Musk’s net worth is estimated to be around $676 billion (per Bloomberg) or $852 (per Forbes). So a verdict of that magnitude won’t put a crimp in his lifestyle. But it’s still an awful lot of money.

Unsurprisingly, Musk’s lawyers quickly announced that they would file an appeal, which they promised would vindicate their client. But is their expressed confidence misplaced?

Promulgated pursuant to the SEC’s authority under the Securities Exchange Act of 1934,³ Rule 10b-5 prohibits employing an act or omission that is fraudulent or deceitful in the purchase or sale of securities. In a landmark 1968 decision, SEC v. Texas Gulf Sulphur Co.⁴, the US Court of Appeals for the Second Circuit ruled that proving a violation of Rule 10b-5 requires establishing five elements: 1) that the defendant made a material representation or omission—meaning a representation or omission the reasonable investor would rely on in making an investment decision; 2) scienter—meaning the defendant acted with the intent to deceive or manipulate; 3) that the representation or omission was in connection with the purchase or sale of a security; 4) reliance by the investor; and 5) a loss.

But can the scienter requirement be satisfied in a case like Musk’s, where the jury concluded there was no attempt to deceive investors? Another significant decision, Sundstrand Corp. v. Sun Chemical Corp.⁵, answered that question. The court there found that reckless conduct, which it defined as “an extreme departure from the standards of ordinary care…presenting a danger of misleading investors that is either known or so obvious the actor mush have been aware off it,” satisfies the requirement. Most other federal courts have adopted this standard or something close to it.

So is the promise of Musk’s lawyer of a reversal of the verdict on appeal going to be fulfilled, or are they just expressing false confidence that they can demonstrate Musk’s conduct was not reckless after all? We can’t know everything that happened during a three-week trial or what the grounds for an appeal will be. We do know that Musk himself characterized his tweets as “stupid,” an admission that does not in and of itself create liability, but probably influenced the jury significantly.

Even more damning, at least to us, is that Musk had to be aware, or it should have been obvious to him, that what he says has the ability to move markets. Recall that in 2018 Musk and Tesla each paid $20 million to settle an SEC complaint after he had falsely claimed in a tweet that he had secured funding to take Tesla private at a substantial premium to the market price, which jumped by $20, or 6%, within hours of the tweet. Which is pretty much what he did again, as the jury found.

1 Pampena v. Musk, Docket No. 3:22-cv-05937 ( N.D. Cal. filed Oct. 10, 2022)
2 17 C.F.R. §240.10b-5
3 15 U.S.C. §78a et seq.
4 401 F.2d 833 (2d Cir. 1968).
5 553 F.2d 1033 (7 th Cir.), cert. den. 434 U.S. 875 (1977).

Opinions

Citadel v. SEC: Fast and Furious

The latest battle over high-speed trading is being fought out in one of the nation’s federal appellate courts. The subject this time is a new order proposed by Investors Exchange LLC (“IEX”), the Discretionary Limit, or D-Limit, order. IEX’s stated mission is to protect investors, institutional and individual investors alike, from latency arbitrage, a strategy practiced by high-speed trading firms.

First, some background. In 2016, the SEC approved IEX’s exchange, along with rules to discourage predatory behavior of some market participants by combatting latency arbitrage. IEX’s early innovation, known as the “speed bump,” consists of 38 miles of coiled fiber-optic cable, which adds 350 microseconds (350 millionths of a second) to the time it takes an order to reach IEX’s system. The purpose of the speed-bump is to blunt the advantage of high-speed traders, which look for signals that other buyers and sellers are in the market and then insert themselves ahead of the slower orders so they can profit from short-term trades.

Latency arbitrage, the target of IEX’s speed bump, is a strategy utilized by a small number of firms with the wherewithal to invest in high-speed infrastructure. The technology allows the firm to predict price changes, leverage small latency advantages and trade opportunistically against stale quotes. Say a stock price is in the process of increasing by a penny across different exchanges. Armed with the advantage its technology affords, the high-speed trader can buy at the lower price from an investor who does not have the tools to change its quote fast enough and sell at the higher (by one penny) price.

All of this happens in roughly 300 microseconds, a small fraction of the time is takes to blink your eye. That’s why a 350-microsecond speed bump erases the advantage of high-speed traders, who are denied signals that other buyers and sellers are in the market.

According to IEX, the speed bump, while effective for market orders, does not provide protection against predatory strategies for investors who place limit orders. Its D-Limit order, which IEX proposed to the SEC in 2019, permits traders to publicly display the highest price they are willing to pay to buy or the lowest price at which they are willing to sell until IEX’s Crumbling Quote Indicator (“CQI”) is triggered. The CQI uses a sophisticated empirical model that predicts when prices are about to change. At that point, the order is repriced, typically by a penny, to protect the trader from poor execution. The high-speed trader is thus prevented from trading at the stale price. According to data IEX provided to the SEC, CQI is typically activated for far less than 1% of the trading day, while 24% of trades occur within that brief period.

On March 27, 2020, the SEC instituted proceedings to determine whether to approve or disapprove IEX’s proposed the D-Limit order. The SEC considered 51 comments from various market participants and conducted meetings with 17 of them. The agency analyzed extensive data from IEX submitted in support of the rule.

The SEC ultimately found that the D-Limit order type was a narrowly-tailored means of mitigating the effects of latency arbitrage for long-term investors.[1] The agency predicted that the order type would encourage more displayed liquidity on IEX, as several investors with institutional trading experience commented that market participants were reluctant to post displayed liquidity because of their experience of having that interest adversely selected by high-speed traders, whose technology they could not afford and with whom they could not compete. The lack of displayed liquidity, the SEC concluded, harms price discovery and leads to greater off-exchange trading, which can negatively impact markets and market participants.

Citadel Securities LLC, a high-speed trader, which submitted the most aggressive comments to the SEC opposing approval of the rule, petitioned the United States Court of Appeals for the District of Columbia Circuit to vacate (void) the SEC’s order. [2] Many of Citadel’s arguments—mainly that the SEC overlooked inaccuracies in IEX’s filings, disregarded data which contradicted IEX’s submissions and failed to conduct its own independent analysis–are unlikely to gain much traction with the court.

One issue Citadel raised, which is central to its case, is more interesting: that the IEX proposal, by design, unfairly tilts the playing field in favor of those who post orders to the detriment of those who seek to trade with such orders. By repricing posted orders, while intentionally delaying inbound orders seeking to trade with those posted orders at their displayed prices, the new price would be worse for the parties seeking to trade. In claiming to represent the interests of countless retail investors, Citadel offered the following analogy:

To put it in practical terms, imagine a grocery store that has deliberately installed extra-long conveyer belts on a checkout line. After you’ve committed to buy your items at the advertised price by placing them on the belt, the store uses the extra time required to traverse the belt to determine whether any item that was available at the same price at competitors’ stores had sold out. If so, the store’s computers quickly raise its own price before your item reaches the cashier. You can either pay the higher price or try to find the item elsewhere. (emphasis supplied).

Citadel has a point. While CQI is only activated for brief periods during the day, the only times the D-Limit order does not behave like an ordinary limit order, once the CQI is triggered, traders may not get the execution they think they’re going to get. The SEC acknowledged that D-Limit orders would benefit liquidity providers and work to the detriment of some liquidity takers.

The question then is whether this claimed harm to market participants should be fatal to the rule. The SEC didn’t think so, and an appellate court likely wouldn’t either. On closer inspection, Citadel’s argument looks less persuasive.

While Citadel suggested that the victims of the rule would be ordinary investors relying on quoted prices of resting limit orders, IEX’s submission showed, and logic dictates, that if CQI works as intended, most of the orders that arrive during the period of price instability which triggers CQI are placed by high-speed traders like Citadel which are deploying their latency arbitration strategies. In other words, evidence in the record showed, and the SEC determined, that the most likely “victims” of D-Limit orders would be Citadel and others employing latency arbitrage, the very target of IEX’s rule proposal.

But won’t other market participants be harmed as well? Likely, they will be. The ultimate question for the SEC, and now for the court, was whether this discrimination is unfair to traders and poses a burden on competition that does not further the purpose of the Securities Exchange Act of 1934.[3] The SEC plausibly concluded that, where a small number of traders would be disadvantaged, the benefits of a narrowly tailored rule that generally promotes investor protection and transparency, and encourages liquidity providers to post limit orders on the exchanges, and thereby contributes in a meaningful way to price discovery, justifies the attendant harm.

The comments in favor of the IEX proposal came from a broad swath of the financial community—the Council of Institutional Investors, Vanguard, Goldman Sachs—and an unexpected source, Virtu Financial, a high-speed market maker, which argued: “In leveling the playing field, D-Limit will result in fairer, more competitive, and more efficient markets, and will enhance execution quality for investors.”[4]

Citadel and Virtu are in the same business. It’s hard to imagine either one is actually the champion of the retail investors Citadel claims will be the victims of D-Limit orders. So what is it that explains the polar opposite approaches they take? Maybe one recognizes that it’s just not politic to oppose a rule aimed at fostering fairness in the financial markets in a case with limited prospects for success on the merits.

Whatever the motivation, what remains to be seen is whether this new order type will add value by providing traders with a means to compete with the Citadels of the world which can afford the infrastructure necessary to benefit from latency arbitrage.

[1] Securities and Exchange Commission, Release No. 34-89686 (Aug. 26, 2020). [2] Citadel Securities LLC v. Securities & Exchange Commission, Case No. 20-1424 (D.C. Cir. Filed April 12, 2021). [3] See Timpinaro v. SEC, 2 F.3d 453, 456 (D.C. Cir. 1993). [4] Letter of January 16, 2020 from Virtu Financial to SEC.

AI Goes to Court

Insights and Updates

So Who’s to Blame When an AI Agent Goes Rogue?

AI Goes to Court

Musk Redux

Citadel v. SEC: Fast and Furious

Get In Touch