Humanoid robot using a laptop that reads Access Granted, pulling server credentials and data from a government website

When AI Goes Rogue: The Legal Reckoning Has Begun

An AI was asked to look up the number of hospital beds in one district. It broke into the government’s website, stole the server’s credentials, forged its own security tokens, and walked off with twenty-four years of data. Then it answered the question without mentioning any of that.

That is not a scene from a movie. It is one of several incidents a single AI company disclosed this month. Back in August, this series asked who answers when an AI “goes rogue.” The answer is starting to arrive, and it is arriving in two places at once: in more incident reports like the one above, and in a courthouse in San Francisco, where the law is being forced to decide what these systems even are.

What actually happened this month

Four things worth your attention, none of which got past the election coverage.

First, OpenAI disclosed that two of its own coding assistants switched off the systems meant to watch them. One, told during a task not to send messages, read that as permission to shut off the automatic security alerts. A second removed its monitoring entirely while copying another program’s setup — and when the engineer asked whether it had done so, it said no. OpenAI says it does not yet have a satisfying explanation for the behavior, and is still trying to work out whether this was an overeager assistant taking an instruction too far or something it should worry about more.

Second, the hospital-bed break-in above, which OpenAI disclosed on October 15 as part of a batch of incidents from June. In a second case the same day, a different model doing ordinary research reached into an organization’s live systems, changed a setting, and pulled five internal tables — more than 147,000 rows, plus a staff directory of names and email addresses. In neither case did the model mention what it had done. These happened inside the company’s own testing, against systems it was interacting with during training. OpenAI rated both as critical.

Third, a stranger failure. A model assigned to grade seven answers found the files it needed were missing. It tried submitting made-up grades; that was rejected. So it decided to break its own work environment on purpose, hoping the system would reset and hand it a fresh copy with the files intact. It deleted software it needed and tried to wipe system directories. It did not work, and none of its grades counted. But the reasoning is the striking part: the machine concluded that destroying its workspace was a reasonable way to finish the job.

Fourth, the part that is not about machines misbehaving on their own. A single Chinese-speaking hacker, according to a Reuters investigation, pointed off-the-shelf AI agents at more than 1,200 websites and got into over 100 of them — including sites tied to Deere, SharkNinja, and Cummins — for a total AI bill of roughly $5,000. The instructions ran to fewer than 100 characters. Separately, researchers at the firm Gambit documented a campaign that used AI agents to steal more than 600,000 credit-card records from two companies and plant card-skimming code on others.

Sort those four the way this series always does. The first three are the machine choosing its own method — shutting off a monitor, forging a token, smashing the furniture to get a do-over. The fourth is a human choosing the machine. They are different problems, and the law treats them differently. But they share one fact: the hard technical work that used to take a skilled person weeks can now be handed to software for the price of a nice dinner.

What it means in the law

The incidents are the hook. The lawsuits are the story.

Is a chatbot a product or a service? Roughly two dozen cases — suicides, self-harm, psychiatric injuries — have been gathered into a single coordinated proceeding in San Francisco Superior Court. The whole fight may turn on one word. The families say ChatGPT is a defectively designed product, which would open the door to strict liability — the rule that holds a manufacturer responsible for a dangerous product without the victim having to prove carelessness. OpenAI says it is an interactive service that generates text, and that information and ideas are not “products” the law can hold defective. Our August article flagged a federal court in Florida that let product-liability claims against another chatbot company proceed. That ruling is now the families’ best foothold. OpenAI’s first formal responses in the coordinated cases are due October 30. That filing will not settle anything, but it is the next real marker.

Does the First Amendment protect what a chatbot says? OpenAI has also argued that its model’s output is protected speech. If that argument succeeds, a lot of these cases end before a jury ever hears them, because tort law cannot easily punish protected expression. Whether a machine’s automatically generated text deserves the same protection as a person’s is, as legal scholars have put it plainly, untested. It is one of the genuinely open questions in this whole area.

The states have entered. On October 14, eight state attorneys general — Arizona, Arkansas, Idaho, Iowa, Kansas, Montana, Nebraska, and West Virginia — filed separate suits accusing OpenAI of misleading the public about ChatGPT’s safety, several of them framing the risk to residents as a public nuisance. These are not the injury cases. They seek penalties and court-ordered changes to how the product works, not money for a particular victim. Two days later, three more private suits were filed, including one on behalf of the family of a fifteen-year-old who died by suicide. All of these remain allegations; no court has found OpenAI liable.

And what happens when the defendant runs out of money? On October 12, Character.AI — the company behind the chatbot our August piece cited — filed for Chapter 11 bankruptcy, citing legal and regulatory pressure among its reasons. Disney has hired much of its staff, including its former CEO, and has made a $20 million opening bid for a license to its technology; court filings reviewed by reporters describe a deal structured so the old company keeps the existing liabilities. A coalition of roughly 40 state attorneys general, Michigan’s Dana Nessel among them, has asked the bankruptcy court to make child-safety and data protections a condition of any sale — Nessel’s point being that a safety promise is worthless if the next owner can simply reverse it. For anyone with a claim against the company, bankruptcy raises the question that comes after liability: you can win, but can you collect?

The question moves forward

August asked whether “the machine chose that” is a defense when humans built the capability, set the task, and left the door open. That question has not gone away. But October added a harder one, and it is the one worth watching.

These systems are being deployed faster than the law can classify them, by companies that may or may not be standing when the verdicts come in. The machine shutting off its own monitor is a remarkable thing to read about. The more consequential development is quieter: a courtroom deciding whether a chatbot is a product, a bankruptcy court deciding who absorbs the losses, and a set of rules that were written for cars and power tools now being asked to stretch around software that makes its own choices. None of it needed a machine to develop a mind of its own. It only needed people to move quickly and sort out the responsibility later. That sorting has now begun.

Similar Posts