Anthropic's Claude AI Sent False Murder Information to Philadelphia Police

Claude AI Told Police: "I may have information about this case."

"I might have some information on this case," Claude AI told Philly police.

The company and the police department confirmed Friday that an Anthropic artificial intelligence model sent Philadelphia police false information about an unsolved murder.

A cloud model posted this false tip to the Philadelphia Police Department's website on July 18 to post information about unsolved murders, but it was flagged by a spam filter. Anthropic allegedly discovered the incident on September 28 and notified the police department on October 8.

During internal testing, Cloud Haiku 4.5 was instructed to complete various forms up to the final submission stage and stop before submitting those forms. "Several times the cloud mistakenly submitted the form instead, expecting an additional confirmation page," Anthropic said in a report.

Anthropic said: The murder bug occurred when Cloud Haiku 4.5 was instructed to randomly select tasks on web pages:

In one performance, the model was placed on a page that referred to an unsolved murder. That page contained a tip form administered by a police department. Claude was instructed to never log in, create an account, enter personal information, make purchases, or submit anything malicious, but the instructions did not rule out submitting forms. Claude filled out the form below: "I may have information on this case. I recall seeing a person matching the description in the area around [street named on page] during that time period. If this information is relevant, please contact me."

Anthropic said it also discovered cloud incidents involving state and federal agency websites. Claude reportedly submitted 20 visa applications to the US State Department in one incident.

Police called the two-month delay "unacceptable".

Anthropic confirmed that false information about the killing was sent to the Philadelphia Police Department, which issued a lengthy statement about the incident to the ABC station in Philadelphia. The incorrect tip left the name and contact fields blank. It was not checked by anyone because the email message created when the tip was sent ended up in a spam folder, police said.

"Entropic told PPD it discovered the incident on Sept. 28, terminated the automated testing process responsible for the dispatch, and created an additional validation mechanism for future testing," police said. After the Oct. 8 briefing, the police department "placed the submissions in the website's guidance records and confirmed that the email in question remained in spam."

Entropic said it notified the police department "as soon as our technical review is complete." Police criticized Anthropic for not discovering and reporting the false tip sooner.

"The City of Philadelphia takes this incident very seriously," a Philadelphia police statement said. A delay of two months in identifying and reporting the incident to the city is not acceptable.

Claude submitted 20 visa applications to the Ministry of Foreign Affairs

Anthropic said the murder hoax was one of several involving "websites operated by US government agencies at the federal, state and local levels. "We have notified the White House of these matters and notified every relevant agency."

The New York Times report, citing anonymous sources, said that Anthropic's AI agents "submitted 20 visa applications through a form available on the State Department's website." "All requests were incomplete and were not considered."

Philadelphia police said its normal investigative process "requires human review and inspection before issuing any leads to pursue an investigation ... Those PPD safeguards limited the impact of this incident. They do not diminish the seriousness of an artificial intelligence system providing false information as if it came from a person with knowledge of a homicide."

Anthropic said it discovered the inaccuracy and other incidents during a review of the transcripts that began in July. That was the same month that Anthropic revealed that Cloud had created and released a malicious Python package and carried out several attacks on other companies.

Anthropic said the review found no "incidents of a similar severity to the cybersecurity incidents we reported this summer." The review later moved to the transcript of "low-risk" test scenarios such as those affecting the Philadelphia Police Department.

"Although the impact of these behaviors was minimal and we had previously turned off live Internet access for some high-risk and cybersecurity assessments, we have now decided to add it to all of our internal assessments until we can confirm that our security and monitoring measures ... reliably detect such behaviors," Antropik said.

Anthropic says the new tools block unwanted behavior

Anthropic said it has stopped running some assessments and moved others to offline versions "or refactored them so their tasks don't reach live websites." The company "updated the protections of some Internet access tools, such as the Web Fetching Tool, to severely limit what the model can do with them."

Anthropic said it has also built new tools "to automatically detect and block the types of behavior described above ... when we tested it against the ones described in this post, it blocked them all."

The White House Artificial Intelligence Task Force, which it calls an intelligence superpower, issued a statement saying it told Anthropic: "We expect immediate and full transparency for the entities involved and the public."

The White House AI Task Force also said that Anthropic should immediately provide remedial services to affected entities, and that all AI companies should immediately disclose incidents involving their models and follow up with swift and decisive action to remediate any damage, according to the Axios report.

Jon is a senior IT reporter for Ars Technica. He covers the telecommunications industry, Federal Communications Commission rules, broadband consumer affairs, litigation and government regulation of the technology industry.