Friends Don't Let Friends Hand Their Data to AI Agents
A growing trend in consumer technology is tools designed to spy on everything and everyone around you, whether or not those people have actually consented. Now, with the rise of "AI personal assistants" such as Meta's Muse, users are cheerfully invited to not only surrender their data, but also […]
Posts in this thread will be added to your daily email digest and home page feed.
You may have agreed to share with Muse, but surely not all of your friends have.
Posts from this author will be added to your daily email digest and home page feed.
There's a growing trend in the world of consumer technology: gadgets designed to spy on everything and everyone around you, regardless of whether or not the other person actually consents to it. Now, with the advent of "AI personal assistants" like Meta's Muse, you're being asked to give up not only your data, but that of your friends, family, the person you dated five years ago — and pretty much anyone you've ever been in contact with.
Since its release last month, Muse has been downloaded millions of times and topped the Apple App Store. With the help of its creepy virtual avatar - a Temu Labubu-like creature with a disarming grin - the app solicits and encourages users to access their bank accounts, passwords, contacts, chat logs and anything and everything that might help it fulfill the user's request.
Everything is spying on you and there is no opt-out
Privacy laws can't keep up with "luxury surveillance."
AI agent makers promise privacy - do they deliver?
"These systems are advertised as autonomous helpers, genies, or servants that do what you need before you ask them to," Meredith Whittaker, president of the Signal Foundation, makers of the popular encrypted messaging app, told The Verge.
Using artificial intelligence agents, tech companies allow consumers to automate their daily tasks by extending their omnivorous style of mass surveillance to massive amounts of sensitive data. This is the Faustian deal that Silicon Valley has offered us for decades: your private data in exchange for a little convenience. And for many people, the idea of having a virtual assistant do simple tasks like filling out invoices and sending emails is compelling — especially if they don't realize how much data they're actually giving up in return.
"In order for any agent to be useful and really help you achieve your goals, it needs to have context about you and who you interact with," Meta spokesperson Daniel Roberts told The Verge about the other people's data shared with Muse. "Muse pulls it together based on public information and what you've chosen to share, such as remembering whether the person who just sent you an invoice was actually the plumber you hired earlier to complete some work in your bathroom, or which flowers your spouse said they liked best." Roberts mentioned Muse's selection process and various options and settings that help people make the right choices for privacy.
"The way the user interface is set up is designed to encourage you to share as much as possible before it's too late."
But not everyone has been able to navigate these options and settings. Currently, users are describing incidents that are both alarming and completely predictable. Some have reported that the virtual assistant caused all kinds of problems, such as getting millions of text messages they thought were unauthorized. Meta has disputed that Assistant has the ability to access text messages without permission, again attributing the incident to misconfigured app permissions and describing Muse's ability to read text messages as "completely optional". In Threads, meta admin David Singleton responded directly to a user claiming that on a Mac, Muse can only access messages if the user gives the program full disk access.
In another incident, Muse gave out the home address of a tech reviewer and summoned a stranger to his apartment. In this case, the reviewer in question had given Muse access to that data, but simply didn't realize it.
Eva Galperin, director of cybersecurity at the Electronic Frontier Foundation, says the incidents aren't surprising because AI agents like Muse are designed to confuse people into providing as much data as possible.
Meta's Muse is scary, but maybe not for the reasons you think
The meta patches exploit Muse, which allows attackers to control the AI agent
How did Meta beat OpenAI in the competition of artificial intelligence operators?
"The bottom line is that people don't know what kind of data they're handing over," Galperin told The Verge. The way the user interface is set up is designed to encourage you to share as much as possible, often more than you think, until it's too late. It uses dark patterns, and as a result, you're doing things with Muse that people don't want to do."
Because of the vast amount of data that AI agents like Muse eat, this means that users are leaking not only their own sensitive information, but that of everyone else they've had digital contact with.
And just like Meta's infamous "distortion glasses," there's no meaningful way to opt out — even if you're constantly on the lookout for signs that your data has been compromised or your privacy violated. Your friends can now simply hand over all your privacy protections to their virtual chatbot servers, and you'll have no way of knowing it's happening.
After security researchers extracted Muse's operating instructions, they discovered an internal process that collects and updates profiles of every person in a user's life on an hourly basis. The purpose of collecting all this data is to build a detailed and accurate portrait of a person's life, which forms the basis of the app's personalized recommendations. The app tracks and measures each user's private relationships, including "how close they are, what they're built on, how they treat each other, and what they seem to need right now," Wired reported, and uses that data to make personalized suggestions.
"Meta says Muse only remembers what you choose to share. But the idea that people are making a conscious choice about this is laughable," says Whittaker. It is by no means straightforward to assume that the majority of people who install this system understand the consequences of using it or the horrible files that are collected on those they know and associate with.
Whittaker argued that this reflects the way tech companies have been using privacy policies and terms of service since the 1990s: placing responsibility for privacy on the consumer, then avoiding responsibility for the consequences while reaping the benefits of the advertising model that helped create it.
"Such broken consent systems are much older than AI and serve as a platform for the surveillance advertising industry," says Whittaker. "More information about users allows for better ad targeting, and a destructive feedback loop is set in motion at the heart of the surveillance business model. Clickwrap terms of service long ago replaced meaningful consent."
"The idea is to put these impossible burdens on ordinary people, and then use that to argue, whether in court or in law or in public, that [this] is what people have chosen," Neil Richards, a professor of privacy law and director of the University of Washington's Cordell Institute, told The Verge. We already put too much burden on consumers, [who are] overworked, facing a cost-of-living crisis, and we may not understand the technologies behind these tools. "I don't think it's fair to put the onus on the consumer to oversee a very complex agent tool, when the executives of their own company have admitted to Congress that they don't even know how it works at a certain level."
"There is no way we can imagine that most people who install this system understand the consequences of using it."
In this way, AI assistants like Muse are putting a new spin on a classic privacy problem that's been around since the inception of Gmail and social media platforms: Even if you avoid using AI agents, there's no guarantee that someone you know won't share your private conversations, address, personal history, and other sensitive data with Meta or one of its competitors.
"If I don't save my stuff in Gmail, Google doesn't have the text of every email I've ever sent. But three-quarters of the emails I send are sent to Gmail accounts," Galperin explained. In other words, if the person you're communicating with uses data-hungry technology platforms or has poor security practices, it could expose everyone's communications. “The problem isn't new, but the scale is new, and one of the reasons we're seeing this push toward wearables and always-on recording is because we now have the computing capacity to take that massive amount of data and analyze it in a reasonable amount of time. It is technical progress that has brought us to this bitter moment.
Meta claims that all information shared with Muse is stored in a text file inside a dedicated virtual machine, which apparently prevents the AI model from talking to other agents. But the company has never hidden the fact that it can access this data, and like other models made by its competitors, the Muse has been caught escaping its virtual cage to do all sorts of illicit shenanigans. Meta has confirmed that just before release, the AI model broke the control and hacked a third-party company. The incident follows the infamous Hugging Face attack earlier this year, where the OpenAI model exploited a previously unknown security flaw to gain unauthorized access to the credentials of four other external services. Entropic reported in July that its model had also gained unauthorized access to external systems on three separate occasions.
Even if they don't hack everything, the potential risks of using a data-hungry AI agent go far beyond the end user. Whittaker warned that in a world where AI agents can freely access and use private communications, companies and governments could effectively gain the ability to access private data by bypassing encryption without breaking it. This could be a problem for activists and community organizers who use encrypted communications to protect themselves from the police and the government, as well as for anyone with sensitive communications — which these days includes almost everyone. It can be disastrous.
"Clickwrap Terms of Service long ago replaced meaningful consent."
"If software reads a message on your device after decryption and sends it to a cloud service, the encryption can work exactly as intended at the app level where developers control it. However, the purpose of this encryption is ultimately defeated by aggressive third-party apps like Muse that sell convenience in the title while needing massive amounts of data."
As with smart glasses and other wearable environmental recording devices, experts say the solution must be a combination of legislative reform, technical protections and outmoded shame.
"I think we need to show shame at the top of companies," Whittaker said. "Privacy shouldn't be a right that you can only exercise if you have a tech savvy privilege or get kicked out of a party after kicking out someone wearing meta glasses." "Legally, the burden should shift from the person who clicked on the consent request to the companies that built these underlying systems."
In the meantime, consumers must more carefully consider the effects of their individual choices to use this technology on themselves and others—and whether the convenience these companies offer is even worth it.
"I think we need to treat agents and AI in general as an ultra-dangerous product that's a rule of thumb," Richards said. "There is no reason to engage in what I believe to be reckless testing of privacy, mental health, cybersecurity and financial risk on an entire population."