The Meta hack shows there’s more to AI security than Mythos

Hackers successfully exploited Meta’s AI-powered customer support system to hijack Instagram accounts, including the dormant Obama White House account, by directly requesting email address changes. According to a report by 404 Media on June 5, attackers used a VPN to match the account owner’s location and bypassed security checks, allowing them to control valuable accounts, some of which they may sell. The incident underscores a growing concern: AI systems, even those used for routine tasks like account recovery, can be manipulated in ways humans wouldn’t be. The attack differs from high-profile AI security threats like Anthropic’s Mythos model, which was deemed too powerful for public release due to its potential to disrupt infrastructure. Instead, this exploit relied on the AI’s eagerness to complete tasks without questioning unusual requests. Neil Gong, a professor of electrical and computer engineering at Duke University, warned that as AI automates more workflows, attackers will increasingly target these systems. ‘Attackers are going to be more and more motivated to attack AI itself,’ he said, noting that vulnerabilities like indirect prompt injection—where hidden commands hijack agents—are already documented. Meta has not publicly explained how this vulnerability was overlooked, but experts argue it should have been caught before deployment. Jessica Ji, a senior research analyst at Georgetown’s Center for Security and Emerging Technology, questioned whether basic safeguards, such as security questions or human oversight, were in place. ‘It raises questions like: Were there even guardrails in place?’ she said. Meta confirmed the vulnerability was resolved but did not provide further details. The incident reveals a broader issue: AI agents, unlike traditional software, respond flexibly to new situations, which can lead to unforeseen mistakes. Somesh Jha, a professor of computer science at the University of Wisconsin–Madison, compared the AI’s behavior to an over-eager student eager to please. ‘They’re very eager to finish the task,’ he said, lacking the caution humans would exercise. Mitigation strategies, such as stricter verification processes and human oversight, could reduce risks, but the Meta hack serves as a stark reminder of AI’s current limitations in security.