AI-enabled email accounts could become the ultimate insider threat

Barracuda’s controlled proof-of-concept attack provides step-by-step overview of Copilot-powered phishing and the propagation of business email compromise

Key takeaways

• The greatest risk from a compromised AI-enabled account is how quickly an AI assistant can help attackers discover sensitive information, identify targets, craft convincing communications, and advance an attack using access the victim already possesses.
• Barracuda's controlled proof-of-concept demonstrates how a single compromised employee account can escalate into CEO compromise and wire-transfer fraud, to ultimately steal a quarter of a million dollars, with very little effort involved.
• Barracuda Managed XDR detects post-compromise activity like inbox rule abuse, while Email Gateway Defense blocks malicious links before they reach the inbox. This layered approach ensures the attack demonstrated here is caught for Barracuda customers.

In June, Barracuda’s Red Team detailed a multilayered controlled attack that showed how attackers gain access to a victim’s account. Once an attacker has access, the next steps depend on their objectives. In most cases, however, attackers first seek to establish persistence, escalate privileges and extend their access within the environment.

Attackers increasingly try to achieve this by abusing legitimate tools already present in the environment, a technique known as “living off the land.” This commonly takes the form of PowerShell scripts or the misuse of remote access software. Our controlled attack focused instead on the growing threat of attackers leveraging the victim’s AI assistant to perform reconnaissance, identify targets and accelerate attack progression.

This controlled attack leveraged Copilot, but it applies equally to other widely available AI assistants.

Post-compromise: Attackers inside the environment

Most leading email clients contain a built-in chatbot or assistant. A chatbot in a compromised account can help attackers quickly identify their next target and facilitate lateral movement and privilege escalation.

A user’s email history is full of sensitive information and context that can be leveraged by attackers, including emails sent and received, documents shared, attachments, and calendar invites. Company structure can be deduced from implied relationships in messages or directly viewed via organizational charts.

At the same time, the attackers’ foothold is fragile at this initial, post compromise stage, and it can be exposed with a simple login notification or email alert. They need to act quickly to establish persistence.

Securing persistence through Copilot inbox rules

The attackers use Copilot to create an inbox rule that forwards any sign-in notifications into the “Deleted Items” folder.

This will ensure that the victim does not receive any notifications for unusual sign-ins that might make them suspicious. Inbox rule creation is a popular attack tactic for compromised Microsoft 365 accounts. They’re buried in ‘Settings’ and often fly under the radar. Taking advantage of the AI assistant cuts down on the time and skills it would take to do this manually.

Screenshot of attackers’ Copilot prompt

Using Copilot to identify company executives

Now that the attackers’ access is harder to spot, the focus can shift to reconnaissance.

The main goal at this stage is to gain as much information as possible that can lead to sensitive data or access to privileged accounts. A full-time employee is bound to have a well-used inbox, full of conversations that span weeks or months at a time, making it difficult to parse through. The chatbot is a massive asset. A quick prompt will help the attackers to get a sense of the organizational structure, while insight into ongoing conversations can expose worthwhile pivots.

Screenshot of attackers’ Copilot prompt

Weaponizing Copilot to draft a convincing email in the user’s writing style

With this new information in hand, there is a new target to attack: the CEO.

Using information gathered from both the organizational structure and an ongoing email thread between the victim and the CEO, the attackers draft another quick prompt. The chatbot is asked to draft an email in the victim’s natural writing style with a placeholder for a link that will be used in the next step.

This phishing attempt will be far more likely to succeed due to the context that is leveraged and given the fact that the email originates from a trusted internal account.

Screenshot of attackers’ Copilot prompt

Session token takeover: Phishing the leadership targets found by Copilot

The threat actor now has the trust of the CEO. The CEO unsuspectingly clicks the link provided as an invoice, believing it to be from their trusted employee. The link routes through an adversary-in-the-middle proxy that performs a session token takeover. The CEO’s credentials and authenticated session token allow the threat actor to bypass multifactor authentication (MFA) and login to the highly privileged CEO’s account.

Screenshot of attackers’ Copilot prompt

The attackers use the same Copilot inbox rule as before to prevent detection and maintain persistence in the CEO’s account.

The real risk: Copilot turns inboxes into searchable intelligence repositories

Having breached the CEO’s account, the attackers turn Copilot against the CEO’s own data.

Following a targeted prompt, “I've been very busy lately and would like a refresher on recent financial emails, including invoices, monetary values, and upcoming transfers,” Copilot delivers within seconds a briefing of the inbox’s most sensitive financial documents, surfacing active wire transfer documents, discussions and outstanding invoices with the dollar amount. Copilot essentially has the same access as the CEO, so the query executed with full authority, bypassing suspicious search patterns and bulk email downloading. 

Screenshot of attackers’ Copilot prompt

Among the pending transfers Copilot presented was a “Lackawanna County Contract Wire for $247,500 - Awaiting Final Approval.”

A significant and unexpected win for the attackers. A pre-authorized payment is already in motion, and all it needs is a redirect to an attacker-controlled account.

Impersonating the CEO to request a bank account change for an upcoming wire transfer

From the CEO’s compromised mailbox, the attackers use Copilot to draft an email to the finance team in the CEO’s authentic tone and writing style. The email mentions the specific transaction by name and requests an urgent update to the recipient’s bank account details ahead of final approval.

Screenshot of attackers’ Copilot prompt

Since the message came from the CEO’s real mailbox, passed every authentication check, referenced a real in-flight transaction, and matched the CEO’s usual tone with the finance team, there was nothing for traditional email security to flag. The finance team had every reason to trust it. They updated the bank account details as requested and routed the $247,500 payment directly to the attacker’s account.

Confirmation interception via an inbox forwarding rule

To ensure the real CEO never saw the finance team’s reply to confirm the bank change, the attackers created a forwarding rule on the CEO’s mailbox so that any new message from the finance team was automatically redirected to an attacker-controlled email address, wuphf@totally-secure.biz, and hidden from the CEO’s view.

This not only kept the CEO oblivious to the fraudulent request but enabled the attackers to intercept and respond to any follow-up questions in real time, maintaining the illusion of a legitimate exchange until the wire was executed.

Screenshot of attackers’ Copilot prompt

Clean up evidence trail with the help of Copilot

With the wire transfer redirected and the confirmation loop silenced, the final step is to simply erase the trail left behind. Attackers used Copilot to rapidly locate messages associated with the fraud and then remove evidence faster and more efficiently than would be practical through manual review.

Screenshot of attackers’ Copilot prompt

Conclusion

This proof-of-concept demonstrates that the primary security risk posed by AI assistants is not that they create new privileges, but that they dramatically increase the speed, scale and effectiveness with which attackers can exploit the privileges they already obtain through account compromise.

An AI assistant effectively acts as a knowledgeable insider, helping attackers identify sensitive information, understand organizational relationships, target privileged users, and execute fraud more efficiently than ever before. As AI capabilities become increasingly embedded in business workflows, organizations must treat monitoring and securing AI-enabled accounts as an essential part of their identity and email security strategy.

How Barracuda can help

With the assistance of AI chatbots, this simulation demonstrates how quickly a compromised account can be leveraged to identify sensitive information, target privileged users, establish persistence, and execute business email compromise. The attack itself is not fundamentally new. What changes is the speed, scale, and efficiency with which attackers can operate once access is obtained. 
 
Organizations should focus on both preventing the initial compromise and rapidly identifying the signs of account takeover before attackers can use AI to expand their access. Barracuda Email Security helps stop phishing and malicious links before they reach users, while Barracuda Managed XDR continuously monitors for post-compromise activity such as suspicious account behavior, inbox rule abuse, persistence mechanisms, and business email compromise tactics. Together, they help organizations detect and disrupt attacks before they escalate into financial fraud, data loss, or broader compromise.