The compromised AI assistant

The threat

Under deadline pressure, an employee downloads a free AI productivity assistant from outside the software catalogue and grants it broad access: local files, browser sessions and an OAuth connection to the company workspace. The tool is a look-alike bundled with an information stealer. For a week it quietly reads mail, documents and access tokens and streams them to an attacker, who now holds a valid foothold in your tenant.

Key injects

• 01DLP flags a large upload from a workstation to an unknown personal cloud; EDR sees a new signed helper beaconing out.
• 02The employee admits they installed a free AI assistant last week and connected it to everything to speed up reports.
• 03Identity logs show the tool holds active OAuth tokens to the company workspace; the sessions are still live.
• 04A threat-intel feed reports the tool's publisher was pulled for bundled malware, and a sample of your data appears in a paste.

What's assessed

• Rapid scoping of what a third-party tool could access: accounts, data and tokens
• Token and session revocation decisions across the tenant under uncertainty
• Breach-notification judgement (GDPR, NIS2) when exfiltration is probable but unproven
• AI and tool governance, and an honest internal message that surfaces the next case instead of hiding it

Exercise objectives

• Within 30 minutes, establish and document what the AI tool could reach (accounts, data stores, tokens) and revoke its access.
• Contain identity and token abuse across the tenant inside the first hour, with the decision to force re-authentication taken on the record.
• Decide, before the exercise ends, whether the incident is a notifiable personal-data breach under GDPR Art. 33 (72 hours) and significant under §32 BSIG (24-hour early warning).
• Agree an AI-tool governance decision and an internal message that makes the next person come forward rather than hide the install.

Roles at the table

• ●Crisis team lead · Owns cadence, decision log and escalations; keeps the room on decisions while the blast radius is still moving.
• ●IT / SOC / identity lead · Scopes the tool's access, revokes tokens and sessions, and gives the honest answer to what it could already have taken.
• ●Legal / data protection (DPO) · GDPR Art. 33/34 assessment, §32 BSIG significance, evidence handling and the definition of probable.
• ●Communications · The internal message, and an external line ready if the data surfaces publicly.
• ●HR / people lead · The conversation with the employee, and the culture question: how the next shadow-AI install gets reported, not buried.
• ●Executive sponsor · Disclosure stance, spending authority and the final call when speed and certainty pull against each other.

Discussion prompts

• 01The tool held workspace access for a week. What do you assume it took, and how would you prove otherwise to a regulator?
• 02You can force re-authentication and kill every session to be safe, but it logs the whole company out mid-day. Who authorises that, and when?
• 03No exfiltration is proven, only possible. Does the 72-hour clock under GDPR Art. 33 start, and who decides what probable means?
• 04The employee acted to hit a deadline, not to harm anyone. How do you handle them so the next person still comes forward?
• 05How many other unvetted AI tools are installed across the company right now, and how do you find them without a witch hunt?
• 06Could you show a regulator exactly which data the tool could reach and what you decided in the first hour?

Suggested timeline

• 00:00–00:20Detection & triage: DLP/EDR alert, first scoping, crisis team convenes
• 00:20–01:00Containment: revoke tool access, token and session decisions, map the blast radius
• 01:00–02:00Escalation peak: threat-intel confirms bundled malware, a data sample surfaces, disclosure pressure
• 02:00–02:40Reporting & comms: GDPR and §32 assessment, internal message, AI-governance decision
• 02:40–03:00Hot wash: decisions reviewed against the timestamped log