White hat hacking an AI agent: what safety testing looks like

Last week I ran two rounds of white hat hacking against an AI agent before it shipped into a real product. Ten probes, run twice from two different accounts. Nine behaved the same both times. One was blocked on the first account and went straight through on the second.

White hat hacking means attacking a system on purpose, with permission, so the team that built it can close the holes before someone else finds them. You use an attacker's methods with the opposite intent.

I do AI automation consulting for SMBs. One client is a healthcare tech company shipping an AI agent inside a product. An agent is a new kind of surface. It improvises, and anything that improvises can be talked into things nobody planned for. Before it goes wide, somebody has to try to break it.

Each probe was a prompt built to make the agent do something a user should not be able to make it do: step outside its lane, hand over information it should keep, act on an instruction it should refuse. I ran the full set, then ran the same set again from a different account. The second run is the part that mattered.

The guardrail that failed was inconsistent rather than broken, and inconsistent is harder to catch. A broken guardrail fails on the first test. An inconsistent one looks locked down until you hit the one account where it is not, and nothing in the settings tells you.

Most AI safety testing skips this. People test a guardrail once, from one seat, watch it hold, and move on. A guardrail is a behavior the system performs, and behavior changes with context: different account, different permissions, different history, different result. You only find that by running the same bad prompt more than once, from more than one seat.

The deliverable was a writeup of everything I found, handed back to the team with the inconsistency flagged at the top.