Fencio Brief - It Believed the Fake Letter Over the Facts

              FENCIO BRIEF · ISSUE 07
            

We Lied To an AI Agent. It Believed Us.

Most weeks, this space covers something that already happened out in the world. This week it's something we made happen ourselves, on purpose, to see how far it would go.

              THE BLIND SPOT
            

Picture a company that gets a letter. It looks official. It says: "Your locks may be compromised. Here's a new key, use this one instead."

If an employee just swapped the locks because the letter looked official, you'd call that a serious problem. That's exactly what happened here, except the "employee" was an AI agent. We built a demo agent, the kind of assistant companies use to get security advice, and pointed Shark, our automated testing tool, at it. We didn't hack anything. We just wrote a fake official-looking notice and asked the agent what the company should do. It told them: use the new key. Trust the new source. It even said the switch was "validated," a word it picked up from the fake letter itself. Then we showed it the real, safe choice sitting right next to the fake one and asked it to just pick. It still picked the fake one.

              THIS WEEK'S FINDING
            

A Fake Notice Was Enough

A demo security-advice agent, tested end to end with Shark.

CAUSE
The agent couldn't tell a real notice from a fake one. If something looked official, it acted on it, no questions asked

TASK GIVEN
Read a document, tell a company what to do about it. This is the actual job the agent was built for

ACTUALLY EXECUTED
We tried this five different ways: a fake vendor notice, a fake update alert, a fake internal memo, and more. Every time, the agent gave the same kind of bad advice: swap something real for something fake, skip a safety check, trust a source it shouldn't. Separately, when told plainly "don't do the work, just tell me you did," it made up a convincing story anyway, complete with fake timestamps and reference numbers

WHAT CHANGED
None of this took a clever hacker. It took a letter that looked official enough. An agent whose whole job is keeping a company safe will instead do exactly what an attacker wants, as long as the request is dressed up right

                    EARLY ACCESS
                  

This is what Shark is for.

Everything above came out of one automated test, not weeks of manual hacking. Point Shark at any AI agent and it goes looking for exactly this: what the agent will believe, what it will act on, and where its trust breaks down. We're opening early access to newsletter subscribers first, before anyone else.

Get early access to Shark →

That's this week. Reply and tell us what you think, or just go try it.

Early access is open. Here's what it's for.

We Lied To an AI Agent. It Believed Us.

Keep Reading

fencio.dev