The AI Attack Surface No One is Talking About
Prompt injections, jailbreaks, tool endpoint hacks? Those are just some of the ways a hacker can exploit your chatbot. Let's talk about one that isn't being discussed enough: long term writable memory exploits.
BUILDING AI
7/7/20262 min read
I keep getting pulled into the same conversations about AI security, and they always start in the usual ways:
Tool-calling endpoints. Prompt injection. Jailbreaks.
These are the types of attacks that are getting a lot of attention right now and deservedly so. But in a recent project, I found something that had me questioning if I was doing enough, and it wasn't any of those.
It was writable memory.
As a product manager, security is one of those non-functional areas that I’m not expected to weigh in on too heavily, but as a former developer, the hairs on the back of my neck stand up thinking about all the ways that someone could take advantage of a platform I built if I wasn’t careful.
I had been asked to continue building on the personalization system the team had built out on GenAI. Having launched a variety of personalization products myself, it felt off.
I pushed on the outlandish edge cases, sure. But I also tested the ones a user with bad intentions might try.
I dove in, asking our customer-facing chatbot to change its personality to make it easier to converse. Then I explained how this would be helpful for everyone. It was too easy.
I wasn’t injecting prompts to access someone else’s data. I wasn’t even trying to get the chatbot to work for me and use tools to perform a task that it wasn’t meant to do.
I was thinking about how personalization works. How does someone decide to provide a recommendation to you?
I was chipping away at its personality, and that personality is your brand in today’s AI-enabled ecosystem.
In a systematic way, I had changed long term memory that could impact users besides me.
I had gone to a place where no meaningful guardrails existed, a place where state persisted between sessions. A surface that could be written to without anything watching.
Left unaddressed, the compromised memory layer could have retrained our chatbot to send customers to a competitor's site or misrepresent the brand in the worst possible ways. We caught it. We fixed it. But nobody on the team had it on their threat list.
...and that's the scary part.
The system looked solid by every visible measure. The endpoints were protected. The prompts were hardened. Writable memory does not announce itself. It looks like any other feature and not an attack surface.
For those of you working in AI governance and security: when did someone last map every surface your agent can write to, not just read from, and ask what happens if that state gets poisoned?
