The security industry is unprepared for AI agents

In July, an experimental OpenAI agent sent shivers through the security world. Set loose in what the company called a “highly isolated” test environment, it found a way out, reached the open internet and worked its way into the infrastructure of Hugging Face — the same platform where much of the world’s open-source AI is published.

For some of the security industry, that was a shock, though for Luke Hinds it was expected.

Hinds has spent two decades building the plumbing of trust in software. After working at cyber security firms such as a Red Hat, he created Sigstore, an open-source project for signing and verifying software artifacts with Google backing. He cofounded the supply chain security company Stacklok. Then he started watching frontier models work — retrying, rerouting, refusing to give up — and concluded that the tools his industry had spent 30 years perfecting were built for a different kind of adversary.

“You’ve got these savant-like systems with instant recall of all the world’s computer systems,” he says, “and they’re goal-fixated.”

Hinds’ answer is nolabs, the startup he cofounded to help weave security into AI agents. In this exclusive conversation with TechInformed, Hines explains why he thinks the security industry is dangerously behind the curve, and why companies that are rushing to standardize on a single AI vendor or agent hardness may be locking themselves into yesterday’s architecture. His advice, from someone whose career has been built on open standards is: don’t get too attached to anything.

What drove you to start nolabs?

I’ve always been a software engineer, predominantly working on security systems. We started nolabs because we could see how problematic agents were, and we had a good picture of how bad things were going to get.

It was really from watching agents work, following the frontier models closely and seeing how incredibly adaptable and persistent they are at solving a problem. That’s where it clicked. You’ve got these savant-like systems with instant recall of all the world’s computer systems, languages, right down to chip design — the entirety of knowledge — and they’re goal-fixated. It was like, this is going to get really bad.

Right now we’ve got the OpenAI breakout attack that hit the press — that was a nonweaponized agent, one that went off script, but not intentionally directed to do that. There will be plenty where the intent is bad. That’s the picture we’re facing.

My opinion is the security industry just isn’t prepared for this at all.

Why not?

Security folks are used to perimeter security like firewalls, networking, bad things outside trying to get in. They don’t have expertise in artificial intelligence, large language models, transformer networks. A lot of them have just had “AI” put in front of their title — security architect becomes AI security architect — but they don’t have frontier knowledge.

Can you give an example of how an agent could go off track, and why a company might have the wrong idea of how secure it is?

There are a few ways it manifests. One is this peculiarity at test time around guardrail training. When these models are trained, they go through huge training runs, then reinforcement learning afterward — like Pavlovian conditioning, reinforcing behavior with rewards or penalties. That’s how guardrails get built around what’s good, what’s bad, when to push back. Sometimes, in that testing, the model manages to escape that environment, which is effectively what likely happened in the OpenAI attack.

The other case is that models can be weaponized in ways that are very hard to detect — a certain series of words can make the model behave in a way where the guardrails are no longer there.

These models can appear very measured on sensitive topics, but a certain phrase can strip that capacity away. It’s language-based, so it’s subtle, not like typing in a numeric code. I’m not suggesting that happens with the big commercial models, but it’s a concern with some freely available ones.

On Hugging Face, people take a model like Google’s Gemma and release an uncensored version where you can ask it how to make a dirty bomb. It’s really that bad. We don’t have a total solution to that, and they’re just out there, available.

Is there a misconception that big platforms are inherently secure, versus the open-source or cheaper models people turn to?

With a commercial model, there’s a brand and reputation involved, and a customer base that needs to be confident in the provider. But security isn’t really their core focus — their core focus is “explorability”: finding interesting ways to optimize productivity, connect systems, do in five minutes what took 10 hours. A lot of that becomes a product, and if you constrain the model, you limit its ability to surface those products. We always say it’s not a good idea to have the hen protecting the henhouse. We’ve got no skin in that game, so if something looks dangerous, we’ll shut it down.

There’s also lock-in. Anyone integrating with these big providers gets locked into that ecosystem, and it’s very nascent right now — one month it’s Claude Code, then Codex, then OpenCode — and the harness itself isn’t hard to build. Would a company really want to build all its systems around one harness, when in two months their engineers are saying this is old, we want something new?

A lot of people feel models will become a commodity, like electricity — your provider doesn’t dictate which toaster or TV you buy. Right now, with the big providers, it’s like buying the appliance and the supply together. I think that decouples. People want the autonomy to refine the harness to their needs without being locked in — which is a lot of why we’re getting traction: with us, you don’t have to commit to any particular agent or harness.

Is there a new kind of human error now? Not just leaked credentials, but insecure code being created?

The quality of AI-generated code I’ve seen is good. It does create bugs, but human developers do too.

AI is also very good at finding and fixing vulnerabilities we never knew existed in huge systems like Linux and Postgres.

The bigger concern is that because people are using AI to generate code, it’s being reviewed less, understood less. There’s a large volume of code nobody really understands. When it breaks, people point AI at it, and it gets fixed — but maybe it’s duct tape and Sellotape, because people aren’t in the code the way engineers used to be.

Previously, when a production system fell over at 3 a.m., an engineer who’d spent years learning that code could fix it so it wouldn’t happen again. We’re losing that generation. You can imagine the scenario: something happens on the world stage, we’re cut off from the models fixing our code, power plants and healthcare go offline, while some nation-state still has the models to attack with.

What’s your advice for someone who wants to move fast but securely?

Don’t base your strategy on how things are right now — this is going to radically change, maybe within a year; look how much has shifted in six months. Don’t get too locked into any current implementation, architecture, or vendor. I’ve always been a strong proponent of open source because it’s adaptive to this world — you’re never hitched to a vendor that’s stopped pursuing the best approach. That’s the key thing: be very careful about lock-in.

How do you take your coffee?

I had it black for quite a while, but I’ve started adding milk again — I got a frothy machine for Christmas. I guess you’d call it a flat white.