The truths about AI hacking that every CISO needs to know (Q&A)

Businesses spend countless hours and dollars hardening their systems against cyberattacks. While public and private organizations of all sizes and sectors could take some comfort knowing that they were matching wits against opponents with natural, human limitations, those days may quickly be drawing to a close, says Heather Adkins, Google vice president of security engineering.

Adkins, with co-authors Bruce Schneier and Gadi Evron, is shouting from the rooftops about the impending threat of autonomous AI hacking — and every CISO should pay attention.

Because threat actors have access to the same classes of powerful AI models and automated processes as their targets, Adkins says this pitched battle is about to change in significant and unpredictable ways.

To discuss these and other vital dynamics CISOs will face in the months and years ahead, Google Cloud's Anton Chuvakin and Tim Peacock sat down with Adkins recently for the Google Cloud Security podcast.

What follows is an edited version of their wide-ranging conversation.

Tim Peacock: In your article, you wrote about an AI hacking singularity, where "key parts of the attack chain become automated and integrated: persistence, obfuscation, command-and-control, and endpoint evasion," and called it an imminent threat. How realistic is it? And are AIs really hacking things at scale already?

Heather Adkins: We're all talking about it quite a lot, but we probably won't know the precise answer for a couple of years.

It's just a matter of time before somebody puts all of these things together, end-to-end. And what I fear the most is somebody developing the capability to prompt a model to hack any company, and the model being able to come back in a week with a root prompt. If that ends up happening, I think it'll be a slow ramp over the next six to 18 months.

We're also seeing defense pick up the same tools and use them for the same purposes, so it may not feel as shocking. Of course, things could go very differently, but these are the things that should be on everyone's mind and we should be getting ready for a really different world.

Anton Chuvakin: What's the case for why that doesn't happen?

Heather Adkins: It's good to challenge these ideas, but skepticism shouldn't prevent you from thinking about the problem. Of course, when LLMs reason about deep research, they often end up in very strange places.

So imagine an AI-powered module on an endpoint trying to figure out how to hack a machine, but without any constraints. It gets bogged down in the depths of CPU vulnerability research. It could have just used a well-known bug that's already on the machine.

We see that in some of the vulnerability research that we've been doing: LLMs taking strange thought paths and having to be reined in. Vulnerability researchers, human researchers, and hackers can intuit when to stop pursuing a path that won't be fruitful. But LLMs have no concept of, "hey, this isn't useful, I'm changing direction."

That's a short term problem that someone will solve. Then there's the question of where all the GPU and TPU resourcing will come from for the inference. Again, someone will figure it out. Everybody's machine will eventually have a GPU and some threat actors just want hacking jump points. They're okay with some of this messiness, so we have to keep an open mind about what's realistic.

Anton Chuvakin: What is the Metasploit equivalent for AI-powered exploitation that we need to watch for?

Heather Adkins: It'll be obvious when we start seeing threat intelligence reports on it. Major threat intel companies and teams are already reporting that little pieces of the kill chain are being weaponized. The Google Threat Intelligence Group recently came out with one where they map out what these different tools are doing. There's no tool that has an end to end yet, but we'll see one someday. It's hard to predict because we don't know how it'll get used.

If it's weaponized in a ransomware toolkit and sold on the underground, the rates of incidents may increase. But if it's closely held by a threat actor with really specific targeting, we may not even be able to tell that there's a fully automated platform on the other end. We may only know when it's physically in someone's hand.

Anton Chuvakin: To me, the more serious threat isn’t the APT [advanced persistent threat], it’s the Metasploit moment [when exploit frameworks became easily accessible 20 years ago]. I worry about the democratization of threats.

Heather Adkins: Yeah, I think the best proxy for this will be when we see it in the open-source red teaming tools. I'm surprised it's not in Metasploit already. I bet somebody's got that.

Tim Peacock: What is the worst case AI-enabled hacking scenario you can envision?

Heather Adkins: There's a few possibilities. First, somebody could unleash this as a Morris Worm-type event, either intentionally or by accident — we wouldn't know until it was fully investigated. Or it could look something like the Conficker worm that didn't really do anything, but everybody still panicked and wrote thousand-page government reports on it.

Maybe an altruistic person unleashes it on the world and it patches a bunch of bugs. It really just depends on who puts the pieces together, and their motives.

Anton Chuvakin: The facts point to a pretty strong attacker advantage. If attackers are operating at that speed and if they can just name a company and the model hacks it in a week, the defender seems outmatched. We sometimes say that to defend against AI, you need AI, so what is the genuinely useful tactic to counter it?

Heather Adkins: Right now, we define winning in cyber as preventing attacks. So patch your systems, use strong passwords, and use a security key. But we have to change the definition of winning so it's not just did an attacker get in or not, but how long were they there and what did they get to do?

That means the defender's job is to change the battlefield in real time. If you're a cloud-based infrastructure and your cloud instance thinks it's compromised, it should turn itself off. If someone's trying to abuse an account privilege, like in a service account, and it's starting to do something anomalous, don't let it do it. The permissions should just be turned off.

We're going to have to put these intelligent reasoning systems behind real-time decision making and disrupt decision making on the ground, without causing reliability problems. Maybe you need human approval. Or you shut down one instance and turn up another one.

There are options other than just the on/off switch, but we have to start reasoning about real time disruption capabilities or degradation, and use the whole information operations playbook to change the battlefield to confuse AI attackers. Particularly because they're stumbling around in the dark a little bit and may be less resilient than human attackers.

Tim Peacock: AI is getting good at vulnerability discovery, so how does this impact software supply chain and open source and maintainers and so on?

Heather Adkins: Long term, the hope is to have AI assistants early in the software development lifecycle available to open source developers, commercial developers, students, and hobbyists. They'll catch and prevent most classes of vulnerabilities from reaching production environments. So if you're coding and you try to create an integer overflow, it will tell you it's a bad idea to ship that code.

This will take lots of trial and error and new norm setting. That might be for commercial software development houses to adopt but open source developers will have to figure out what this means for their community and who will support it.

Anton Chuvakin: What makes me nervous is that the science is there, but what about organizational readiness?

Heather Adkins: There's always a natural tension inside business between velocity and safety. These conversations come down to what the solution is. When those solutions are on the market, developers are coming out of school having used them there, and they'll demand them.

This can be cost-efficient and a win-win for business because no CIO wants to ship unsafe code. We just haven't had great tools for doing it yet, and once there's enough public proof points, case studies, and wins, you'll see this change.

Plus, over the long term, you might start to see insurance companies and regulators nudging the market.

Anton Chuvakin: Going back to EDR [endpoint detection and response] days, I remember selling it in a certain region and companies said that since the EDR goes back to the U.S. cloud, they couldn't use it. So I asked, if you're better by 10% or if you're better by 10 times than the next-best on-premises EDR, would they go with regulations or would they go with risk reduction? Everyone said they'd go with regulation. How do you think this concern will play with AI?

Heather Adkins: We're trained to look not just at the risk in front of us, but in the rest of the business too. In regulated environments, ecosystems, and sectors, the regulatory risk may be higher than the cyber risk because the regulator decides whether you exist or not.

We have to avoid being so focused on cyber risk that we forget that businesses have other kinds of risks. Our job is to balance those out for them. Don’t let chasing compliance be the death of a good internal security posture, but also don’t forget that in this fast-moving space, compliance will develop faster than we realize.

Governments are interested in how AI is used in society, and regulators, as we are seeing, are already deeply engaged. Ultimately that means we need open conversations with regulators so that we can use the technology we need to defend ourselves and yet doing so in a way that is responsible. I’d hate to find anyone in a situation where they can’t use the best model to keep hackers out because of a regulatory concern. But that also has to be balanced with some realities. As with most complex societal issues, dialogue is key.

Anton Chuvakin: What do businesses need to change if AI innovation outpaces legislation?

Heather Adkins: It's really important that they talk through these issues with their regulators. As these solutions become available and relevant, and we can put pieces together, we may be able to move the regulators. Of course, that's not always appropriate or possible. The world is highly fragmented and very complicated, and there are no easy answers. These are tough societal issues and the only way to deal with them is to talk them out.

Tim Peacock: What would you recommend people read to better understand these dynamics? And do you have any advice for CISOs facing them or for early career folks wondering how these issues will change the security industry?

Heather Adkins: First, read newsletters like Daniel Miessler's Unsupervised Learning, and listen to things like the Google Cloud Security podcast.

Pretend you don't know anything about tech and learn it all over again, because it's changing so quickly. CISOs and teams wanting to try out AI should start with small pilots and proof points. Go slow and don't try to boil the ocean. Pick a few places where you see industry peers being successful and talk to them.

Also, make sure you've got good governance over how it's getting used inside your companies. The workforce, especially the early career workforce, will drive a lot of the innovation because they expect to use these tools.

All this requires an incredible amount of curiosity and critical thinking and really challenging what you're looking at. It's going to be a very interesting few years as we see all this develop.