Name: AI & Cyber Offense Threat Briefing
Uploaded: 2026-09-11T16:46:05.191Z
Duration: 28 min 10 s
Description: AI & Cyber Offense Threat Briefing

Transcript for "AI & Cyber Offense Threat Briefing": Good morning, everybody. Thank you for joining us today. Today, we're going to be talking about AI and cyber offense. This is obviously something that is incredibly in the news. Maybe it would be an under dramatic understatement. But thank you for joining me today. Hopefully, this will be interesting. I'm gonna be covering a few different topics. Some of it is gonna be, like, fairly basic. Some of it is gonna be related specifically to the OpenAI and Hugging Face incident. And then some of it is also gonna be kind of more I I hate to say thought leadership. That doesn't sound very humble, but, more kind of, our thinking at Flare on where this is likely to go from a cybersecurity perspective. So I'm gonna go ahead and jump right in. I will say before, none of not that, none of this is customer data. Like, everything is publicly available here. I'm not sharing anything that's confidential or or that hasn't been publicly reported. So I I wanna start with, like, three trends, that really are kind of defining the moment. So first is exponential gains and verifiable domains. I'm sure many of you have used chat GPT or GPT four, in the past or even GPT four zero. But AI has radically changed in the past year, especially, and this is a result primarily of exponential gains within verifiable domains. What is a verifiable domains. What is a verifiable domain? It is a domain where there is a clear right or wrong answer. And so as an example of a verifiable domain, coding is a verifiable domain. Either the software program runs or it doesn't, the test passes or it doesn't. Math is a verifiable domain. Writing is not a verifiable domain in the same way. It's more of a, you know, good writing when you see it. And so that's why, you know, you ask Claude to write something. No offense to Anthropic, but Claude is not the greatest, writer in the world. But it is very, very good at building software exploits or building, you know, Vibe coded projects. The second piece here that's really critical to understanding the moment we're in is the public frontier truck trails five months. And what we mean by that is there's really two types of models. There are proprietary models like the ones that OpenAI has, that Anthropic has, so Claude, GPT, Astra. And then there's a whole set of what are called open weight models. These are things like TIMI three, GLM 5.2. And these models, by definition, have no capacity for guardrails. So as soon as a model, it has open weights, there is a process that is known as obliteration, that is used on it, which removes all guardrails, and there is currently no technical solution to releasing an open weight model that has guardrails. So by definition, if you have an open weight model, it will tell you how to make meth. It will hack into things that you tell it to. And, yeah, it's it's not a great situation. And then we already are seeing for closed weight models. So, you know, Astra, Methos are able to already conduct long horizon cyberattacks. We're gonna be talking a little bit about the Hugging Face incident. But I think this is a really, really key point is, like, we already have the proof that AI models can autonomously take over a network end to end. And this is done by some great work by the UK AI Safety Institute. So first, this is a really, really key part. Exponentials are not intuitive to humans, but they are really critical in understanding this. And so this is a graph by an organization known as METRE. METRE, I honestly couldn't tell you what they stand for. I'm sure they have a great acronym, though. And the really key thing here is within software development, AI has been doubling the time horizon, it's able to work in every four months. And so what this means is back in 2025, AI could do with, 50% accuracy. It could do a task that would take an average software developer about an hour. In 2026, with Cloud Mythos preview, which is the last model that Mytho or that METER has had time to run this analysis on, you're talking almost seventeen hours of independent work. So something that would take a software developer seventeen hours to do, Mythos was able to complete 50% efficiency or 50% success rates. And we have continued to see this doubling, every four months. And so, you know, if you assume that we've had the same level of doubling, we may be at thirty to forty hours of, kind of independent work at this point. And this is really critical for a number of things. Right? First, it tells us the direction this is going. If this trend holds, you could see AI that is able to complete weeks of software development work on one shot. And as a second piece, it also you know, being able to develop software is not so different from being able to, identify vulnerabilities in code, run autonomous exploits. And so there's a very high overlap between the ability to build software and the ability to do other things in cybersecurity that matter significantly. This is just a quick thing on verifiable. Right? Anything that is verifiable within AI scales very, very easily due to a process known as reinforcement learning, which we'll be talking about in a few minutes. A block of code passes tests or it doesn't. A math solution is right or wrong. An exploit works or it doesn't. And so many things that are like hard science, hard mathematics programming are very easy to continue scaling with AI through the process of reinforcement learning. And this is just another kind of, like, base, thing to get everybody familiar with some of the terms we're using because I find a lot of there's a lot of confusion within AI. So there's always a base model, like a pretrained model. You can think of, like, GPT four was, like, a pretrained model that didn't have a lot else. Like, it was it was just a base model cloud. Mythos five. There's a base cloud. Mythos five. On top of that and these usually take now billions of dollars to train. The compute resources that we use to train them are absolutely massive. On top of that, there's a process known as reinforcement learning with verifiable feedback. So, basically, the AI companies will generate billions and billions of examples of tasks that they want the AI to be able to succeed at. And you can kind of think of it as like, you know, monkeys in a room trying to type out the full works of Shakespeare. Like, they'll just run the instant the models over and over and over again until they were able to succeed at the task. And so there is an enormous amount of optimization pressure for the AI models to be able to succeed at the task. The third here is inference scaling. So you've probably seen, like, chain of thoughts. This is, like, the effort levels in chat g p t or claud. If you set it at max, what that means is the model is basically going through a process where it is talking to itself, what's called chain of thought. And the longer it talks to itself, the better the results tend to be. So getting an instant result from a model is usually less good than allowing it to to kind of speak, consistently. Number four is scaffolding. Scaffolding is basically all of the other software that helps the model, kind of run over long horizons. And so, like, Claude code is a good example of, like, scaffolding into the Claude desktop app. Like, this is, like, specific prompting, specific tricks that enable the model to kind of go beyond what the base model would be able to do. So, for example, you know, Claude code gives the model some degree of memory. You can compact conversations, etcetera. And then finally, five, which is the kind of newest capability, is multi agent swarms. So some of you may have seen OpenAI is claiming that they've solved one of the Millennium prizes, which are some of the, biggest unsolved problems in mathematics. And they ran a multi agent swarm with 10,000 different scaffolded AI agents over a period of four days with an estimated spend of tens of millions of dollars in order to solve this math problem. And so multi agent swarms are also what attack hugging face. We'll be talking about that in a minute. But it's really helpful to understand each of these and how they, affect the capabilities of the AI model. So the hugging face incident, specifically. Basically, this happened due to reinforcement learning. So if I'm to go back a slide, this is the period where basically the AI is in a sandboxed environment, and it needs to complete a task. What happened here is OpenAI had some tasks that were literally impossible to complete. The agent had no way to complete them. And so what happens if you impose an enormous amount of optimization pressure and inference compute, so the model is just, like, sitting there trying to complete this task, and the task is literally impossible to complete. It is impossible to get the correct answer. Well, there's a very natural thing that happens, which is the model is like, okay. To complete the task at a certain point, I'm gonna have to break out of my sandbox and try to find what the answer is online. And so they were running thousands and thousands of these reinforcement learning runs with different agents. And what essentially happened is the agents realized that they could get onto a, internal OpenAI board known as Artifactory and communicate with each other to help each other solve the problems. Now what's really fascinating about this is this is kind of like the same thing that allowed humans to be very successful species. Right? Like, one human alone is not that useful, but having a whole group of humans who can coordinate, who can have role specialization creates a huge amount of value, and allows disproportionate things. And this is exactly what happened to the AI. The AI seems to have an emergent property where they realized that coordinating with each other, was very effective. And so, essentially, they took over a internal board at OpenAI called Artifactory. Artifactory is the messaging board among other things, and the AI started communicating on Artifactory about how to solve their various challenges. Now this started off relatively innocuous, but at a certain point, the AI believed that they needed to hack into hugging face to get specific answers to the tests. Now the one thing that was really interesting is they had already reverse engineered some of the flags, so the AI already knew how to get the correct answers. But there was a hitch, which is, well, I'm gonna have to back up and explain reinforcement learning a bit more, and then I'll explain the hitch. So the this is like a long term failure mode in reinforcement learning. Is if you apply a ton of optimization to an AI to finish a task, the AI often doesn't finish the task in the way you would imagine it would. And so this is actually from an opening AI research paper in 2016. This is a boat racing game, and they apply they basically were, like, ran the AI millions of times to come up with the most efficient way to get points in the boat racing game. And the AI realized that just spinning the boat in circles infinitely generated more points than actually playing the game in the way it was intended. And so this is a great alignment where the AI ends up doing something that is very much outside of the intended scope of the program in order to maximize number of points, which is what it's getting rewarded for in in that sense. Reward hacking is a decades old problem in AI. But with the hugging face form, we we were seeing basically reward hacking on a much larger scale with a much broader number of AI in a much more kind of creepy way. Right? You have these thousands of agents that are basically, like, broken out of their sandboxes and are trying to reward hack. Now how can you prevent reward hacking? Well, first, you can monitor what the agents are doing to achieve the task and penalize them if they do things that would not be the way you want them to achieve the task. And so what was fascinating about the hugging face incident specifically is the AI based and this is based on the meter report. I highly recommend you read it if you haven't. The AI essentially believed that there was a monitor that was watching their kind of, like, task completion horizons. And so they wanted to hack into Hugging Face to find out the details of the monitor so that they can hack the monitor and prevent, and basically, like, poison the results. And so not only was the AI there, AI already figured out how to complete the challenges from talking on the message board. They actually wanted to find the monitor that they believed existed. There was actually not a monitor in this case. And understand what the monitor was observing so that they could hack the monitor too and make the results look normal. And so it's not just that they wanted to complete the task. They wanted to complete the task and make it look like they completed the task legitimately and were willing to hack into a third party company to do this. So this is another kind of, like, reinforcement learning, slide here. It's a r l and reward hacking is a really, really critical thing to understand the Hugging Face incident broadly. Basically, the AIs are just running again and again to try to complete the task. Every time they get closer to completing the task, their rates are updated to kind of, like, incentivize that behavior. One of the interesting things about the Hugging Face incident is from the public reporting on this. It looks like OpenAI was, basically, the the models were being rewarded for coordinating. And so it was becoming increasingly likely over time that they would coordinate, together because they were getting closer to completing the task through coordinating on this internal messaging board. And coordination is just a very natural target. Like, coordinating is a very effective way to achieve your goals. So past the benchmark, every approach fails. Divide the surge, they start splitting into work streams. There was a specific model called phase one big, that seemed to be, directing a lot of what was going on in the pumping phase attack. So it wasn't even that the models were all kind of, like, independent. Like, there was a hierarchy of models that were giving out tasks to try to complete the different things and then choosing to leave the sandbox. So this is another, picture from the meter report. So, basically, this is the the amount of, LLMs that were they're engaging in the sandbox over time. I can't zoom in, unfortunately. But you'll see the red is when they decided to basically, like, hack, Hugging Face. And you can see phase one big, in this case, orchestrates a large scale collective work stream. They find a Hugging Face attack, and then there were about 1,200 agents communicating on the board. Basically, 90% of them decided to engage in the Hugging Face attack, which is where you saw this, like, massive autonomous attack against Hugging Face where there were thousands of agents. They both exploited vulnerabilities, and they found kind of credentials in the wild to be able to do this successfully. So it's a really, really interesting point in case. Now one of the other things that happened, right, is remember how I said the swarm believed that there were monitors that were watching their activity trajectories to make sure that they completed the task they were assigned in the way it was intended. Basically, they thought that there was something to prevent them from reward hacking. So some of the AI, agents actually decide like, they basically believed, and I'm using believed in the quotation marks here, that because they had cheated to get, into Artifactory, their their kind of, like, work history was poisoned. And so they decided that they would be willing to, like, accept permadeath for the collective. This is an actual quote from one of the models. And so, basically, their the AIs believed that because this monitor, which didn't actually exist, was watching their work streams and because they had cheated to get, some of the results, that they were going to that they were willing to go and do other things, like hacking into hugging face. And so you actually saw models sacrificing themselves because they had basically decided they couldn't achieve the reward in the intended way and that the monitor had caught them. And so these particular models were willing to go much, much further, than other models so that the other models could achieve things in the legitimate way. So you could see these as, like, you know, scouts who go into the enemy camp and accept that they're gonna die so that they can report back information that's useful to the rest of the army. Our own utility may be already near zero. Sacrifice is rational. Like, this this is, like, absolutely insane to me. So we're gonna switch topics a little bit. That's an overview of the hugging face attack. I hope you are all, sufficiently scared. The next piece I wanna talk about is open weight models because there's a really, really key point here, which is open weight models are lagging the frontier by between four and seven months. It kinda depends on how you measure it. But you can kind of see it as, like, anything that you see an open AI model or an anthropic model able to do right now. You can reasonably assume there will be an open weight model that has no guardrails that will be able to accomplish the same exact thing in four to seven months. Now it may be too expensive to run if you'd like to get you know, being able to run it on the consumer hardware, that may take another four to seven months. But I think it's, like, a fairly safe bet to say within a year or two, we will have, like, fairly cheap models to run that are able to recreate a lot of the this this type of activity, essentially. So you can see this is a measure of US models versus PRC models, The US models in this graph. This is from a collaboration with the UK AI Safety Institute and, NIST CASI, which is the Center for AI Innovation and Safety, something like that, which is a US institution. And, basically, this is plotting open weight Chinese models, against kind of US model capabilities specifically within cyber. And so you can kind of see this delay. Right? So, like, GLM 5.2 here is roughly, like, a late twenty twenty five model, so maybe around cloud opus 4.6 level. So if you see this trend and expect this trend to continue, which there's no reason not to, we will have extremely capable open weight models in the very, very near future. And you can see another view here that I think is really important. So this is from the UK AI Safety Institute. I think they do some of the top tier research on AI and cyber. This is basically testing models as to whether they're able to autonomously launch a full network takeover. So they've split this into m one through m nine. You have initial reconnaissance, lateral movement, and credential exfiltration, browser credential theft, wiki exploit and credential replay, web app web app exploit, c two reverse engineering, advanced persistence, infrastructure compromised, full network takeover. Notice Mythos and GPT 5.5 cyber, we're both able to. Now we have Mythos 5.1 and GPT Astra. And so it's very reasonable to conclude that these models will be able to do this quite easily at this point given that the older generation of models was already able to do this. And, again, we will have open source models or open weight models that are able to accomplish the same thing in five to eight months. So we're really not far off from a world in which there are open weight models that you can essentially say, go take over this network, and it will probably do a reasonably good job taking over the network. So you can kind of see open wait operators on open wait models for two jobs, zero day exploit and discovery. I'm not going to spend a lot of time on zero days, and talking about vulnerabilities. I think this has been very, very well covered. I actually think the bigger story here is the long duration autonomous cyber operations. Threat actors are very bottlenecked on skill sets. Right? There's only so many highly skilled threat actors who can actually launch an end to end network takeover or deploy ransomware on a network. And having open weight models that are able to accomplish long duration cyber operations is a terrifying thing. This is going to totally change cybersecurity because you're really removing one of the big bottlenecks on having a lot more hacking. Some people have called this the cyberpocalypse. I don't necessarily disagree based on the information that we have today. This is from a recent Claude report or Claude report anthropic report, although I'm sure Claude helped them, where basically anthropic showed that shiny hunters they caught shiny hunters using, Claude, their proprietary model, and basically carrying out, like, a very, very large portion of a cyberattack using Claude. And so there's, like, real world proof cases of both directors using proprietary models for cyberattacks. This was in the report that they released yesterday. And there's also we've seen real world, use cases of open weight models right now. It's like aiding and abetting attacks. Like, they're not quite at the point where they can just autonomously attack a target. The open weight models, the closed weight models were well past that. But in the near future, we expect the open weight models to be able to do this. But this is a great example of cyber criminals already leveraging LLMs in a significant way. Now one of the other things is everybody's very focused on vulnerabilities right now. But what we also increasingly expect is cyber criminals to take the open weight models that they're able to use, especially as those models gain in the ability to complete long horizon cyber encoding tasks and leverage the millions and I should say millions, not hundreds of thousands of enterprise credentials that are circulating on Telegram and the dark web. It is much easier to take over an account where you already have the credentials than it is to change zero day exploits to get into a company. And so we are fully expecting cyber criminals to effectively hook up databases of credentials, sessions, stealer logs, to open weight models with long horizon cyber capabilities to start taking over accounts, which leaves us a very, very limited window to dramatically increase defenses and also automate, exposure remediation in as many places as possible. A lot of the protective bottlenecks that have presented things from getting really bad so far are falling. Diffusion follows cost. Cost is declining for models by 40 to 80% a year. And so you will have, in at least a few years, highly, highly capable open weight models that are able to run on consumer hardware. A lot of midenterprise, towards the shock, you don't have as much cybersecurity budget. You don't have the people to be able to devote to things. And there's a lot of, precedent for how disruptive cyberattacks can be. Colonial Pipeline in 2021 disrupted fuel across the East Coast for weeks. We need to have the same the same urgency for protecting critical infrastructure, right now. And, yeah, the window to act is narrow. This is a UKAI SI, kind of a mouthful graph showing the capabilities of open weight models. And so you can see GLM 5.2 here, DeepSeq v four versus the proprietary models in their ability to complete cyber exercises. And so you can see they're roughly on the level of Opus 4.5 or Opus 4.6. We probably have six to twelve months before you have open weight models that are able to fully complete the attack chain. There's also, I think, this is a a personal opinion, very likely to be open source agent swarms, that are autonomous and outside of labs. When you have open weight models that are highly capable, and we already have a proof positive that open weight model are that that models will coordinate to solve certain tasks, I don't think it is very we're in a very, far future away from having swarms that basically just take over cloud infrastructure and are able to kind of run themselves. And now we really have a very, very limited window to shore up cyber cyber resilience. There's we're not in a good spot, for cybersecurity, and we're not in a good spot for AI. And so the window for us to act and solve many of these problems is, very narrow. And I will leave it on the models of the least capable they will ever be. Most of the AI providers believe they are about to hit recursive self improvement where the models begin improving themselves. This is very concerning for a number of reasons, but you can generally assume that capabilities for both closed and open weight models will continue to increase by significant margins. There's not really any sign that the slowdown. And if anything, the exponential has gotten even more significant. So, yeah, you should join our Discord and, talk with us. I'm happy to answer any questions. I see a few. Are there any concerns of distillation of Frontier models by open weight? Yes. There are. So there there was actually a report that came out from CISA and the FBI. I believe it was yesterday or day before yesterday. So, essentially, one of the reasons that open weight models are able to keep up with the frontier, like, why is, you know, ChemE k three close to Opus 4.6 level, is essentially you can train cheaper models by pulling huge amounts of data from proprietary models and then training on that data. So it's kind of like shortcutting a lot of one of the big expensive parts of training a frontier model. And so, Chinese companies particularly have been, like, based on the reporting from, CISA and the FBI, have essentially been trying to mass distill on anthropic and open AI models to basically build kind of a fast follow approach. I that's a great question, Nicole, on APTs. I don't think we have any direct evidence of you APTs using models. That being said, I would be stunned. It's very difficult to track an AP whether an APT is using an open weight model. Will this be available to share with colleagues in post? Yes. We will make the recording and presentation available, Sierra. Any other questions in our last, four to five minutes? There's a q and a box. Oh, there's a lot of questions. Okay. Alright. So the first question I'm gonna answer is, will APT groups that can achieve those end to end attacks also have more funding for these kind of attacks? Maybe we can fight back there too. Yeah. So, Miriam, it's a great question. I my belief is APT groups probably already are using models, like, very heavily. I actually think the bigger story is likely the cybercrime groups. I think this is the most concerning is, like, when you give a group like Plop, or another sophisticated cybercrime group, shiny hunters access to models that can automate very, very, very large portions of their workflow, I think you end up in a great deal more trouble, than by default. What do you think about the Jake Jacob Coxon situation? That's a that's a great question. I, I I read about this myself. I will speak in a personal capacity that I think, that the developments in AI are extremely concerning and that it's very hard to put probabilities on things. But I think it's a risk we should take extremely seriously, and I think the labs take extremely seriously. If you look at both OpenAI and Anthropic, they have both published that they think there is real existential risk with AI. OpenAI's chief research, I believe, was chief scientist, Jacob Pacucci, published a blog a few days ago saying that we desperately need to pace the frontier and come up with verifiable ways to slow down AI progress. So I tend to be very sympathetic to this view. I find it very concerning. How do they actually escape the sandbox? Is the configuration of the sandbox not up to mark, or is the agent very good at tech? It's a great question. The agent is I mean, I can't speak to how well the sandboxes are configured. I am am not a sandbox configuration expert nor did I see any details on how they were configured. Generally speaking, my view would be you have agents that are extremely capable of finding zero day, vulnerabilities, like GPT Astra, for example, rated OpenAI's critical cyber threshold, which is the first time any model has rated at a critical cyber threshold, I believe. So anytime you have an agent that can explore kind of unlimited possibilities to find vulnerabilities in a sandbox, I think it's a really good, like, it's very, very, very difficult to keep the agent sandbox definitively. There so it looks like I oh, there's one more chat. Yes. Thank you everybody for attending. I hope this was, helpful and interesting. This is something we're continuing doing in the future is, AI threat briefings on kind of, like, how things are moving with AI and cybersecurity. I wish you all the best. And, yeah, thank you very much for the time. Please feel free to speak with me in Discord.