Facets Cloud (00:00)
Hey, hi everyone. Welcome to another episode of AI in DevOps podcast. Today with us, we have Nathan Hamiel, who is the director of research at Kudelski Security. His focus is on AI security specifically. And I personally think that he's a voice to listen to when it comes to cybersecurity and the AI space. So welcome Hamiel to our show. Thanks for being here.
And maybe a little bit of your background for our audience.
Nathan Hamiel (00:24)
Sure. Well, thank you very much for having me. So my background has mostly been in software security and research. So I've been in the cybersecurity space for about going on 25 years. So kind of more in the early days of cybersecurity. through that, I've mostly been working at service providers. So I haven't had a...
I haven't had one particular customer. I've had many customers throughout the years, and I really enjoy seeing a breadth of challenges. And, I think it kind of keeps me honest because when I hear people saying things, I can look at my own experience and across, you know, my own set of customers and say, Hmm, that just doesn't, that doesn't jive with what I'm seeing. That that's been a majority of, of how I spent my time.
I also speak at conferences. I've been doing that for the past 20 years. Many conferences like, you know, Black hat and DEVCON and many others where I've presented my research and I'm on the Black hat review board, something I've been doing for about the past 15 years and applicable to this conversation. I am the AI machine learning and data science track lead for Black hat
which is, you know, it's a very exciting time right now because you get to review the research and things that people are working on and see where progress is being made, where the challenges are, and how people have found unique ways to address those challenges. That's pretty much me in a nutshell.
Facets Cloud (01:52)
Awesome, awesome, truly in the veteran bracket in the cybersecurity space. And I truly enjoy seeing a bunch of your writing specifically, especially on LinkedIn, your blogs on your personal website as well as your company website. So that's very insightful and it's a refreshing take, especially, you know
Nathan Hamiel (02:09)
Thank you.
Facets Cloud (02:14)
for us product developers, it's easy to get carried away in the moment with all of the generative AI revolution going on. Sometimes going through some of these blogs help us stay grounded, also reflect back a little bit. And I think that's the purpose of those blogs anyways.
Nathan Hamiel (02:29)
Correct. Correct. Yeah. It really is. mean, and it's even too, it's not, it's not just to help keep developers grounded. I find I've been in the security community a long time, and we have a unique tendency to not stay grounded either. And I really feel that security and, know, product basically development, like these should be a collaboration and there needs to be
you know, trade-offs that are made. And that's something that the security community hasn't always done very well. However, I feel like right now with the AI stuff, we've gone in the opposite direction. Like we've seen extremely bad security practices and we're like, yeah, that's just fine because that's just how this stuff works. And I'm like, no, you know, maybe we should have just a little bit of the skepticism back just a little bit. And a lot of my writing either focuses on those types of things or, you know, technical issues or, my personal blog mostly focuses on
you know, AI and how it relates to humanity, so to speak, like how it relates to our brains. And there's a lot of new research coming out, like for years, I've been talking about cognitive atrophy, which is a condition whereby you start to lose your skills. And I've called out specific examples of those types of things. And I said, it would be worse with things like generative AI, because, you know, they're more generalized to a larger problem space. So I'm, I'm just trying to call attention to a wide variety of topics,
to get people thinking about those, those types of things and how they may augment their processes in light of that. Because, you know, these things are just tools, right? They're tools that we can use to either accelerate certain tasks or augment our productivity or something like that. But there's no warning label on them. So for example, things like social media, when social media, well, first of all, they were called social networks back in the day. when, social networks first came on the scene,
Facets Cloud (04:07)
Right
Nathan Hamiel (04:15)
Nobody really thought about negative aspects of them. It was just people sharing photos of their dinner and things like that. But once you started getting like algorithmic timelines and you started getting, you know, people becoming more extreme and you know, there's a lot of negative aspects that came with that. And maybe we would have made some different decisions along the way had we known that.
Facets Cloud (04:35)
Yeah, So yeah, so that's an interesting point. So your profession or your...
your outlook on AI has a security lens to it always. So how has it affected probably your adoption of AI in your day to day, maybe personal, in personal capacity, maybe in professional capacity. So would you say the skepticism has held you off from trying a lot of these new cool toys?
Nathan Hamiel (05:02)
I would say no, because I work in research. It's our product team that has to worry about these things. So I will say that from a research perspective, I mean, we try to use them for, everything. Like right now we're, building something based on small agents. Right. And these are code execution agents by default. So I think we're just trying to understand what the capabilities are first and foremost.
And I think it's important also to like before we would ever propose something for production, propose something to be built, we would go back and look at the architecture, look at how it operates from a security perspective. So that's, it's a little bit different with our team because we also do new product prototyping. So we can't be just security all the time. Like we have to evaluate the benefits and trade-offs.
And then propose those, but a lot of times we're also proposing along with those things. It's like, Hey, you can't just ship this the way it is. It's not secure in this manner. So let's work with our dev team to put the appropriate safeguards in place. And that's once again, where we come back into the production aspect is like making those security guidance recommendations for the products that we're building and,
So, yeah, I mean, I think my, skepticism with, with things like large language models is I don't think, they're going to be AGI, artificial general intelligence. And I don't think they're going to do everything that humans can do. I think they're tools like anything else. Right. And they can be used for certain things. I mean, we've seen where they've been able to do things where, you know, related to coding.
where they've been able to do things as far as like information aggregation and things like that. These are, these are very useful things. where, where you have like influencers and everybody is saying this is going to and so, which is not even a fathomable thing to do. I think you are getting a certain percentage of productivity gains out of these things. But in the long run, when you look at things, and this is just purely coding assistance for the moment, like something that writes code for you, we've kind of seen that.
that instead of getting like a 100x productivity, what we're doing is trading one set of challenges for another. So for example, you may have saved time writing the code, but now you, you may have to spend more time debugging. You may have to spend more time, you know, getting it to work in the context of a larger application. So the work isn't, isn't one for one. You may say, I would rather do this than this, or it takes me less time to do debugging or something than writing all this code.
That's a fair point. And that's where trade-offs come in. You always have to kind of evaluate those trade-offs.
Facets Cloud (07:36)
Yeah, that's an interesting thing that, so us being in the platform engineering space, Facets Cloud, I have been going through some of the DORA reports, Google's DORA reports recently. And one of the things they analyzed is the effect of AI-generated code.
While they did report that personal productivity, people reported there is a bunch of improvement. But the amount of valuable work that was done or valuable features that got generated at the end of it, it had very negligible impact or sometimes even negative. And that I think is a eye opener for me as well. I think, I mean, just getting the code generated is probably not good enough. think that there is a little bit more of building that needs to
happen on top of all of these technologies to actually bring the value out of it for product companies as well.
Nathan Hamiel (08:26)
Yeah. And some of this comes down to vibes. I mean, it's one thing to feel more productive and it's another thing to be more productive. And that's why you need to have good measures and benchmarks for those things. And that's something that's kind of been missing. I mean, the benchmarks that we have, you know, people just write things to the benchmark, but the benchmark may not represent what happens in production or in real life. So it's a, it's a different environment. And I think people should just keep a bit of an open mind. Like,
You know, there are, whenever one of the CEOs from the AI companies talks, remember, remember, like it's in their best interest to convince you that things are going in a certain way. I mean, it's marketing material and you have to count on your own experiences and how it works in your environment. Like I've even seen, experiments that work very well for one organization that don't work in another organization. I mean, these tools, these tools are using, I mean, you need to have.
You know, your data in the right spot. And even at my own company, I joined these meetings and everything. know, everybody wants to do everything with AI. And I'm like, well, the AI part is the easy part. Like doing this with AI is the easy part. There's a bunch of data engineering and a bunch of other things that happen in the backend. That's the hard part. Like getting all of that data and context is the hard part. Doing the AI stuff is the easy part, you know.
And I think that's what people are starting to understand as they try to industrialize some of these into their environments. And some organizations will find it easier to do than others. And it really just depends on how your current environment is set up and the complexity of the problem you're trying to solve. Because as the complexity increases, your performance is going to decrease pretty drastically.
Facets Cloud (10:07)
Right. Yeah, so we do need to tailor and create workflows. Forget about AI. If we had humans who are ready to do any work, any time, with maximum energy all the time, how would you have that workflow set up? So we figure that out first, and then employ AI wherever you don't want to waste human eyes. Yeah.
So I think I was also looking since you are from the cybersecurity background and I was just reading up a bunch of stuff on cybersecurity and looking into our customers experiences as well. I did find a bunch of new potential avenues that AI unlocks in a positive manner in the cybersecurity space as well. For example, code scan with context, auto remediation. So for example, things like Microsoft Security Copilot or
Github's copilot autofix. These have been able to identify vulnerabilities much quicker for you also suggest easy fixes or suggest fixes at least.
This seems to me, at least, there is a bunch of positive impact in speed and scale and precision, especially for product teams when it comes to product teams to keep tabs on the quality, I mean, on the security posture of their application. I did find some of these AI tools actually, superficially, it looks to me that it is making a positive impact. So from your perspective, somebody who is on the field and looking at, I mean,
seeing this data real time. So do you think these things are actually going to change the security culture in companies and probably for the better, I mean hopefully for the better.
Nathan Hamiel (11:38)
Yeah. I mean, I think time will tell. I mean the dream, here is to have everything self-contained. So code is written, it's pushed, it's scanned, contextualized, vulnerabilities are found and then patched automatically. And then it's put into production. mean, that's the Holy grail, right? And I think what we have today is some anecdotes and we have some things on a small scale, but you also have to understand that some
vulnerabilities, right, are, are a lot harder to find. So it's like a SQL injection vulnerability is typically pretty easy to find. mean, you're concatenating a string together and executing it. So things like that are easier to find than something like a business logic flaw or something that's a lot more, or something where you have to understand the flow of inputs through a larger application until it gets to some, some sink that could be vulnerable.
And, and I think that those more complex issues are going to take time to, I mean, because some of these vulnerabilities are found with a traditional vulnerability scanner, you know, so it's really, there's, there's some interesting work out there and we're kind of waiting to see how this plays out. I hope that it brings some additional usage. I mean, really where you're kind of seeing the most benefit is where you have something like a traditional
vulnerability scanner augmented with a large language model. And there are some other cases that aren't like that. So we, built a fuzzer called Fuzzomatic. So fuzz testing is something that developers, a lot of developers don't do. It's, it's a lot more complicated than it seems. I mean, your goal is to try to define fuzz targets and get proper or enough code coverage of your application. And we kind of took something that Google did and
supercharged a little bit, would say. So for Rust, the Rust programming language, we built a fuzzer that was completely automatic. So it would use large language models to kind of go through the application structure, go into metadata and readmes and generate fuzz targets and then fuzz the application automatically. So it was a fun experiment and we got some great results from that.
And I think that there are ways of augmenting traditional tools with something like large language models to really kind of push things forward. Now, whether it turns into a huge sea change remains to be seen because on the flip side, we're seeing a monumental number of old school vulnerabilities coming back into production because of AI development. So now we have like this, this strange trade-off. I kind of made the joke that we
as a security community, we praise organizations like CISA in the United States for pushing memory safe languages to avoid things like remote code execution. Yet at the same time, praising companies who are basically building systems, which are remote code execution as a service. And, it's, it's a, it's a strange time right now, with all of these vulnerabilities kind of surfacing and affecting people when we thought we had mostly not mostly squash them cause they always happen.
but were really reduced drastically.
Facets Cloud (14:48)
So, Nathan, if I were to draw some parallels to the world of Infrastructure management
or infrastructure as what our personal take at our organization has been that we could leverage AI to generate some deterministic systems that create some predictable results out of the modules that they develop. So it's not doing the decision making per se. It is generating something and it goes through a deterministic pipeline that has all the checks and safeguards. And then it brings out something at the other end that is reusable and
more secure than if what a human would have written. So this is a pattern that we really see could benefit. And probably that is, I can relate that with the first testing analogy that you said. So it is the AI is integrating the test cases, but then the fuzzing, the actual testing, presenting the results is happening in a more deterministic fashion, right? So AI could help in that fashion, maybe an organization who, I mean, at least
within an organizational context, somebody could write a scanner that understands that particular project. Maybe if there is an API that doesn't have a required authorization checks, or for the API. Maybe the AI could flag it because it understands the context much better, which probably a traditional system might miss out and vice versa. Like if there are traditional checks that perform much better, the AI could just generate those checks and then get executed from the same pipeline.
Nathan Hamiel (16:18)
Yeah. And I think with any AI related tool, whether you've written the code yourself and you're using the AI tool to generate unit tests or something like that, with any of these things, just kind of have to think almost from a threat modeling perspective, you just have to think, okay, what could go wrong here? And what is the impact of it going wrong? I think having a tool that runs on a developer's system that writes code,
it's pretty easy to understand what could go wrong, right? It could generate vulnerable code and then that vulnerable code, it makes a new production. Same thing with any other type of scanner. I think scanners themselves, as long as they don't have, you know, permissions and they aren't running in sensitive environments, are relatively low risk because they're not performing any actions on their own. They don't have, you know,
access potentially to sensitive information, like what I mean by that is like, encryption keys or, you know, cloud secrets or something like that. So as long as that's kind of isolated, the impact is relatively low. You have to think about how that would be attacked or how that would go wrong. And, and an attacker would have to be able to, get their malicious input into your code, which may or may not even be possible in certain contexts.
So you, once you start thinking in that manner, where you start kind of directing out like what is the, you know, what is the issue here? Then, then it makes it much easier to, to kind of determine those things.
Facets Cloud (17:45)
So I think you also have this blog specifically around that as well, like evaluating the risk against the rewards of adopting AI this way. And I think, I mean, so to quote from your blog, the application of generative AI to a use case in which the capability of the technology is far too in the red. By "in the red" it means that,
there is a potential risk surface out there. And for the use case, people get pushed, yet people push it anyway. This is an observation that you made back in 2024 that for a particular use case, AI is still
Nathan Hamiel (18:11)
Mm-hmm.
Facets Cloud (18:21)
probably not reliable enough or it wouldn't have consistent reliable secure results and people still push it. This is an observation that you made back in 2024. One year since, what do you think right now? Is it still the case or it's gotten worse?
Nathan Hamiel (18:33)
Yep. I think,
I think it's way worse. Yeah. And, and, you, you actually shared that link with me and, I'm very upset with my marketing team because the, the actual diagram isn't showing on the, on the blog. There's a blog that kind of shows when I said in the red, it was referring to the diagram and there was like a plot along like a line in the red. Yeah. So I think it's getting way worse and we have proof of that because our talk at Black hat this year
is an example of all of these vulnerabilities and issues that kind of crop up in these tools. And it's definitely getting far worse because people are just pushing things into production and hoping for the best, is, which is not a great way to do that. And, and really where this, I mean, if we take a step back and we just talk about AI risk, all AI risk is use case specific. Right? So when you think about a use case,
And you think about what could happen I mean, there, there are definitely different risk profiles. So for example, if you have a chatbot on your website, that is a chatbot over frequently asked questions or something like that, that is a relatively low risk use case versus a chatbot that may be dispensing medical advice and prescribing drugs, which they don't do today, but you can see the difference in risk profile and you'd have to take a different set of, precautions there.
Even if you just have a simple chatbot over your own data at an organization, say you've collected a bunch of data together, the chatbot natively has no understanding of permissions. So you may have sent data in there that, that may only be available to a certain group of people. Like let's say finance or something like that. And a regular user can come along and say, what is the salary of whatever? And then that gets, you know,
output to the user. those, those are types of things architecturally that need to be thought about, in that case. And I think when, applying AI to a use case, the first thing you should ask is, "Is this high risk or safety critical?" Because I, I wouldn't want ChatGPT running like air traffic control or, you know,
My doctor just using ChatGPT and says, well that's strange, but I guess, you know, it's ChatGPT. So it must be right. Like we don't want those things to happen. And maybe in some case in the future, that profile changes. But for today, you know, we need to make sure that we're applying these use cases and areas where the trade-offs are beneficial and not, that won't make things worse when they fail.
Facets Cloud (21:01)
Yeah, so I mean, I also had an observation that in the recent flurry of MCP servers, because everyone wants AI to be able to use their product. So in the flurry of MCP servers, I was coming across an article where it mentioned there are several high profile MCP servers that leak information
or provide you unauthorised accesses, things like that and they ended up reporting a bunch of them and the interesting part around that is from the feedback, mean when the security researchers reported it back, I think the number was something around 25 % who acknowledged it and went ahead with a fix. There were
another 30-something percentage who responded that it is an acceptable risk which according to the security researcher it is not and the remaining just did not respond. This kind of shows you the rush that is there to get the MCP out into the market
Nathan Hamiel (21:45)
Yeah.
Facets Cloud (21:56)
while there are lot of rewards potentially for their customers. I think even just think about your customers and have that empathy, you would want to address these security concerns. And it's probably not hard. And these are very similar to your conventional vulnerabilities. When you make an API, you make sure no unauthorized access is there. So you have the same responsibility when you build your MCP.
Nathan Hamiel (22:15)
Yep. This is the exact same thing we found in our research. So we are about to present vulnerabilities that are in production systems at Black Hat this year. And vendors have not responded back to us and we reported vulnerabilities six to nine months ago to them. I mean, that is pretty unacceptable. we have had some people who fixed the problems immediately. I mean, these are devastating issues though. Like in one case,
we were able to exploit a tool and gain access to over a million GitHub repositories. I mean, if we were malicious attackers, we could have caused a large amount of damage. And I think that the, the organization we reported this to, understood immediately the problem because only because I shouldn't say this. I don't know what they were thinking, but I mean, it seems like they understood the risk because of the impact. Like, Hey, you could literally take over our customers', GitHub
Not take over. Let's say we could write malicious data to customer GitHub environment, which would be terrible for their reputation if an attacker had done that. So thankfully we found it and got patched and everything was good, but some, some people just never responded. And, you know, everybody who develops a product, everybody who develops even a library should have a way for security people to report
vulnerabilities. There should be like a security dot or security.md that has an approach or a security.txt on a product page or something like that, that just gives instructions for how to report these things. And so at least you're aware of them. There has been a bunch of AI generated vulnerability information, like people using
Large language models to quote unquote, "find vulnerabilities" that aren't actually vulnerabilities. And this has kind of flooded like bug bounty programs and things like that. Like normally you'll get those types of things if you pay for bugs, because there's an incentive for people to make money by doing that. So they're trying to accelerate that process. And that's caused a bunch of problems, on a triage side, because I mean, is the vulnerability real? it not like somebody has to determine whether it's an actual vulnerability or not.
And then determine what the impact of that fix is going to be on the product. And that's, you know, that, that takes time and it takes a little bit of work.
Facets Cloud (24:26)
And this was another curious thing. So I think you mentioned it earlier as well. The amount of conventional vulnerabilities in AI written code is something that we all probably should be a little bit wary about, not probably a little bit, much more. And I spent a little bit of time thinking and also experimenting a little bit on this side, where I just add
a quality gate that the AI itself is instructed to invoke, that is to pass it through some sort of static analysis, conventional security scanning tools and then quickly corrects itself, right. It is well within the AI's capability to understand the mistake it made or
it can iterate on the product or on the piece of code that it wrote. Do you think this is the right approach? mean, so we are really sure that AI will soon be writing a bunch of code, Probably 90 % by 2028 or something is what Gartner quotes. So if that is the reality, what do you think should the AI's development workflow be? Because I'm sure there has to be deterministic checks to determine what is good to go and what not.
Nathan Hamiel (25:30)
Yeah. I, so on this front, I think the, the vulnerabilities from AI generated code, so in the talks, even going all the way back to 2023, I've talked about these two paths for development related to AI. So it's the kind of like the development path and then the functional path. So one of these is using AI tools in the writing of code for applications. The other one is using AI
integrated into applications. And I think the using AI to develop, you know, as far as like, writing code is something that we've already kind of been addressing for years because humans write, you know, insecure code as well. So we have, you know, things like DevSecOps, we have software security programs, we have secure software development lifecycle. We have tools that are, that can be integrated into development environments to try to catch these things.
Facets Cloud (26:11)
Yeah.
Nathan Hamiel (26:23)
And these are the types of vulnerabilities that are kind of shaking out. Like occasionally you'll get something that's, you know, vulnerable to SQL injection, or you'll get like an older deprecated library that's being referenced, those types of things. So I think we already kind of have a basis for addressing those types of issues, or at least a starting point. Like we may need to step up our game a little bit.
You know, there is no silver bullet tool that will address that and you need to do it in stages. And a lot of times those things are augmented with human oversight. So maybe you have a human doing like a pen test or something. But one of the things that I recommend to people is if you're going to use these tools a lot, like you, kind of understand where the critical code paths are. But, for
you know, for years we worked with web three developers who were managing millions or billions of dollars in assets. And I'm like, if you don't know where your critical code paths are, you are in trouble because you need to understand where these, these critical decisions are being made and ensure that you have multiple eyes on that. So even if AI writes the code, you might have two layers of human peer review over the top of any critical area or critical code path. Now,
this is me as a non-developer saying this to developers because it's not always obvious to you when you're writing your own code, Like what becomes critical and what doesn't become critical? Cause it could be like, well, this is just a function, this isn't critical. And then later on down the line, the output from that ends up going into a critical function or something like that.
Facets Cloud (27:49)
Hit the gun.
Nathan Hamiel (27:54)
So it's not as easy as I'm saying it is, but it's something that should at least be thought about. If there are obvious areas of critical functionality, like those should have multiple eyes on that. And you should have a process in place to try to catch these vulnerabilities before they make it into production for sure.
Facets Cloud (28:13)
Yeah, so I think this is also an urgent need because I think the occasional vulnerabilities are now going to be frequent vulnerabilities because the amount of code and the speed at which things are going to iterate are going to be much quicker. It will be multiple folds quicker. So I think that is one thing that I think developer community has to take seriously now that there will be more features going out. Now the occasional vulnerability is now going to become
Nathan Hamiel (28:28)
Mm-hmm.
Facets Cloud (28:40)
little bit more frequent and we ourselves for example we get AI to generate some infrastructure as code on the fly and it used to make the occasional mistakes in terms of the security posture things like that. What we could do is just employ Checkov which is an open source you know static analysis tool that at least catches most of the
basic mistakes that it would do and I can write my own custom checks as well. Now I plugged it into the AI workflow itself. So now whenever he writes a bit of code, the AI itself runs the checkov, sees that, now hey, this is throwing an exception, so I need to go and correct myself back. So this is something that we have found it fruitful, but yeah, for any general purpose, product development, this does not directly translate. It would probably need a pair of human eyes looking in the critical parts.
Nathan Hamiel (29:28)
Yeah.
And I think it's, just like, any case, are always going to be situations that are easier to catch than others. And that's really what, so for example, like a vulnerability scanner, you know, wouldn't necessarily catch the fact that you have dependency bloat. So maybe the AI coding assistant recommends multiple libraries to kind of do the same thing throughout your code. Now you have this dependency bloat. The other thing
that isn't really a security related, but is more important for developers is that a lot of AI generated code isn't optimized. Right? So you, human may write more optimized code, whereas a, you know, a coding assistant is doing things less efficiently. And when I mean "less efficiently", like maybe it's using more operations than it should to complete a task. So you might be introducing latency or something like that. And I think at some point that's going to become a bit of a problem, but you know,
Not necessarily a security problem, but these are like little things that can kind of creep into the process.
Facets Cloud (30:27)
Yeah, I think the dependency part, I think that there is already this term, "slopsquatting" ⁓ where somebody could create a harmless looking library out there and fool your LLMs that have the browsing capability right now to use it in their projects. And slowly you start injecting vulnerable code into that harmless looking piece of code. So I think that's something
Nathan Hamiel (30:32)
Yeah.
Facets Cloud (30:49)
pretty AI native. Like if I look at it, as you said, most of the vulnerabilities that we encounter nowadays predates AI. It used to exist even before. But techniques like slopsquatting seems like a little bit more susceptible to AI development workflows. And I think there are a few types of those vulnerabilities that will become prominent as well.
Nathan Hamiel (31:11)
Correct. Yeah. With the, with the slopsquatting thing, I haven't really heard that term used very much, but I'm familiar with the concept. You know, this kind of hallucinated dependencies whereby, you know, malicious user finds that, an AI tool hallucinates a dependency and then they go out and register that dependency and, add it. And it has like malicious data in it.
Like that, I mean, it's, hard to tell at this point, just how big of a problem that is. because I mean, a lot of stars would have to align right for an attacker to be able to kind of exploit that and make this attack work. mean, first of all, the attacker has to identify which models hallucinate which packages, cause they're all different. And then it's highly unlikely that one hallucinated package would replicate across multiple different models and even, different model versions.
And then they'd need to go create that module in the hopes that somebody use that hallucinated dependency and something other than a hello world application. That's something that just happens to be building because they're trying to test the model out. So I, on the one hand, it, seems like it would be kind of hard to pull off. I mean, not impossible for sure, but on the flip side, this is a new issue that we didn't have before. It's not like developers just kind of made up. It's like, let me just
make up this module to implement in my application with no knowledge of. Maybe it's happened once in history. I don't know. Maybe they thought they remembered a module and went to import it. I have no idea, but it would become obvious when they tried to like, go find that module. I mean, out of all the potential issues with coding assistance, this one seems fairly low risk.
But, you know, I'd be more concerned about the dependency bloat than the hallucinated dependencies. But once again, yeah, it's a problem that didn't exist before this. So it's definitely something to, kind of keep in mind as you're, as you're developing your code and your product.
Facets Cloud (33:00)
Another one of those AI native risks is also prompt injection. I think you can share some light on your research as well how much portion of it was traditional vulnerabilities that we are already aware of and how much of the AI vulnerabilities are, you know, prompt injection.
Nathan Hamiel (33:05)
correct you.
Yeah.
Yeah, so prompt injection is the vulnerability of generative AI, right? So hopefully everybody's familiar with prompt injection, but prompt injection is an attack that exploits the model's inability to differentiate between system instructions and user input. So if we try to think of an analogy, I don't really like this analogy, but if you think of SQL injection, the vulnerability is caused because there's...
no, there in the concatenated query, there's no differentiation between the command plane and the data plane. So if you just concatenate a query together versus using like a parameterized query where the system knows the difference between the command, there's no way to do that in generative AI. Like as you're having these conversations with these chatbots or agents or whatever, you know, your input ends up becoming part of the context for subsequent prompts.
And the, system has no idea or it doesn't understand the difference between the two and the way prompt injection, and this is, know, you'd have to look at it visually, but this is something brand new because, we're not used to people talking to our applications and convincing them to do something they weren't programmed to do. So this is very new, very applicable. And all of the most devastating attacks
have really included prompt injection in some way. And, I mean, this is a huge problem and this is one of the reasons why it's difficult to, defend It's going to be a problem for quite some time . And if you think about this too, like any time an attacker can get data into your system, they could potentially compromise it.
I mean, you mentioned MCP before. Well, the GitHub MCP vulnerability was an indirect prompt injection attack. Somebody put a prompt injection inside of a pull request and then used the GitHub MCP to iterate over the top of that. It attacked the MCP. MCPs, first of all, have no security built into them whatsoever. And that kind of leads other types of risks, like supply chain risks. If you don't
Facets Cloud (35:22)
Yeah.
Nathan Hamiel (35:27)
If you can't completely trust the MCP provider, then you're probably going to be compromised. An attacker can, basically, you know, do things like remote code execution and things like that. Or even, you know, you could have remote code execution on the MCP provider side. There's, a lot of different attacks, but I would say prompt injection, if we take a step back, like let's say,
including prompt injection, which is a whole type of manipulation attack. With these AI systems, we kind of have like this new execution environment. It's not like an operating system, but it kind of is. You have features and functionality built into large language models that no developer programmed into them, which is why you can put input in base64 format and don't say, decode this base64 information and it does it anyway and it provides you a response.
I mean, there's like almost unlimited number of these undocumented protocols that an attacker can potentially use to encode data, right? And the system is just going to understand and kind of take those actions. It's a very large attack surface and that's why we're seeing so many security issues. And yeah, definitely the most impactful one and the one that, you know, there are some other paradigms happening here too.
I think the AI development space has started to accelerate a lot of things we've known in security are very bad for a long time. Like taking user input and making a request to an LLM and then executing the output, like taking the, the generated code from that and putting it into a Python exec statement and just executing it. Like that's just executing random input from a user, AKA an attacker.
We've collected a bunch of, take like sensitive data, non-sensitive data, internal and external data and collect it all together and then put an LLM over the top of it because we're like, well, we want insights from all of this data, but you shouldn't mix those data contexts because: 1. An attacker can get data into your system. 2. Now an attacker has access to sensitive data that was only internal before. So there's a lot of these security paradigms that are being broken. Things that we've known are bad since the dawn of when
cybersecurity was called information security. And these are coming back to haunt us. And there seems to be no letting up on that at the moment. It seems to be, we were accelerating into it instead of, you know, I mean, we've had Microsoft's recall, which is a system that basically records everything on your computer. It's a nice little Trojan that you install yourself to hope to get insights out of. These are things that even very large companies are pushing and they're, you know, they're violating very basic
security and privacy rules that we've known for a long time that we shouldn't do.
Facets Cloud (38:00)
So where do we look to for fixes for things like prompt injection? So do I look at open AIs of the world or the LLM provider to come up with a generic solution over there? Or do we just be conscious of it and reduce the attack surface if it happens?
Nathan Hamiel (38:19)
have been trying to cure prompt injection since it was a known issue. And there's research out there, people trying to do things. There's they call them LLM firewalls that try to do this through deterministic and yeah they all fail. I, back in 2023, I wrote a blog post
that talked about, you know, addressing prompt injection through architecture. I honestly believe the best way for you to protect against prompt injection is to reduce your attack surface and address it architecturally. Because if you're, if you're kind of aware that the situation exists and you've minimized the ability for an attacker to get data into your system and you've minimized the output, I mean,
the, if you're making a critical decision based on that, you really have to kind of consider your options. you have a chatbot over once again, frequently asked questions and somebody prompt injects is really low risk. But if you have a system whereby there's a critical decision being made. Let's say for instance, you have an LLM system that is looking for spam emails and phishing emails, and it just runs in the background and then
somebody prompt injects your, and it removes them. Like let's say there's a ability to just delete them so that the user never sees them. And then somebody sends somebody an email with a prompt injection that says, delete all email, then their whole inbox gets deleted. You have to look at what the blast radius is from that attack and then do your best to reduce permissions, reduce capabilities and functionalities based on that. So that's, that's my
approach. I call it RRT, refrain, restrict and trap. You know, so the final case when you've kind of, said, okay, I can't refrain from using this, then LLM is best use case. I've restricted as much as possible. Then you go to trap, which is like looking at the inputs and looking at the outputs too, because if somebody is trying to prompt injection, they may be trying to get sensitive data to be exfiltrated out. So you want to have a system that kind of looks at the outputs and says, Hey, this is sensitive. I mean,
Facets Cloud (40:24)
Yeah.
Nathan Hamiel (40:29)
None of this stuff is foolproof, but you don't want to not put, roadblocks in place to stop an attacker from, from doing what they're going to try to do.
Facets Cloud (40:38)
Yeah, so I think for AI developers, be conscious, sandbox your agents, make sure you are planning out for a prompt injection situation as well.
Nathan Hamiel (40:47)
Yeah. Okay so that brings up a very good question or a very good point cause a lot of containerization, technologies are being used with these things. However, comma, if you have sensitive data in your container running along with your agent and an attacker is able to get remote code execution, they can exfiltrate all those secrets. That's something we've done. the other thing to keep in mind is you want to restrict your container.
Which is also another vulnerability that we found. even though people may have containerized an application, they may have not have any sense of data running in there. They may think they're fine, but you have remote code execution and internet access. Then that means you can proxy attacks through your container. Even though, it may be quote unquote "contained", you may still be able to do damage. You can turn your system into an anonymous proxy, and then you have people proxying attacks through your system, which is also not a
good thing to do. I mean, and this stuff is hard, which is why we still have vulnerabilities, right? Like it's not like this stuff is just easy. so I'm sitting here talking about all these problems and I hope I'm not making it sound easy because it's not, this is a very hard problem. And anybody who's been working on addressing them for years will know, just how difficult all of this stuff is. And if you're developing a product or, or if you're even a security professional developer, whatever,
and you are defending anything, this is a very old thing from information security. Like as a defender, you have to address every single potential problem. And if you're an attacker, you'll need to find one instance to be successful where an attacker or a defender needs to defend against everything to be successful. a challenge that's gonna be with us for quite some time, AI or not.
Facets Cloud (42:26)
And so just going back a little bit to the point about a lot of AI products being rushed into market and lot of the security concerns remaining unaddressed. Is this also seeping in probably maybe even originating from some of the architectural decisions of the platforms themselves, not particularly the product that is built on top of it. Say for example, MCP servers, the architecture of
the MCP servers itself. The first version out there is the simplest version and the simplest version is also the one which users could end up building the wrong way. So is there a concerted effort at least from the service provider or probably I mean
to be an AI standards that is emerging soon, much like our web standards. Is there a concerted effort where the security part of it is taken care of? Like for example, in the web world, all the protocols, at least right now are matured a lot in terms of the security protocols that are in by-bind. Is there a concerted effort in that direction?
Nathan Hamiel (43:28)
Yeah. I mean, so it is kind of like what we would say in the United States. It is kind of the wild West right now where people are just kind of doing whatever they want. That's certainly causing problems. And I think, I mean, there's nothing complex or there's nothing even very interesting about MCP. It's just, you know, a simple way of, you know, adding context to LLMs, either through tools or data or something like that. I mean, there is some efforts by
by hyper scalers basically. Google came out recently with A2A, which is agent2agent and has some security built in. I think there is some of this stuff that exhibits vulnerabilities through architecture or protocol. We're seeing that with MCP happen right now. A lot of the vulnerabilities that we've found in products have had architectural issues.
So we may have used an issue in the code to get remote code execution, but what we found is the, all we had access to all of the secrets, which is an architectural problem. Like you shouldn't, host your secrets along with your application, especially if you're, if you're doing anything with code, that would be very bad. So there are all of these different lessons that we can learn. I think it's going to take some time to, to kind of shake out and it's going to depend on the rapidity of adoption. So how
fast are people trying to use these things? I think a lot of people, a lot of enterprises at least are using, you know, AI as a service. So maybe they're using, they're using tools that are all hosted in Azure or, or GC or a Google or any of these other, or AWS or whatever. And they're kind of taking advantage of those tools. Cause when you take a step back and you talk about enterprises, a lot of times they're worried about their data.
So they want to make sure they have data residency. They want to make sure they have contracts in place that, you know, their, their data will only be, you know, used in certain that they don't, that they won't have their data being trained on by AIs and things like that. And usually it's the, hyperscalers like Microsoft and Google, who will give you those contractual obligations. No way of verifying whether they're doing anything or not with your data, but at least you have a contract in place and, and a lot of your data is there already.
But one thing I will say, it's, there's a lot of people, you know, who go that route. And I said, you know, I'll tell them like, look, yes, there is almost a 0 % probability that some employee at Google or Microsoft is going to go into your cloud storage bucket and look at your data. Right. But when you're doing something like, a chat service, right? Like say you have a chatbot that's stealing something, you know,
I don't know if there's, if there's a zero percent chance of that, because you're trying to improve the product. You're using APIs and you know, it's helpful to understand the requests and responses from a chat session to understand, is this working as intended? Is this, you know, going off the rails? Should we do some further training on this to make sure that these things don't happen? So those types of things, right? Like those are definitely,
things that could happen. Now, once again, you got to go back to your contracts and make sure that you know, you're contractually covered. But in a lot of these services, they want to look at that data. Now, once again, if you have somebody having a chat, and now there's sensitive data in there, you also have to make sure that any conversations that are being logged don't have sensitive data in them. It's like once again, we're going back to work.
We're seeing old issues come back to haunt us again in this environment.
Facets Cloud (46:53)
Missing.
Yeah,
Makes sense. Yeah, so I think one other topic that I remember from your blog
says when AI makes mistakes or when LLMs make mistakes, how it's worse than when humans make mistakes, that was pretty thought-provoking and funny in itself.
You also mentioned that even if more accurate or better systems emerge, the damage that is going on right now is going to hold them back or at least reduce the impact of it. So is that something, can you explain a little bit around on that thought process and how you are worried about the mistakes that LLMs are making today?
Nathan Hamiel (47:37)
Correct. Yeah. So that I wrote a blog post on how the hallucinated data of today may end up being the facts of tomorrow. And that's what the post was about, because I was thinking about that just the vast number of mistakes that are made on a daily basis or hallucinations that are happening on a daily basis. A fair amount of these are ending up in a data set somewhere. And it would be really strange
to have some hallucinated data end up in a data set about you. And then you go to some, I, I can't remember what example I gave. I think I gave like the DMV or something. And you go up there and it's like, you don't even own this car. And it's like, I own the car. It's parked outside. Well, I don't know. The system is telling me you don't own it. So you must not own it. And like these types of things can end up being, you know, can end up being problems. And usually when you point these things out, somebody will come along and say,
but humans make mistakes too. That is true. But human mistakes and AI related, generative AI related mistakes are not the same. Like human mistakes are very predictable. Like we know why humans make mistakes. They are tired, right? Maybe they've worked too many hours on the same task. Maybe they're claiming to have skills they don't have, right? So they're trying to do something outside of their ability. These are very predictable things. And also
Another important point is that humans make more mistakes as complexity increases, which makes sense. The more complex a task, the more opportunity there is to make a problem. AIs make mistakes for reasons that are somewhat random, and it applies across all complexity levels. That's why some of these systems will make very silly mistakes like...
saying, you know, George Washington wasn't the first president of the United States. It's like, well, why did it say that? Like there's just no rhyme or reason for that. And some of this comes from, right? Like the stochastic nature of the tools, but I mean, trying to protect against that is much harder than trying to understand, Hey, we have people working 12 hour shifts and they're obviously going to make mistakes at certain points in time. So we need to put, you know, controls in place.
So I think there there's a bit of a mismatch there and we haven't really figured out how to handle just random mistakes at different complexity levels. But yeah, the, the hallucinations thing kind of got me thinking because, you know, with all of the data out there being tainted by AI generated output, I mean, anybody who goes on social media today, we'll see vast numbers of AI generated output, whether it's just
text or image slop or whatever, it's all being, you know, shared across the web. So we have a tainted web now of, of this data. Now there was some papers in the past that kind of dealt with model collapse whereby when an AI learns from other AI generated output, it tends to perform worse on subsequent iterations. And then there was another paper that came along and said that wasn't the case. I don't know, but I think we're starting to see
some of the, some of the instances of that, because otherwise you'd see, you know, AI companies basically generating a bunch of text output and then trying to learn off that text output. And, and there are some cases where you can use synthetic data to kind of train models, but that's mostly for specific use cases. And once again, there's another problem there because what you're trying to, when you use synthetic data is because you have a lack of real data and
to me, when you have a lack of real data and you're really trying to understand to train a model, you want to anticipate edge and corner cases. But it's hard to generate synthetic data for instances you're not aware of, like the edge and corner case you're not aware of. I think there's no, once again, there's no silver bullet here. There's different things we can try and implement to make things better.
And yeah, I mean, I think we're at a point now where things are starting to plateau. Even though if you go on social media and you listen to the influencers and everything, things are moving exponentially. But if you believe your eyes and you believe what you're seeing and you use these tools, you'll know that there is not an exponential improvement happening here. There's certainly sometimes linear improvement, like you're, gaining some advantage or the new iteration of the model works better at one thing than another.
You know, so you may see some improvements, but exponential improvements would be exponential. It's not an exponential, isn't a percent, you know, 10 % and 10 X are both have 10 in them, but they mean completely different things. And I think that that's, that I think that's being a challenge at the moment. So we'll see over the next couple of years, whether those challenges can be addressed. I really think there needs to be a new innovation that kind of push LLMs forward.
I don't think you can scale your way out of this problem. I think you'll, you may get some better performance adding data. Like if you have like real data, you get more data that could potentially help. And, you know, more GPUs may help a bit, but I don't think that's going to get us to a techno utopia or whatever. I don't think we're going to get to super intelligence by doing that. There needs to be a fundamental rethink and some more innovation that happens before that, that comes to fruition.
Facets Cloud (52:46)
Yeah, that actually is grounding in itself, I think that whole statement too. When I reflect back and think probably I did have a mental picture that yeah, things are improving exponentially. Yeah, when you do have a critical look at it and how we could improve it, yeah, it makes sense that we need another breakthrough over here in this.
Nathan Hamiel (53:11)
Yeah. And humans are very bad at evaluating these things because we're very feeling creatures. That's why the word vibes is so, so fitting for this moment. Because whenever a new model is released, right. Immediately you're like blown away by it. You're like, Whoa, that's, I can't believe that happened. But when you start looking at the output, you're like,
Yeah, that's actually not right. Or that's, that's not as good as I thought it was at first glance. Like every time I do an experiment to try and be like, I have to, you know, I have to write a description for this and I always do it for work. never like when I'm, when I'm writing, you know, personally, like when I'm writing blog posts or whatever, I never use generative AI to write because the whole point of writing is to refine your thoughts. Like, especially for my personal blog.
I have so many draft blog posts of things I'm writing because I'm trying to figure out my perspective on it. I'm trying to, to really dig into what I think. And, you just don't get that with a writing assistant, but if it's something for work, you're like, I need a paragraph. I always end up just writing it myself because I'll try to use it and I'll be like, wow, that's actually pretty good. And then I start, I read it again and I'm like, wait a minute, that's not really good at all. Or it's okay, but
It didn't incorporate the points I needed. And then you try to refine the prompt and then you get it to do it. And it's like, well, that might be a little better. Let me try it again. And by then I could have just wrote what I was doing myself. And, yeah, so those are the types of things that I think we need to be kind of mindful of when it comes to advancements, because a lot of people will take advantage of that. Like when a model first launches, you know, you have all of this stuff, all of this attention. Look what this did, look what that did.
And then about two or three days later, you start to see people posting, yeah, it couldn't play this game of chess. It do this. It couldn't do that. All the same things that you had challenges with before. I hope people take away from this too that all these companies claim to be building artificial generative intelligence. Like this is what they're trying to do. They're trying to move towards super intelligence where nobody on earth has jobs. Like that to me,
silly and almost beside the point for those of us who are working on a daily basis. These are, tools that we can use to solve jobs, whether to solve tasks, I should say, whether it becomes AGI or not is almost irrelevant. Like, can you use the tool in your current context to help you with your job? And if the answer is yes, then who cares whether you're on a path to super intelligence? Who cares whether you're on anything else? It doesn't matter. It's kind of a distraction away from
what you have to do on a daily basis.
Facets Cloud (55:50)
Yeah. Yeah. So that's a great note that will probably end our episode there, Nathan. So I think thanks for your time. I I hope the audience take away as much as I do from this episode, especially in terms of product engineers who are trying to build cool new stuff on top of
this AI revolution. I think all of the points that you brought up and the material that you have written which we will link in the podcast description is extremely important so that we build the right way essentially. I think there is a right way to do things in this space and thanks for showing us that in this episode, Nathan.
Nathan Hamiel (56:31)
Thank you very much for having me. It's been a great conversation.
Facets Cloud (56:33)
Thanks Nathan, thanks everyone.