Rohit (00:00)
Hey, everyone. Welcome to another episode of AI in DevOps podcast. I am Rohit Raveendran, your host. And today I have with me Anshul. He is one of my co-founders at also a longtime friend and colleague. I'll let Anshul introduce himself first.
Anshul (00:15)
Yeah, hi everyone. Thanks for the introduction. So like Rohit said, I'm the co-founder, CTO. So we talk a lot, but not in this setting maybe. So yeah, I have been in the tech industry for the last 18 years. So I'm almost an adult now. And before this, me and Rohit were together in Capillary also. So yeah, that's about me.
Rohit (00:33)
If you're wondering, we are in fact in the same office in different rooms ⁓ talking to each other. So I'll set some context about the topic of discussion for today. So MCPs are THE thing right now. Everybody is writing MCPs. So we thought, let's discuss MCPs -
Model Context Protocol servers, in today's episode.
So, one of the interesting quotes that I heard recently is that in this era of AI, you all need to adopt and then adapt, right? we all have been, at least in facets, we all have been in different capacities for different use cases, we have all been adopting AI in our day-to-day. Many for development, many for debugging, some for coding, some for...
automating some of the product features. So we have gone through the journey. We have tried doing chat completion and copy paste the code or get some information out of chat GPT or whatever you use. And then it has evolved over time. There are assistants. We have all dabbled in multiple of these implementations. And here we are now, MCPs. So Anshul, maybe for the audience,
Maybe we'll start with what is MCP and do you think it will stick?
Anshul (01:37)
Yeah, so even before MCP, right, like what I wanted to talk about is that when this AI came, right, most of the early adopters will be lazy fellows is what I feel, because you got an easier way to do your things. How did MCP come about, right? It's important to understand that, right? So initially you all know the chatGPT came and then it was more like a Alexa and a chat assistant kind of feeling to it, right?
And it was answering all factual things wrong and then we slowly realized that no, this is something different, right? And of course, like Rohit, you said, we used it a lot for coding initial days by copy pasting stuff and whatnot. And then there was this, I don't remember, assistants, the OpenAI assistants, we tried that also. And then this concept of function calling came in, right? And that is where the game changed, I feel, right?
So basically, whatever, so you just think of it as a cognitive load for me. Initially, my cognitive load was to let him describe things, and I will do it for him, and then I will tell him what I have done. And that part is now, again, offloaded from me, right? Like, my life is better, his life is difficult, and so on, right? And this tool calling, we experimented using Swarm and even LangGraph and whatnot.
And that is where this MCP was seeded, I would say. Because anything to become a mass adoption, the first utility should come in. And if you remember, Rohit, we designed a lot of our own coders and whatnot. And they were pretty easy to do.
Rohit (03:05)
With OpenAI Swarm and agents coming about, we had a lot of toy agents here who could probably write some code or interactively debug that used to correct its own mistakes and things like that. And that was fun. And in fact, we also wrote a bunch of stuff on assistants to ease onboarding onto our product, things like that. That was pretty powerful at that point in time. But again, some of the abilities that
I mean, so two things, I think the ability to distribute your tools to the different AI agents became a problem. Otherwise, that I think is one of the things that MCP solves for you.
Anshul (03:40)
Yeah, and that is where I was coming to. For example, the first agent we made in the product, we had to think about the UI, how will it interact with the user, which keys to use, how to limit, and whatnot. And now we have a situation where we can go where user is, and just expose your product or your tool chains which are there to him in his comfortable environment rather than him learning anything else. And that is the, I think, inception point of the MCP,
wherein people realized, OK, I am able to write a agent, but for me to expose it to my users, it's an effort. And also, there is no standardization. Obviously, I have to write a model agnostic code to make it function with other things. But what if everyone had a standard, just like HTTP? I can use Chrome. I can use Firefox and whatnot. And I think we have gone into that domain now, where MCPs are more of connectors or protocols via which
you can expose any niceness to the user.
Rohit (04:38)
Yeah, so just to imagine what it is, it is a protocol to expose your tools and more to any AI agent. So basically, I think in one of our earlier discussions, you described it as decoupling the client and the tools of the server part. And that's a beautiful way to think of it.
Anshul (04:56)
And another thing, if you remember, we had a discussion long while back, like how do we actually design these tools, right? And the one very basic formula we came up with was that let's do it like a human, any task, and just note down all the tools which we are using. And if we can give them as the tool to the AI, then most probably it can also do stuff. And same applies for MCP. So I would say MCP is more like
designing tools itself, although it has other aspects which we will talk about. But I think, yeah, as long as we get that right on how and what needs to be done to accomplish a bunch of tasks, that is what you need to design.
Rohit (05:35)
Right. Yeah. And so on that point, so with MCPs, when I first came to see the blog post announcing MCPs and things like that, one thought that crossed my mind is, hey, what's the big deal? Maybe most of these are going to be some API calls to some applications or tool chains. So why can't you just have a generic function that
exposes the ability to call any open API spec, right? So basically you have an open API spec that defines all of your APIs, all of the contracts that you have in your product to the LLM underneath. And why don't we just get it working, right? So to me, it felt like maybe this is something that could have been replaced with a simple library function where you just pass around the open API spec to it and...
it'll make all of the tools possible available to the LLM. And soon, just to play around to understand what all the fuss was, the first ones that I played around was with the Kubernetes one. And it came with the CRUD operations for literally everything that you can do with Kubernetes APIs. And I did find some utility out of it when my prompts were super awesome. But if I would just say, hey,
can you just find out why my pod is crash-looping? Sometimes it would check the logs, sometimes it would check the events, sometimes both, sometimes only one, right? So I had to have some macros with me that, yeah, this is how I'll prompt. And eventually I figured maybe my CLI hands are faster anyways. Like I have some macros in the CLI that just works for me. But then soon, I think once we started seriously thinking about where it would come in and...
and how we should model the tools that are made available via your MCP server. I think that was very insightful. Maybe you can take through that thought process of how you decide what are the tools to expose.
Anshul (07:24)
So I would rather notch it down a little. I will say that when we imagine LLMs, we imagine them closer to a code. Like this is a machine. This is a code. But in my experience, I feel that they are closer to us. for example, let's say, why did we build APIs? There were APIs. I could have called APIs.
That truly says that these APIs are not human friendly. There is lot of context which is missing in and around the APIs. They are very stateless. You need to figure out which to call next and whatnot. And if you just expose them to me directly as a human, I will struggle to work with it. I will have a hard time doing it. Of course, the LLM has a much more context and much more ability to retain data. But then for them also, I realize that is the same problem.
So that means all the open APIs which are out there are not designed for LLMs, right? And when actually you write the MCP server, your job is not just to translate those APIs into two LLMs, right? You have to work there to make them consumable. It's almost like you need a UX for MCP, right? You need a UX for them to figure out, these are the most natural actions. What if these were UIs, right? And like that, if you were to tell,
and design your APIs, it will be easier for the LLMs to at least and I'm talking state as of today, maybe tomorrow the number of tokens it can consume can expand and whatnot. But as of today, I feel that that designing part is very critical.
Rohit (08:56)
So what would be a mantra for deciphering that particular problem? Even before you start writing your MCP server, I think first you need to define that contract. Do we just relate it to what are the use cases that your end users often have to perform? And just try and model that. Would that be a solid approach?
Anshul (09:16)
I think we should take it as a UI development exercise. This is the new interface, right? So for example, how do you develop a UI? You come up with use cases. You figure out the best design for a user to be able to comprehend it. And that is what we need to do, right? Like we need to come up with a use case first and then figure out what all tools are needed for me to accomplish that tool, what all information from that tool are actually relevant, what all things can be skipped, what instructions can I send along with it, right?
how to make it a little more stateful in that sense. then you come, like writing code is last, like you should first figure out like, it may so happen that you had three API calls, but you need to write one tool for that, right? Because those are like create X, create Y, and then edit Z. And together logically, they are doing a logical function, let's say sending your campaign or whatever, right? I don't know, in user's context. So, and then I will not like,
AI to say, first create user groups, then create a message, and then do this. Instead, I would say, send campaign with these payloads. So that is the translation which happens. And that is something which we are used to, of doing it for UI. So I think we need to do it for MCP as well.
Rohit (10:25)
Right. And while implementing these as well, think, you know, LLMs can course correct that we have seen with the agent tech workflow where it encounters an error and then it tries to correct fix it by its own. And probably while designing the, I mean, we have talked about designing the interface for it, but while implementing the individual tools itself, I feel like there is a little bit more of, you know,
more checks and balances that needs to be added to the functions than usual so that LLMs can be encouraged to course correct or they can figure out by themselves and course correct by using these tools. yeah, so one of the interesting ones was just making sure that any right operations go with some sort of a confirmation, right?
So can you talk a little bit about that piece, Anshul, when we are writing our own MCP servers?
Anshul (11:12)
Yeah. So I think, so there are two things, two ways to imagine about it. Like you give MCP a vanilla tool and then a hundred instructions. You can't do this, you can't do that and whatnot. And more often than not, what we have seen, unless you're writing a specific agent to do specific work, this is not that, right? This is more like a user will decide what to do with this, right? So, so the guardrails should be stricter on the implementation side, right?
And how do you implement those guardrails are by, you know, because I can figure out what he's trying to do and then tell him, give no boss, you can't do it directly like this, right? And maybe have a validation around it. I'll give you a simple example. Like if you remember, we had a use case where the AI had to write a file following a particular schema. And we tried giving him all sorts of instruction. No, we need these keys over here and these keys over there and so on. And it used to still hallucinate, right?
And then what was the solution? We said, OK, boss, whatever he sends, let's just validate everything. It all matters. And then what is the response to it? And the best part about LLM tools is, like when you write functions, you generally have to adhere to a contract saying, OK, this has to be my output format. With LLM, it can be anything. So in one case, let's say if you are violating certain constraints, I will just give an instruction, you're violating this constraint. Maybe this is an alternative. So you are trying to assist him
while he is calling the tools. And even you can say, if it is a successful execution, then also you can say, after doing this, most probably you would like to do this. It's more like your Amazon recommendation. Most users after this do that. And then let LLM take the call on the basis of the context which user has made. And to the exact example which you said,
Especially when you are developing stuff and you know like in our case we were using MCP to create a lot of infrastructure code right and you need certain confirmation like so like I have described what I want But LLM if it goes and starts doing it I need a discussion before that I want to see a preview of what he is trying to do right and there was no such functionality in the current MCP and I'm sure it will come right both client and server has to do it
But we hacked around it. What we said was in our functions, there is a parameter called dry run. And in the dry run mode, what we do is we don't do anything. We just tell this is going to happen. And along with it, we send the instruction saying, while it is happening, you go and confirm the user, show him the information in this format. Once he says yes, call the same function again with dry run false. So that is a nifty hack. These are the things which you will need once you are creating MCP server
for your enterprise tool, right? Because you know it's limitations then.
Rohit (13:48)
So, yeah, so we are having to work around and hopefully MCPs will address this in a native fashion. So, introducing a flag there for dry run did help quite a bit. But isn't still there a possibility that the LLM could just invoke it without the, I mean, with dry run equal to false in the first place. So, did we do some prompt engineering around it? How do you wrap around that?
Anshul (14:11)
That was also very interesting. So in fact, when we did first this time, what LLM used to do was, OK, it used to do dry run. And then self-assert, yeah, this is right. Let me do a proper exertion now. Even though we said that confirm with the user. So I remember that out of frustration, I asked Claude desktop itself, why are you not understanding this? Like if it has said, and the general like,
answers of LLM's, you know, I apologize, I should have done this and not. And then I just asked him, what would have made you understand it better in the first place, right? And there is something funny, I don't think it's a mantra or anything, but then what it suggested was, if the name was better, I would have realized it much better. Then actually we realized the most context which LLM derives from, and it might vary from model to model, but it should not, because it's the function name. Now function names are like this.
You know, call this first and then, confirm it with the user, then only dry run. Like it's huge, but then it's sort of a prompt in the function name rather than in the comment section of the function. And that is working very nicely.
Rohit (15:15)
So the function name trumps the doc string. So maybe the first part of the decision tree is always the function names, whether it makes sense to use it in this particular context. And probably after that is when it looks into the doc string for further instruction.
Anshul (15:30)
Think it as like a human, right? Like if I had two tools, it had good instructions. I would remember them, right? Now, but you are making 20, 100 tools. So what will stick? Will be the names, right? Like rest of the things will be there. But then those are the things which you will never, because first thing, before calling you are anyways reading that. Right.
So that is immediate feedback. There is no guarantee that it is going to read the instructions at that time because it might rely on your old context, right? But at least at that time it is reading it, right? So that is one good realization which we had.
Rohit (16:02)
Right. And so coming to that part of what are the tools that are available. like I mentioned earlier, when I tried the Kubernetes MCP, it had, think, around about 60 tools available to the LLM. And I found that also throws the LLM off quite a bit. And Recently, Claude Desktop has the feature where you can pick and choose what tools to enable and disable.
So that helped me out to a certain extent to actually gain some utility out of it where I had to go about and filter out what I would want it to use. Probably, you know, go filter all the GET methods so that I have some guarantees that, this guy's not going to go and edit something on my cluster. So coming to that, So MCP servers essentially has a contract of retrieving tools, retrieving resources, and retrieving prompts. And this is dynamic.
Could this be something that we could solve to some extent by each time when the MCP server is prompted with the Get Tools API? Would there be a mechanism with which I can restrict or expand dynamically based on the context of the chat?
Anshul (17:06)
I'm 100 % sure this is going to happen natively again. So I don't think it will start with the context of the chat, but it will start with the personas. As Rohit, you can do certain things. As Anshul, I can't do certain things. Because I have access to Jira, you don't. And this kind of things. And once the MCPs goes remote, and that is a discussion for later. But then that is the time when the tools will get aligned according to what you are doing.
And I think how they have designed it currently is they have given you prompts. So prompts, you can think of them as mini agents, right? So in the prompt, you can write, you only interact with these five functions and accomplish this, right? And once you do that, then you are anyways restricting the caller to be within the boundaries, right? So you can do it today also. But the problem is that it becomes a choice for the user. So it's like a drop down and then, OK, I want to.
create a new module, I want to review an existing module, and this is in our context. And then depending on that, there will be a bunch of instructions which will make this process further easier. Or in your case, I want to debug a crashing pod, I want to, you know? And depending on that, you just write the instructions over there. And like you said, these things are API calls, which are giving static tools right now. Obviously, we can write logic over there.
and not let it choose from at all depending on how and when you are calling the MCP. So yeah, that is there. So I agree that this is going to give us lot of leeway in the future.
Rohit (18:35)
So you also mentioned the prompts bit, you can have prompts, you can have it return prompts to control what sort of tools it uses, and probably the behavior to some extent. But when we ended up implementing, we did not, you know, per se use a lot of prompts. In fact, we don't use prompts. We have a technique of giving it a tool to retrieve instructions. Yeah, so can you just recap on what that
reasoning behind that was and what were the practical issues that you ran into?
Anshul (19:01)
Right. So I would say we will use prompts in maybe V2, V3. So initial, whenever you're building the MCP, your job is that I am assisting in human to do certain stuff. And for that also, I need some context. Right? So saying that every time you talk to my MCP, select this prompt, doesn't make sense. It becomes tedious. So instead, what we thought was, how can we make it automatic? The only thing which LLM calls automatic, and we
I think you remember we dabbled with resources also. We thought that maybe, and it is a client implementation, but since the specification says that it need not be automatic, so it is not, right? And I think Cursor doesn't even have it right now. So the main point here was that, okay, tools can be called. And again, we know that the function names are paramount. So in fact, we ended up saying in our MCP, there is a function called First Step Read Instruction.
Yeah, right. So that tells it and obviously there is an elaborate prompt with it saying this is how our enterprise tool works and these are the things which you can do and it is very generic like this is these are the things you can do maybe we read the context from the clients deployment and say whether it is open tofu or terraform or whatnot but all of those things becomes dynamic now so when I say "go and do this"
your LLM will go and fetch the instructions first. And now this is the generic context which he got. And it is templatized to his, you know, in a way we are personalizing also. Like it is personalized to his access controls and other things.
Rohit (20:32)
Yeah, makes sense. Makes sense. when I first saw this implementation, especially around the function naming and using this sort of a hack, right? Like you are telling that it is a first instruction and things like that. So I did try it out with an alternate client. I tried it with MCP CLI
and used it against OpenAI, the 4.0 model. And it behaved exactly the same way. So I don't think it is a client or LLM specific thing. Probably it makes sense. Why do names exist in the first place? Like it is for easier semantic identification of things. So maybe we don't need to feel guilty that it's a hack. Maybe this is the way forward now.
Yeah.
Anshul (21:14)
And I think there will be, I don't know how, what the future holds that maybe tomorrow, so like how everything has evolved over the time. Right now you are saying LLMs, it's like we typing. You have these many tools, these are each tool's instructions, and then you remember them, and then let's work with it.
What if tomorrow the client becomes intelligent? Like the first time he is trying to access a tool, he enriches the context again with the whole payload, right? So there is some variability which may come with the client-side implementation. But like you said, the name part is going to stay. So maybe it is not a hack.
Rohit (21:50)
Yeah, maybe it's not a hack. Let's leave it at that for the time being. So Anshul, so another point that I wanted to discuss was when do you decide that we should, I mean, if I am building a product, maybe a SaaS product, when do I decide this is the right time to go ahead and build an MCP for myself, right? Is it, I mean, in the earlier discussion that we had,
Anshul (21:56)
Yeah.
Rohit (22:15)
we were making analogies to having a UI. So do you think in the near future, just like any application has a web front end, do you think this is going to be a thing that any SaaS-based application or any application for that matter, any tool chain, is going to have an MCP interface? And if so, how good or scary is it?
Because personally I do feel a little bit of skepticism around around this because some of the decision-making honestly is getting offloaded and is that a good thing or is it a brave thing is the right thing to do is this another question?
Anshul (22:50)
So I think technology never evolves for the right. That's the problem, right? But the idea is either MCP or equivalent is going to exist. People will then invest more around security, RBAC, authentication. I think all of those things are not thought through right now. They are in the roadmap for MCP as well. But of course, they have to be there, right? The checks and balances have to be there so that
I think currently we are relying on the role of the user who is running the MCP to make sure that the tool which is being operated on is within the boundaries of his limitations. Maybe there will be a time where you will say I can do this and my assistant can only do this. I don't know, like that can be the future. But the whole point is once the things start getting easy, like which has gotten now, it is there to stay.
People are not going to go back into doing click ops and other things, not click ops. Anything which you have to do by clicks or say you have to talk and do it, you'll prefer that. That's why Alexa has the sales through the roof. What it does for me is just plays the song where I don't have to type and search. That's all. ⁓ Convenience, yeah. Convenience trumps everything. So in my regard, this is there to stay. And yes, everyone should invest in this.
Rohit (23:56)
Yeah, convenience.
Anshul (24:04)
There is one more thing which is there, which I realized. If you remember, we have so many aspects in our product, but we chose the module writing part first. And everyone has to take that decision, like which part of my product should I do the MCP first? And again, I think this came from you only. You should choose the most impactful and the part where there is most resistance, and where you can add the most value.
So for example, for us, there are so many things our platform does, but to enable it, we need good solid modules. And for that, which we are relying on good solid DevOps team. So if we can make their life easier, the platform becomes more awesome. So your moat is your platform, and you are making it awesome by letting people augment it very freely. And for any SaaS tool, instead of just
And I see lot of shallow implementations like how to do this and it will just tell you things to do or just telling you the AI way of generating a message or correcting your grammar and whatnot. But that is okay. That was okay maybe six months back. But now you should think about the value your tool will get on giving this interface. And then you take a call. And I think that's how we did it, if you remember.
Rohit (25:20)
Right. So yeah, so you definitely are going to have to invest in developing some sort of toolset for the AIs to use your product with. Like you said, choosing which parts of your product is easy enough, just identify the highest resistance, highest impact problem statement, and apply to it. Now comes, I mean, in terms of in the larger picture, the timing.
In fact, you already touched upon it that MCP already has in its roadmap to make it truly remotely hostable, make it session aware and things like that. And that was one of my beefs when initially starting off with going with the MCP route, is that should we just wait it out till those features come around? Because...
it would require some re-engineering, right? Even the servers will have some additional contracts to fulfill by the time you have a truly remotely hostable MCP server which is session aware and things like that, right? So yeah, I remember that it it was the urgency of trying to see the impact that it brings about that drove us this way.
But let's say for others who are thinking of writing an MCP. Would you wait these features out of session management and remote MCPs?
Anshul (26:33)
I would say this is your practice run. don't even know what is possible. Right? You don't even know what things can be done. Right? It is pretty early and we decided to try this out internally. If you remember, we never thought of launching it for the customers. We thought let's just use it ourselves. Right? And even that requires some investment. And like you rightly said, like in order for me to make this a proper "product" product, I have to wait it out.
Like if tomorrow Slack hosted just like its API and its MCP server, they'll have to either hack it up right now, which is possible, but it requires effort, or wait it out. And same applies to us. But I would say whether you roll it out to your customers or not, or you select a few and roll it out to them, I would suggest that you do it now. Because it will at least, if nothing else, unlock the potential.
And if you think about it Rohit, it has affected our roadmap also. Just the way we are consuming things, have dished certain features, this is not needed now. AI can do it. Why should I create an API which can call two things and do certain things? So your thinking process changes. So that's the most important reason to invest in MCP right now in whatever nascent form it is.
Rohit (27:46)
Right. yeah, honestly, I also have found post fact or midway through the journey that it has been a very useful exercise in that you get to know a lot of the ways in which MLLMs would work so that you don't have to spend time figuring it out at a later point. Like earlier, you know the better. And in fact, many of the contracts and many of the contracts of my deterministic tooling, deterministic APIs also change to suit
and LLM in a much better fashion, right? So in fact, our CLI tool for scaffolding our Terraform module, many of the things are not LLM friendly, right? Like you have interactive drop downs and things like that are not super LLM friendly right now. So we could preempt many of the pitfalls that could have slowed us down if we had gone down the route of developing how are we where, and then all of a sudden trying to adapt
to introducing an MCP.
Anshul (28:43)
Even the workflow changes, Rohit. Now you tell me that when your team used to write modules earlier versus now, it has changed. Forget about the product, forget about your users. Your internal way of working has changed. You can shed more light on it, but I'm just seeing it from outside and I can see that people are more productive and whatnot.
Rohit (29:03)
Yeah, think productivity is one of the obvious consequences of intervention of AI. But I think this is also set about at least my team on an upskilling journey in a different sense. They are looking at the larger problems that needs to be solved, trying to develop the skill of analyzing the right problems. And implementation of the solution itself or that part is now more trivial to them.
they do tend to use that extra time that they have in their hands to try to think more about the product and figure out what sort of problems to solve and more generic problems to solve or more bigger problems to solve. So that, I think, has been a big positive impact on the team in general by using these MCP servers ourselves internally within the product. And that's one cool thing about our product also that we are the first users of our product anyways, so that helps a lot.
Anshul (29:52)
The off-putting helps. Yeah, and one example which I would like to give you is that imagine we discussed this import feature around one year back also. And the kind of effort which we thought it will take was much different than today because nothing changed. The technology is same, everything is same, but the way we approach that problem is very different now. And your team only did it. I think Unni came up with the CLI so that LLM can do it.
One guy used it to actually create all the modules which are now importable now, brownfield is not a problem. But the whole point is that that was not even our intention while building this MCP and other stuff, right? Yeah.
Rohit (30:31)
Yeah, and in fact, some of the larger audits and cleanups also the MCP server is able to take up now, which is a great bonus for all of us. Yeah, and now coming over to the part where adapting your APIs, like I touched upon it, like our CLI workflow had some
Do you think some of our backend APIs and things like that, maybe is REST the right way to model? People have been saying that GraphQL is more LLM friendly because you can make any sort of queries there and maybe a Graph Query API fits directly into, as an MCP tool to be made available as compared to REST. So are there some development
changes that you anticipate. I did find some examples where people made a compelling point for this, where they replaced their REST APIs with something with GraphQL and LLMs are now able to use those APIs in much better effect without hallucinating or without losing control of what to do next. So is that some lines that you have thought about?
Anshul (31:30)
So there's something which we'll have to try and see, but I'll give you my honest opinion right now. See GraphQL is what? I always felt GraphQL more difficult to be used as humans because you know, it needs more context on what you need, what not, in regards to the tool you are using, right? Like you told me, okay, you don't need 50 fields, you can now request 10, but which 10, right? Now that is the problem statement and that would require really elegant design
for LLM to guess those 10 fields, right? And if anyways you are landing up writing a middleware to transform your APIs into LLM friendly and these are not actually APIs, right? Like this is a middleware, proper middleware. So it doesn't matter whether it is rest or that, but the unit of work has to be smaller for LLMs to innovate. That is for sure. Like you cannot have very complex APIs doing bunch of side effects.
You have to break them down eventually so that you can be creative about it. Right? Today you have thought that this process can be done in this way, but if you make it smaller chunk, then of course, so I would not side with the GraphQL part of it. Like in the same API, have a variability of what I want. Rather, if the same action, I can do it in two parts. Yes, maybe yes. Right? Because it will help me write the middleware in different ways. And in fact,
One of the insights which we got is that, you know, we wrote two functions to check release logs. Right. One for our module writing MCP and one for our other MCP where it is for users. They are very different. They're very, very different. It's the same thing. But because a guy who is writing a module will need to do it in a certain way, he will only work with test projects and whatnot. And he will wait it out. He will figure out what is the problem and then go and edit the code and fix it. That is his workflow.
For the other guy, it is more about what is the misconfiguration. It has nothing to do with module code. So the same APIs can be used in very different ways, depending on the MCP server you are making. So yeah, I feel that middleware is important until it becomes a, if we go totally remote, then you have a server-side code which can just expose these. And those are as good as APIs. You don't even need your APIs until there is a UI.
Rohit (33:39)
Yeah, so I think the argument for the GraphQL side of things, what people suggest is that, you know, one of the things which LLM thrives is converting structure to natural language and natural language to structure. So it seems like, you know, a structure like that of GraphQL where it describes what are the filters querying capabilities that are available.
Anshul (33:41)
opinion.
Rohit (34:04)
Would be more understandable to the LLM I think is the argument over there.
Anshul (34:08)
Maybe
they don't have to write so much context around it, maybe. But like I said, it still needs elegant design and I can only comment once we write our first GraphQL and it is on you, Rohit
Rohit (34:22)
And one of the other aspects that you just mentioned was also about the side effects, right? I think that is another thing that I have found that earlier where you just used to create a 201 OK sort of created sort of response. Maybe you include some additional info in the responses now so that the feedback loop is more coherent and especially so with error messages and
and things like that where the failure points are there. So that the course correction or reinforcement of what an LLM has achieved is present over there. I think this is some of the intuitive things that are already baked into many of our UIs. We click on a form, things get created. There is some sense of what has gotten accomplished by the feedback on screen that you see. The same we need to build for LLMs as well.
Anshul (35:11)
Yeah, and if you think about it, we did that. And we did it exactly like UI. We do a post call, and then we do a get call and send the LLM, this is what you did. Maybe we don't need these two calls. But that aside, the middleware is for this only. The UI also, if you think you filled the form, and then you will get it again, and then say, OK, this has been done. And this is what we did in our middleware.
made that, made a get call, returned, okay, these things have been created, right? But I feel this dry run hack can become more mainstream because I am still simulating it that this is going to happen, but the backend can exactly tell me what is going to happen, right? So that part, I feel every API, if it can have a dry run, that would be like awesome for LLMs. That is, yeah.
Rohit (35:55)
Yeah, so potential side effects and also communicating it back would be great. I mean, I don't think there will be a time when we trust LLM, so we'll definitely need this sort of modeling going forward as well. Exactly
Anshul (36:07)
You remember in capillary, ⁓ we we had this, I think, before even buying something, you need to tell how much points they will get. And that was a big problem statement initially, right? Without doing it, I have to simulate the whole thing and there is a time factor to it and what not. And so it's not a trivial problem to solve. And now trying to solve it from outside the system is furthermore non-trivial. I feel if we start thinking about
these things then I think our APIs will become more LLM friendly
Rohit (36:36)
Right. Yeah, but I think there is some time and thought to be spent there as well. Yeah, of course. Like your existing tool chains, backends. How friendly are they to design a tool that is easy to use for an LLM and safe to use with an LLM? I think these are two different things and both have to be addressed. I think the safety part is something that is often overlooked. I recently found a product where it allowed me to run Terraform plans.
And I know the consequences I wouldn't want to mention publicly. Yeah.
Anshul (37:06)
That's not the ours, by the way.
And one thing Rohit, I remember that the other day and I wanted to talk to you about it anyways. Let's do it here itself. So you remember you were saying that there will be a time that we will need an orchestrator for LLM MCP servers as well, right? Because the odd termination, the session management and figuring out who has the access to what. Right now it's lot of hassle to do it in my laptop. I understand it's early, that's why it's like this.
And I have seen lot of open source solutions out there who are trying to do it in certain ways but I think this should be more standard. What do you feel about it?
Rohit (37:41)
So I think this discussion, think in one way or the other, have had, even in the last episode, so where I was suggesting that maybe all of the cataloging solutions like Backstage could eventually go away, where it would be one product where in natural language you query different data sources. And when I tried to imagine how that would work,
I tend to converse to this point that there should be a central sort of MCP orchestrator, for lack of a word, that would be able to terminate the auth, identify who has access to what, what is the, probably even, you know, there should be a structured way to, I mean, maybe MCP servers should be able to say, these are the contexts that I understand. So when making further calls to retrieve tools and things like that, maybe
that structured context gets sent back to make intelligent decisions for me. So for example, I'm working on a particular application and this particular catalog needs to know which Datadog application to refer, which data from state to refer, or maybe things like that, right? So I think some of these gaps in the modeling would be addressed as the roadmap progresses with MCP, especially around remote hosting, session management, and I would rather have
I mean, I feel like this dynamic ability to return tools and resources would be really cool. And that I think would unlock a lot of potential and replace a lot of full products, probably.
Anshul (39:05)
Yeah, and I feel that this is something until it is done in the mainstream in the open, the best people who can do it are like people like Salesforce and other people who anyways let you, you know, have the whole contract in place. Like this is my model, this is the things you can do on it and whatnot. And they should be able to just say the same things for an MCP, right? And that would be like really something. We need a Salesforce for MCP.
Or any other domain is also fine. But the whole point being, it's not only about orchestrating MCP, you need to have that central source of truth where you have to have the modeling of the organization. It cannot be very generic, like anyone can pick and do it.
Rohit (39:48)
I
should be able to assemble my own set of MCPs and that suit my organization, my users.
Anshul (39:53)
And even relationship between them, like you said, right? Like these two MCPs can be used in this context. I always tend to fall back into our domain, but that is what is natural to me. Let's say I am a developer, I am working on an App X, right? It is being monitored by New Relic. The App Y is being monitored by Datadog. Quite possible, right? So that somebody has to know this, right? When I say fetch me the metric, then somebody has to know that, okay, if he is talking about App X, it has to come from New Relic.
Here is the idea of this app in New Relic. I have to fetch the logs. It's in Loki . This is the key to fetch it from Loki and whatnot. So this context is something which is going to be very important. And LLMs can't keep this long term. There has to be a traditional DB for it.
Rohit (40:35)
So much like the OTel agent sort of provides you distributed tracing by injecting that one common thread throughout. So I think that sort of a propagation is definitely, that will definitely be cool, especially in terms of keeping the context accurate and also threading things together in a logical sequence as well. I think LLMs will definitely excel. think, yeah, so these are things on the roadmap that we have to keep our eyes out
and watch for and again innovate, right? Like probably we'll have to throw away, we'll have to, I think if anybody who is writing an MCP right now, we'll probably have to be willing enough to throw all of that away and, you know, start from scratch when there are new features coming. But it's definitely worth the journey that you should all go through. Right? True, true, true.
Anshul (41:18)
Actually that is valid for any software at this age, right? You should be willing to throw it. I think I also wanted to discuss about this new thing, A2A, which I am not very aware about, maybe next time. But yeah, that is going to be something. It's pretty early, but once we start writing agents on this MCP, that is the next need of the hour, I think.
Rohit (41:18)
Yeah, yeah.
Anshul (41:38)
Any opinions you have?
Rohit (41:40)
No, I think Agent2Agent, we have tried in some miniature formats where at least at that point in time, I think personally I was trying to avoid hallucinations as much as possible. So have agents which are very purpose driven talk to each other. Now I think it will more be of persona driven and interactions because the hallucination problem, people have realised that
it is solved in other fashions and making sure the right tools are available, the right prompts have to be injected. Now, I think, just like you said, you have to realize that agents are closer to us, humans. And we'll have to come up with persona-driven interactions just like that between agents. And for sure, I think once we actually implement one, is when we'll realize all of the nitty gritties and the power of it.
Anshul (42:25)
Actually we have Rohit if you don't realize but I feel that this Agent2Agent sometimes becomes so abstract as a concept that we think it is too much but if you think about it what we did in Swamp is Agent2Agent in a way because it's a function call and this is something again very interesting I want to discuss right now when we write our MCP key MCP server we don't have a LLM there
Yeah. The caller is the LLM. What if I also had LLM? Right. Then my tools can be actually agents. Right. Let's say I enter the key and then I am able to do certain agentic flows and then return it to the other guy. It has become at least one hop agent anyways. Right. Right. And at least that guy is more like a triager and it has seven agents right now. So the same MCP protocol can evolve into at least Agent2Agent for this kind of context.
Rohit (43:15)
Yeah, I think LLM to LLM already exists because there is an OpenAI MCP server or something that you can hook up to claude desktop and then claude will eventually talk to OpenAI to answer a question and then return. Yeah, so something of that sort, right?
Anshul (43:30)
Nice, nice, nice discussion Rohit.
Rohit (43:32)
And yeah, so what have you been hearing from our customers who have, like it is still early alpha days and there are a few select customers who are using it. So just for context for the listeners. So there's an MCP server that helps write very modular Terraform and that works well with the Facets platform. So a few customers have been trying it out as I understand it. What are some early feedback and
What would you have included or excluded or changed if you had known some of these feedback?
Anshul (44:06)
So I think for the listeners, first thing they need to understand is we feel that the Terraform modules as today are more like functions which are taking a bunch of parameters and there are a bunch of side effects which the only way to know about them is to read them. They are more like libraries. The modules which Rohit is talking about is sort of, we have imagined our Terraform modules as Lego blocks, so they have strict contracts that
This module can snap into the other one. And then there are the things which a user can tune and so on. So one of the problems before MCP and like Rohit said is pretty nascent, but I have already started seeing wherever we have given this, the number of organization-specific modules they write has increased a lot. Earlier, if you were happy with that generic Helm chart of deploying service,
Now you will not be because you will you have the capability to say no I just want this. I don't want the other part or those kind of things and this is another shift which I have seen earlier. Let's say even if I had to create a VPC or something I would look for a module which is already doing it, the open source module since my requirements are very defined that these are the things which I want to this MCP servers can directly generate that for you.
Right, and obviously we have the full feedback loop where they can run it and then fix it and whatnot. So I think the early feedback which we are getting is pretty decent and people are saying that it is useful. Time will only tell that whether it will stick around or not, but I can definitely see the speed has increased and the experimentation level has increased a lot for the people who are doing it. Obviously they'll talk about, you know,
things which are wrong and we are constantly improving that and everyone is accepting of that fact. But with all those things, think I would like to continue in this direction and then roll it out to all our customers and the new ones most probably.
Rohit (45:56)
Yeah. And that makes me think also. So you mentioned that now people are experimenting more building on top of facets to their organizational needs. That is something I think is a takeaway in general for SaaS products also. Leave that extra configurability out there. Maybe you expose it only via MCPs and let organizations customize it now. You don't have very big forms or
you know, limited screen real estate to worry about now. You just design your MCPs the right way and probably you can have a very extensible, configurable MCP, sorry, product without compromising on usability.
Anshul (46:32)
And we have seen it firsthand. Like we designed the product for Kubernetes and cloud deployment. People are using it for Databricks. People are using it for GitHub Actions. Because, you know, once any SaaS product will have some moat in it. And there is some modeling element allowed over there. And everyone will give some aspect of it to their users. Like you can model this part of it. If you remember in Capillary, it was like, what attributes does a user has? What attribute does a transaction have. So once that...
flexibility is given to MCP and then a user wants to model something the modeling becomes much more easier and then it can go in directions which you didn't plan for and it's a good place to be in.
Rohit (47:10)
Right.
Yeah, Anshul. So I think that's a good note to end this discussion. Otherwise, it could go on and on and on. Yeah, I know. And thanks for all of your insights, Anshul. And I hope the listeners today can go back and try out writing their MCPs, play around, or use some MCPs, and understand how these LLMs are going to behave when tools are made available to them.
And that's important to know, like the future is going to be this. So I think, I hope everyone plays around and also keeps this angle in mind while designing their products and contracts further. So thanks, folks. Thanks, Anshul. Thanks,
Anshul (47:49)
Thanks for doing this. Thanks everyone.
Rohit (47:51)
So I hope all of you guys enjoyed that awesome discussion about MCPs. Wherever you are listening.
I hope you all enjoyed the great discussion about MCPs and whatever platform.