How Caisey Works

Hey everyone, J here from the Caisey dev team. I wanted to take a few minutes to talk about how Caisey works and explain the flow of information through the system. The AI landscape is crazy, and we want to be explicit and upfront about how we work and do things so that you understand the tools you use.

In this post we’ll take a brief walk through the flow of information through Caisey, how decisions are made, what partners we work with and broadly how the system is architected.

Information flow

Information in Caisey starts with the technician sending a request. That request (or prompt) would of course be sent through helpforsupport.com’s dashboard and from there it travels to the routing layer of the application hosted on our compute/storage provider Cloudflare.

When the routing layer receives the prompt request it reaches out to validate that Caisey on the endpoint is alive and waiting to receive a request.

If we successfully connect with the Caisey harness on the endpoint we notify the dashboard UI of a successful connection and simultaneously hand Caisey the technician query along with all the ancillary info that goes along with making an agent work.

Caisey then makes a request back through the routing layer to do an inference call with our US based inference provider. We use open weight models with one of these providers for security, privacy, and data governance reasons. Our current provider is Fireworks, but that may change in the future, as the AI landscape moves extremely fast.

Once the AI model Caisey uses has the prompt it begins reasoning. As it reasons, it generates tokens which stream back to the routing layer and we render on both the client and the dashboard UI.

Now that Caisey has had a chance to think, he can (and usually does) opt to run tools to get more information or explore a hypothesis. A simplified version of a tool looks something like this

<Tool name=”powershell”>

<command>Get-Service -Name Spooler</command>

</Tool>

When a tool gets requested, we check in the routing layer to see if it’s pre-approved for Caisey to run.

If it’s not preapproved (let’s pretend it’s not) we render a permissions prompt window for the technician, and send the command off to a second, independent model for risk analysis.

Why a second model?

We route to not just a different model, but a model with a different architecture and completely different, fully controlled context, to defeat jailbreaking via prompt injection. The secondary model sidesteps both model specific jailbreaking quirks and prompt injection, and any malicious instructions in the main model context.

The secondary model produces both a high-level risk analysis (low, medium, high) and a reasoning around the risk level and what the command does. We also maintain some triggers where certain types of tools/commands will trigger a tool request window with alerting that the command is risky or possibly destructive.

(one of the reasons we use the provider we do is because of the prompt injection protections they implement at the infra layer, you can read about it here)

Once the secondary model does its analysis, we render that in the technician UI to help them make an informed decision about whether Caisey should be allowed to run the tool.

The Technician in our imaginary scenario approves the command for the Org, this fires a mesage to the router which creates an entry in that Org's database of approved commands and hands the tool call off to Caisey on the machine which then runs it and returns results (it may do this for multiple commands at once)

Once the command finishes running on the endpoint the endpoint returns the tool output to the routing layer who hands it back to the LLM and the Technician UI.

Caisey then continues working with this updated data. Caisey then may reason and think and iterate on more tools or even ask the user questions until convinced it should stop either because it needs directions/decisions or it believes the task is complete. In other words; lather, rinse, repeat.

And that’s essentially the whole event loop for Caisey! It’s just automated troubleshooting that can take level 1 techs to level 3 and makes level 3’s twice as fast.

How decisions are made

When building Caisey one of our “north stars” has always been human-in-the-loop. This is a product of our experience working with AI but also particularly our MSP background. While AI has come a long way, it is most certainly not a replacement for intelligent humans and should not be broadly trusted with consequential decisions.

This creates a fairly simple Rubrik for decision making.

1. When a human might pause, we pause
2. Make it easiest to make the right decisions
3. Make walking back permissions simple
4. When in doubt, hand the decision off to a human

Some of the ways in which this shows up are subtle. It takes one key to deny a tool, but two to approve it, for instance. You can easily filter and remove approved permissions too and currently we only do exact tool approvals. Wildcard approvals probably have a place, but as of yet we’re not confident implementing that into Caisey.

We also view our relationship with LLMs as mildly antagonistic. LLMs are after reward signals, not true helpfulness or wisdom so it is our responsibility as the tool maker to treat them as such. Sort of like a Dolphin, super cool, not perfect, and mostly just here for the fish. IYKYK.

We also view ourselves as defenders of the humanity and industry. Many fear getting left behind in a sea of cheap (and often wrong) AI computer advice. We’d like to prevent that from happening and believe that humans, not bots, are the best troubleshooters.

Our Partners

Currently we work with a very limited set of partners. Some of these could change, but we’ll be up front and visible in that at all times. We exist to serve you, the end customer. Dollars, not data, not deals.

Currently email, database, and compute are purchased from Cloudflare.

Currently auth services are purchased from Clerk.

Currently inference is purchased from Fireworks.

And that’s it.

That’s a quick intro to Caisey! If you have any questions, I’d love to hear them.

Hit me up at [j@zelek.tech](mailto:j@zelek.tech)