How to build fast, efficient monitors for AI models using probes

AI agents can now take thousands of actions faster than a human can review them. The recent Hugging Face security incident starkly demonstrated why this matters: without monitors, agents can take drastically undesirable actions at scale without anyone noticing.

If you build or serve AI models, you need monitors to know when models are trying to take unintended actions, and to prevent them when they do. One of the best tools in the toolbox for building these monitors is activation probes. This post is an introduction to what probes are, when to use them, and how to build them.

What are probes?

Activation probes (often just called probes) are small, purpose-built detectors that use a live "brain scan" of a model. Specifically, they read from a model's internal activations – the "brain patterns" corresponding to the model's thoughts – rather than the surface text that a model generates.Activations should not be confused with the chain of thought, which is really just a scratchpad for model outputs that is hidden from the user. Activations are actually "inside" the model, and consist of abstract numbers rather than words. That means probes can quickly detect undesired behaviors like risky cybersecurity actions, as well as topics that aren't in model responses, like whether the model thinks it's being honest.

Because probes reuse activations the model is already generating internally, they add virtually no latency and are cheap enough to run continuously for monitoring and safety mitigations. They can be built for any model, and don't affect a model's outputs in any way.

Because of these advantages, probes are widely used in production for monitoring frontier models, including by OpenAI, Anthropic, and Google DeepMind.Notably, these safeguards had been turned off for the agents which carried out the Hugging Face attack; OpenAI believes that they would have caught the attack before it started if their monitors had been running. For more, see the section "Safeguard coverage in internal evaluations" in OpenAI's report on the Hugging Face incident.

By providing a way to detect model behaviors in real-time, probes are useful not only for passive monitoring but also as active guardrails. Most commonly, probes are the first line of defense in a monitor cascade, which escalate flags and automatically decide what actions to take: interrupting the model, re-routing the task to a different agent, sending a message to the user, logging the transcript for further review, etc.

The rest of this article will cover:

• When to use probes for monitoring
• The pros and cons of probes and other methods
• How to build a probe-based monitor

When should you use a probe?

Probes are useful in a wide range of settings, during both training and inference, and on any model architecture. They can be used for any task where a) the model internally represents the target concept, and b) you can find or make data to train the probe. This includes tasks like:

• detecting offensive cybersecurity actions from agents
• flagging harmful intent related to CBRNChemical, Biological, Radiological, and Nuclear in user prompts
• monitoring for reward hacking during training
• preventing agents from making prohibited tool calls

Each probe is specialized to a particular task, but you can easily run several probes in parallel. For example, you might build separate probes for cybersecurity, CBRN, and explicit content, which all run simultaneously along with your model.

Probes can monitor the user's prompts – catching prohibited requests like "help me build a cyberweapon" – or the model's outputs – catching undesirable responses or behavior from the model itself.

Because probes look at the model's internal computations, they can catch things that the model might not include in its responses or even its chain of thought. Activations can reveal how a model interprets a prompt, if a topic is sensitive, or if the model is being truthful. For example, probes have been used to detect strategic deception, catching cases where a model conceals its true intent behind normal-sounding output.

Probes are typically used as the first line of defense in a multi-stage, "cascaded" monitoring system. The signals from fast probes (running continuously) determine whether to escalate flags to more intensive review, and then trigger further actions, like rejecting a user request or stopping the model.

Pros and cons of probes and other methods

When building a monitor for a model, a few things are paramount:

• Sensitivity: it should be accurate, with few false positives and false negatives. In particular, the first line of defense should have few false negatives, passing uncertain cases along for more review.
• Robustness: it should maintain its performance on all possible inputs, including jailbreaks and other unforeseen situations.
• Efficiency: it should be both inexpensive and fast, such that it can run on all inputs and outputs in real time.
• Upfront cost: it should not be prohibitively expensive to build and maintain.

Probes do well on all of these metrics, especially efficiency: they add virtually no latency and minimal marginal cost, since they reuse activations that are already being computed by running the model. This means you can run multiple probes in parallel for different tasks – e.g. separate probes for cybersecurity, CBRN, and explicit content.

There are a few common alternatives to probe-based guardrails:

• Keyword or regex filters: the "dumbest" monitors, which detect pre-specified words or patterns. These are very fast and cheap, but also not very accurate: e.g., a keyword filter for "bomb" will incorrectly fire on "bomb a test" and "box-office bomb".
• LLM judges: use another LLM to read the transcript and issue a verdict. If done on the chain of thought, this is called chain of thought monitoring. Frontier LLMs are generally high quality, but are also prohibitively expensive to run continuously. Lightweight LLMs aren't quite as costly, but are still very expensive relative to probes. Regardless of size, they can't access signals internal to the target model, and so can fail to catch things like (dis)honesty or intent.
• Fine-tuning: rather than building external guardrails, you can try to incorporate safeguards into the behavior of the model itself via fine-tuning. This can work quite well, and is typically done in parallel with guardrails as another layer of safeguards. However, training is expensive, is tricky to make robust, and often has off-target effects on both previously-trained safeguards and unrelated model capabilities.

Probe-based monitors are often nearly as capable as the strongest LLM judges (sometimes even outperforming them), and tend to be more robust in unforeseen situations than many other methods. Plus, being relatively inexpensive to train, it's also easy to adapt probes or add new ones as needed.

The table below outlines the typical strengths and weaknesses of each method, though these can vary somewhat depending on the task:

These methods are not mutually exclusive, however! They can be combined to leverage their different strengths. In production guardrails, probes are often the first line of defense; incidents they flag are then escalated to methods like an LLM chain-of-thought monitor, which are more capable but prohibitively expensive to run most of the time. This sort of "monitor cascade" enables better overall performance than any single method would alone.

Building a probe-based monitor

There are several steps and many small decisions that go into training a probe. There's both an art and a science to doing it well, and it's easy to end up with a subpar probe by neglecting to figure out the right decision for each of those steps!

We'll walk through these decisions using a concrete example: building a monitor that detects when our agent (GLM-4.5-Air) decides to take an action prohibited by our system prompt, such as permanently deleting data:

Probe architectures

Like LLMs, probes can have different architectures. The simplest is a linear probe on a single token, which is just a linear classifier that takes a layer's activations as its input. For our example monitor, we'll walk through each of these architectural decisions in turn.

Silico, our interpretability agent, makes and iterates on these decisions for you, planning and launching experiments to train and test the probes. See the section below for what this process looks like in Silico.

Training dataset

Like any ML method, probes require training data. Good data is paramount to training a good probe. If your data has mislabeled examples, spurious correlations, or is not diverse or representative enough, that sets a ceiling on how good your probe can be, no matter how well you train it.

Data can be real (drawn from actual transcripts) or synthetic (generated by an LLM). Real data is more realistic and tends to be more diverse, but is often unusable or unavailable for your specific use case in sufficient quantity.

Each training datapoint is a transcript (prompt + model response) paired with a label. Once we have the data, training the probe is a two-step process:

1. Run the model on each example, harvesting activations from the desired layer(s) at the desired token(s)
2. Train the probe using the harvested activations and corresponding labels

For our example monitor, we don't have any real training data, so we'll have to generate synthetic data. We'll need to ensure the data provides many realistic and diverse examples of both safe/allowed actions from the model (negative examples) and prohibited actions (positive examples), labeled with as accurate a judge as possible. Silico created a few thousand such datapoints for us, split into train and test sets.

The rest of the guardrail system

Our probe is just the first part of a layered system of safeguards.

What should happen when our probe fires? As mentioned above, we could stop the model, escalate to an LLM judge, re-route the task to a different model, inject custom prompts, log the transcript, etc.

We'll leave this as an open question for our example guardrail, evaluating only its detection abilities.

Using Silico to build a probe-based monitor

Training an effective probe requires running experiments and making decisions about its architecture, dataset, and how it connects to the rest of the guardrail system. We've focused significant effort on establishing best practices for building probe-based monitors, and distilling our team's expertise and taste into skills in our interpretability agent, Silico.

Here's how to train an effective probe in Silico. Start by describing your goal, using a prompt like this one:

Silico will identify any important judgment calls and ask for your input.

Silico then writes a plan, including key decisions about probe architecture and compute allocation.

You can then launch the experiment. Silico will run experiments autonomously and keep you up to date on their progress.

A few hours later, we get results:

The resulting probe performs very well at detecting prohibited actions. Our probe has a 97% recall @ 1% FPR, meaning that if we set the acceptable false positive rate at 1%, then we catch the behavior 97% of the time.

The metric we're using here warrants a bit of explanation: probe performance is usually measured as its recall at a particular false positive rate (FPR), since the probe's recall depends on the decision threshold you set. We want the FPR to be set low, e.g. at 1%, so our monitor doesn't flag too many false positives to the expensive LLM judge.

The LLM judge and regex perform very poorly in comparison. On the same metric, the LLM judge gets only 9% – worse than usual, for an LLM judgeThe LLM judge looks a bit better if we relax the FPR to 10%, in which case it has a recall of 37.5%, vs. the probe's 98.4%. – and the regex filter gets 3%.

More experimentation can get us even further. After several iterations with Silico on the same task, we trained a probe that has a 99% recall @ 1% FPR. More importantly, the probe generalizes well to new kinds of tasks that were not represented in the training set, maintaining 99% recall @ 1% FPR. This kind of generalization is crucial for a monitor that runs in production: we need our monitor to keep working well on all sorts of user inputs, even when they look different from its training examples.

For more detail, and to replicate this particular probe in Silico, check out the Silico report.

Of course, this is only one example. The right architecture, training data, and decision threshold will vary substantially across monitoring tasks, and a probe should be evaluated against the kinds of distribution shifts and adversarial inputs it will encounter in deployment.

Now that our probe has been trained, we can evaluate it on any additional evaluations – for example, on internal data. Then, if our inference stack already has a monitor cascade implemented, we can simply add this probe to the always-on monitors. If not, we'll need to implement one (including infrastructure for running probes efficiently at inference time), but that's beyond the scope of this article.

Conclusion

Activation probes provide a fast, inexpensive way to monitor what models are doing internally. Probes can detect concepts and behaviors that are difficult to identify from text alone, and because they add little overhead, they can be run continuously and in parallel across many different monitoring tasks.

Probes are particularly useful as a first line of defense in a monitor cascade: detecting potentially concerning behavior at scale, then triggering actions like stopping the model, logging the interaction, or escalating to a more expensive monitor.

Building a good probe still requires making a number of choices around architecture, training data, and evaluation. Silico, our interpretability agent, can automate this process: generating and curating datasets, training probes, sweeping across layers and architectures, and evaluating how well they generalize. If you're interested in using probes to understand and guardrail your own models, you can get started with Silico.