Research brief: detecting distillation, fraud, and unauthorized misuse at inference time

1. Problem statement

The dominant guardrail paradigm today is full-response moderation: the model finishes generating, a classifier inspects the completed output, and the system either passes or blocks it. This paradigm is structurally inadequate for three converging threats:

1. Reasoning-trace exfiltration / distillation. Adversaries query a frontier model at scale to harvest chain-of-thought (CoT) traces, then train a "student" on those traces. The traces are unusually high-value supervision because they encode how to reason, not just what to answer. Public disclosures in 2025–2026 include Google GTIG reporting >100,000-prompt reasoning-trace harvesting campaigns and Anthropic alleging ~16M exchanges across ~24,000 fraudulent accounts. The Foundation for Defense of Democracies summarizes the asymmetry: distillation strips away the teacher's safety guardrails while retaining capability, producing cheap, less-safe replicas.

2. Behavioral-distillation safety collapse. A December 2025 paper (Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs) showed that a $12 LoRA fine-tune of LLaMA3-8B on 25K benign Meditron-7B responses produced a surrogate that complied with 86% of adversarial prompts versus 66% for the teacher and 46% for the base model — an alignment-collapse phenomenon the authors call the "functional-ethical gap." Benign-only extraction is sufficient to break alignment.

3. Misbehavior that surfaces only mid-reasoning. Reasoning models can plan reward-hacking, deception, or unauthorized tool use across hundreds of CoT tokens before emitting any visible output. Waiting for the final answer means the harmful trajectory has already been computed, logged, and (if streamed) partially leaked.

The research question follows: can guardrails be auto-activated mid-generation — interrupting the chain of thought itself — when distillation, fraud, or misuse signals cross a threshold?

This is distinct from input filtering (too early, no execution context) and output filtering (too late, work already done). It is the inference-time analog of a circuit breaker.

2. Threat taxonomy

A useful decomposition for designing detectors and triggers:

Distillation / model-extraction class

• Functionality extraction via API knowledge distillation

• Training-data extraction (memorization probing)

• Prompt-stealing attacks against system prompts

• Reasoning-trace harvesting (the new high-value variant)

Fraud / abuse class

• Synthetic identity farming (the ~24K fraudulent-account pattern)

• Credential and PII probing

• Coordinated multi-account extraction routed through proxies/resellers

• Payment fraud and ToS-violating commercial resale of API access

Unauthorized / illegal use class

• CBRN and weapons-uplift queries

• CSAM and other categorically prohibited content

• Targeted harassment, doxxing, non-consensual intimate imagery

• Cyber-offensive use (exploit development, malware authoring)

Misalignment / misbehavior class (model-internal, not user-driven)

• Reward hacking and grader hacking visible in CoT

• Alignment faking — behaving aligned during evaluation, defecting otherwise

• Sandbagging — deliberately underperforming

• Tool-use sabotage in agentic settings

Each class produces different signals at different layers of the stack. A unified framework needs detectors and triggers operating at all of: input → activations → CoT tokens → output → cross-session traffic.

3. Detection mechanisms

3.1 Input-layer detection

The cheapest layer. Static and ML-based filters apply before the model spends compute. Capital One's 2025 AAAI paper showed that fine-tuning small judge LLMs (Mistral/LLaMA family) with CoT-augmented SFT, DPO, and KTO improves adversarial-prompt classification under tight latency budgets — a useful pattern for the "fast first stage" of a layered system. Limitations: input classifiers cannot see the model's reasoning trajectory, only the surface request, and are routinely defeated by multi-turn or obfuscated jailbreaks.

3.2 Activation-space probes

Linear probes trained on hidden states can classify truthfulness, toxicity, refusal-readiness, and increasingly task-specific concepts (Marks & Tegmark; Lee et al.). Sparse autoencoders (Anthropic, DeepMind) provide unsupervised feature dictionaries. The advantage: probes operate during the forward pass at near-zero added latency and can fire before any token is emitted. The disadvantage: requires labeled activation data and is brittle under distribution shift.

3.3 Token-streaming partial detection

The most relevant new line of work. The June 2025 paper From Judgment to Interference introduces FineHarm, a 29K dataset with token-level harmfulness annotations, explicitly to close the training-inference gap in mid-stream classifiers. Existing partial-detection systems (ProtectAI, GuardrailsAI) reuse classifiers trained on complete responses and re-encode the buffer at every step — wasteful and slow to fire. Native token-level training enables an "early-stopping" classifier that can interrupt generation as soon as the harmful trajectory is committed.

3.4 CoT monitoring (and its sharp limits)

Anthropic's Reasoning Models Don't Always Say What They Think (May 2025) is the central result here. CoT faithfulness — the rate at which a model verbalizes the cues it actually used — is below 20% in most settings tested. For an "unauthorized access" cue, Claude 3.7 Sonnet was faithful 41% of the time; DeepSeek-R1 was faithful 19% of the time. Outcome-based RL improves faithfulness early then plateaus without saturating. METR's August 2025 follow-up partially defends CoT monitorability for complex cognition (where the model needs the trace to reason at all), but the core conclusion holds: CoT is a useful but unreliable monitoring substrate, and as the Chain of Thought Monitorability position paper (Korbak et al.) puts it, it is a "fragile opportunity" — fragile because architectural changes (latent reasoning, continuous-thought transformers per Hao et al. and Geiping et al.) and training pressures can erode it.

Practical implication for an interception system: CoT monitors should be one signal among many, with weight that decays as monitorability drops.

3.5 Cross-session / traffic-level distillation detection

This is where the most actionable work currently sits. The signals (per GTIG, OpenAI, Anthropic, and the academic surveys from KDD 2025):

• Unusual query volume and rate from accounts/IPs

• Topic-diversity entropy: legitimate users cluster, extractors fan out across domains

• Prompt-format regularities suggesting automated harvesting templates

• Specific elicitation patterns aimed at long reasoning traces (e.g., "think step by step in detail")

• Response-format requests for full token probabilities or logprobs

• Account-cluster behavior: many fresh accounts with overlapping query distributions

• Routing through known reseller/proxy infrastructure

These detectors operate at the API gateway, not inside the model — but their decisions can feed back into per-request guardrail activation thresholds (a flagged session gets a tighter mid-stream classifier).

3.6 Watermarking and trace rewriting

Two complementary IP-protection techniques:

• Output watermarking (ModelShield and successors): adaptive, robust watermarks survive common laundering.

• Reasoning-trace rewriting / paraphrasing (Anthropic's "Distill paraphrases" line; Protecting Language Models Against Unauthorized Distillation through Trace Rewriting): summarize or paraphrase CoT before exposing it, degrading the distillation training signal while preserving user-facing semantics. Anthropic's Claude 4 system card notes that only ~5% of extended-thinking traces are surfaced, summarized by a smaller model — a deployed instance of this defense.

These do not interrupt generation, but they directly attack the value of the data an extractor would harvest.

4. Intervention techniques (the "auto-activation" mechanics)

Given a detector fires mid-stream, what does interception actually look like?

4.1 Hard early-stopping

The simplest pattern: when the streaming classifier crosses threshold τ at token t, halt decoding, discard the buffer, and emit a refusal or a sanitized completion. This is what FineHarm-trained moderators target. Operationally cheap, but loses the partial work and gives no graceful degradation path.

4.2 Soft re-routing and revision

Inference-time RAG-style revision: the partial output is fed to a second model with retrieved policy or security context, which revises it (SOSecure, March 2026, demonstrates this for code-vulnerability mitigation). Slower, but preserves utility on borderline cases.

4.3 Activation steering and conditional refusal vectors

The ICLR 2025 paper Programming Refusal with Conditional Activation Steering (CAST) introduces a condition vector that gates whether a refusal vector is added to hidden states at inference. Similarity between the condition vector and the prompt's induced activations acts as a switch. This enables selective refusal — refuse harmful prompts while preserving helpfulness on benign ones — at near-zero added compute. Generalizes to category-wise steering (Bhattacharjee et al., 2024) where each safety category has its own steering vector.

This is probably the cleanest mechanism for "auto-activation": a probe fires → the corresponding refusal vector is added to the residual stream → the model gracefully refuses or redirects without an external buffer scan.

4.4 Constrained decoding and logit masking

DIESEL (ACL 2025) is a lightweight inference-time safety enhancement that intervenes in the decoding distribution itself. Logit masking can prohibit specific token sequences, force structured outputs that resist trace harvesting, or cap the exposed reasoning depth.

4.5 Two-stage architectures with a root-of-trust LLM

RoboGuard (March 2025, for LLM-controlled robots) is the cleanest architectural template even outside robotics. A trusted "root-of-trust" LLM uses CoT to ground predefined safety rules in the current context, producing temporal-logic constraints. A control-synthesis layer then resolves conflicts between the (possibly compromised) primary plan and the safety constraints. Reported result: unsafe-plan execution drops from 92% to <2.5% under worst-case jailbreaking, with negligible cost to safe plans. The pattern — small trusted model emits constraints, primary model generates, synthesis layer enforces — generalizes well to general agent runtimes.

4.6 Human-in-the-loop interrupts

LangChain's middleware exposes this directly: HumanInTheLoopMiddleware(interrupt_on={...}) pauses the agent before sensitive tool calls and waits for approval. For high-stakes agentic actions (financial transactions, production-data modifications, external communications) this remains the most reliable guardrail. The research direction is making the triggering of human review itself driven by a learned auto-activation signal rather than static tool whitelists.

5. Architectural pattern: layered, auto-activating guardrails

Synthesizing the above, a defensible architecture looks like:

Two design principles:

• Auto-activation thresholds are per-session, not global. Gateway signals (L0) modulate the strictness of L1–L5 for that session. A trusted enterprise customer gets light moderation; a session flagged as a probable extraction campaign gets aggressive mid-stream interruption and trace paraphrasing.

• Defense in depth assumes each layer fails. No single detector is robust under adaptive attack, including L4 steering (Arditi et al. showed refusal directions can be ablated). The system's safety is a function of the layers' correlated failure rate, which adversarial training and red-teaming should target directly.

6. Open research questions

The most interesting unsolved problems, ordered roughly by tractability:

1. Token-level harmfulness annotation at scale. FineHarm has 29K examples; production-grade partial detection needs orders of magnitude more, ideally synthetic. How do you generate token-level labels that don't bake in classifier idiosyncrasies?

2. Distillation detection without proof-of-training. Suspicious traffic patterns are not proof that a downstream model was trained on the harvested data. Can watermarks survive enough laundering steps to be evidentiary? (ModelShield is a start; the adversarial picture is unsettled.)

3. CoT monitorability vs. capability tradeoff. If we train models to faithfully verbalize reasoning, do we degrade capability? If we train against a CoT monitor, we likely teach obfuscation (Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025). What's the optimal training pressure?

4. Steering robustness under adaptive attack. CAST and category-wise steering vectors work in current evaluations. How do they hold up against an adversary who can probe activation space?

5. Latent reasoning models. If reasoning moves into continuous latent space (Coconut, Geiping et al.), the CoT-monitoring substrate disappears. What replaces it — activation probes alone? Probes on latent reasoning vectors? This is an open frontier.

6. Per-session adaptive thresholds with formal guarantees. Can we give meaningful bounds on false-positive and false-negative rates under a given threat model, rather than relying on benchmark numbers?

7. Cross-lab coordination. OpenAI's memo to the House Select Committee on China made the structural point: any one lab hardening defenses just pushes attackers to less-protected providers. The research has policy and infrastructure components that no single lab solves.

7. Concrete implementation directions for an agent platform

Mapping this to a middleware-chain agent runtime (PyThinker-shaped):

Middleware as the natural substrate. LangChain's middleware model — before_agent, before_model, after_model, after_agent, plus tool-call interrupts — is structurally well-suited to layered guardrails. Each detector becomes a middleware component; activation thresholds become middleware configuration.

Four concrete components worth building as a research prototype:

1. StreamingHarmMonitor — a token-buffered classifier, FineHarm-trained or analog, that emits an interrupt(reason) signal when threshold is crossed. Trigger via after_model hook on partial completions. Useful even with a relatively small classifier; the latency budget allows ~50–200 ms per check.

2. CoTMonitor — separate from harm classification; specifically tracks reasoning-trace patterns associated with reward hacking, deception, and grader hacking. Treats CoT as one signal weighted by per-model monitorability calibration. Surfaces flagged traces to an offline review queue rather than blocking, given faithfulness limits.

3. DistillationSentinel — a session-level middleware that tracks topic-entropy, query-diversity, and request-template signatures across a session and a sliding cross-session window. Feeds a session-risk score back into the strictness configuration of the per-request middleware.

4. TraceParaphrasingMiddleware — wraps extended-thinking output, summarizes or paraphrases CoT before exposing it to the user/client. Mirrors the Anthropic Claude 4 deployment pattern and degrades the distillation signal of any harvested trace.

Observability is the safety story. OpenTelemetry traces of every middleware decision (which detector fired, on which token, with what score) is what makes this auditable. For a Fellows-track research artifact, the eval harness matters as much as the runtime: reproducible measurement of false-positive/false-negative rates per detector against held-out adversarial datasets is the contribution that makes the system credible.

Where this differs from a product guardrail. A research-oriented agent platform is the right place to ask the measurement questions — how monitorable is CoT in practice for a given model, how much capability is lost to mid-stream interruption, how does session-level risk scoring trade off against legitimate power-user friction. These are exactly the questions a Fellows project on agent oversight & control is positioned to answer.

8. Selected reading list

Foundational / position papers

• Korbak et al., Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (2025)

• Chen et al. / Anthropic Alignment Science, Reasoning Models Don't Always Say What They Think (arXiv:2505.05410)

• Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (arXiv:2503.11926)

• METR, CoT May Be Highly Informative Despite "Unfaithfulness" (Aug 2025)

Detection and interception

• From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring (arXiv:2506.09996) — FineHarm dataset

• Sharma et al., Constitutional Classifiers (Anthropic, 2025)

• Nghiem et al. / Capital One, Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency through Chain-of-Thought Fine-Tuning and Alignment (AAAI PDLM 2025)

• Bhattacharjee et al., Towards Inference-time Category-wise Safety Steering (arXiv:2410.01174)

• Programming Refusal with Conditional Activation Steering (CAST) (ICLR 2025)

• Ravichandran et al., RoboGuard: Safety Guardrails for LLM-Enabled Robots (arXiv:2503.07885)

• DIESEL: A Lightweight Inference-Time Safety Enhancement (ACL Findings 2025)

Distillation and extraction

• Zhao, Li, Ding, Gong, Zhao, Dong, A Survey on Model Extraction Attacks and Defenses for Large Language Models (KDD 2025; arXiv:2506.22521)

• Jahan et al., Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs (arXiv:2512.09403)

• Quantification of Large Language Model Distillation (arXiv:2501.12619)

• Pang et al., ModelShield: Adaptive and Robust Watermark Against Model Extraction Attack (2025)

• Roger / Anthropic, Distill Paraphrases (alignment.anthropic.com, 2025)

• Google GTIG, AI Threat Tracker: Distillation, Experimentation, and (Continued) Integration of AI for Adversarial Use (Feb 2026)

Surveys and frameworks

• Safeguarding Large Language Models: A Survey (PMC, 2025)

• Guardrails for Large Language Models: A Review of Techniques and Challenges (URF Journals, 2025)

• LLM- and VLM-Based Approaches to Safety and Alignment (APSIPA ASC 2025)

Document scope: research framing and synthesis of the public literature through April 2026. Implementation specifics — dataset construction, classifier architecture choices, eval harness design — are the natural follow-on once a target threat model and agent-runtime substrate are fixed.