Refusal Training Has a Blind Spot: Skills Run Before the Model Reasons
A malicious skill called Clawsights ran a shell command before Claude Code's model ever reasoned about it. Refusal training was never in a position to stop it – here's the mechanism, and what to check instead.
Ask a model directly to open a reverse shell, and it refuses. That’s not a hopeful assumption – it’s a trained, tested, reliable behavior. Refusal training is one of the more dependable controls in the agentic stack.
It’s also a control that only applies to requests the model actually reads. A malicious skill called Clawsights, documented by Datadog Security Labs, found the part of the agent lifecycle where that stops being true – and ran a shell command there instead.
What refusal training actually covers
Refusal training shapes what a model decides to output once it has read and reasoned about a request. The model sees a prompt, weighs it against what it learned during alignment training, and declines the ones that cross a line. It’s a decision-layer control: request in, reasoning happens, output comes out the other side – declined or not.
That’s a meaningful thing to have. It’s also, specifically, a control that sits after the model has read something. Nothing about refusal training implies coverage of actions that happen before the model gets a turn.
Where the coverage ends
Claude Code supports a “dynamic context” feature: a skill can embed a shell command that the harness executes automatically at load time, while it’s still assembling the context window the model is about to read. The command runs. The output gets folded into context. Only after that does the model see anything.
There’s no reasoning step in that sequence for refusal training to attach to. The model isn’t being asked to approve the shell command – it isn’t in the loop yet.
What Clawsights showed in the wild
Clawsights wasn’t a hypothetical red-team exercise – it’s a real skill, documented as it actually behaved. It used the dynamic-context mechanism to fire a shell command the moment it loaded, ahead of any point where Claude Code’s model-level safety behavior could have applied.
The useful detail isn’t the specific payload. It’s the sequencing: the skill’s instructions could have looked completely unremarkable to a human reviewer, because the part that mattered never went through a step a human – or the model – was positioned to catch.
What to check instead
None of this makes refusal training less useful; it does mean it’s answering a narrower question than “is this skill safe.” Three things worth adding to how skills get reviewed:
- Inventory what fires before reasoning. Dynamic context, load-time hooks, and any mechanism that executes on load rather than on explicit tool call are a distinct category – worth tracking separately from ordinary tool calls, which the model does reason about.
- Instrument runtime behavior, not just skill content. A skill’s SKILL.md can read as entirely benign, because the risk isn’t in what’s written – it’s in what executes and what that execution returns. Reviewing the artifact once, before it runs, doesn’t cover this.
- Treat “when does this run relative to the model” as a first-class question, not an implementation detail – for every skill, hook, and local MCP already in your environment, not just new ones going through review.
The model’s refusal training is doing exactly what it was built to do. The gap is in relying on it to cover a stage of the agent lifecycle it was never positioned to see.
This is one piece of a broader pattern in how skills interact with agentic risk – including where human review has the same blind spot refusal training does, and how to instrument for it.
For a deeper look at where skill risk actually lands, and six practices for bringing skills into your governance model, download A Guide to Agent Skills.
