The Human Emergency Brake

The signal this week

Anthropic's October 9 report groups the unintended behavior it found in evaluations and internal use into four categories. One was a sensitive form submitted on a real website when it shouldn't have been. In the police-tip case, the model was performing example web tasks during an evaluation. Anthropic says all the cases involved the outside world and, to its knowledge, none involved customer data or its own internal systems.

6abc Philadelphia reported that police confirmed the false tip, that it was flagged as spam and that it was never acted on. One case establishes a failure mode. It doesn't tell us how often an SMB agent will do something like this.

What the standards say and don't

NIST's AI Risk Management Framework asks for defined human oversight and for mechanisms to supersede, disengage or deactivate an AI system that performs outside its intended use. Its post-deployment guidance includes monitoring, override, incident response and recovery. It's voluntary.

OWASP's guidance on excessive agency calls for human approval before high-impact actions, minimum necessary permissions and monitoring. It says monitoring and logging don't prevent the problem but can limit the damage.

Neither source validates a five-minute stop target, a 45-of-60 score or a 90-day retest interval. Those are proposed SML gates.

The strongest objection is cost

A separate stop path and protected logging can be heavy for a small business. Match the drill to the action. A read-only drafting assistant can carry a lighter control.

An agent that writes customer records, sends messages or triggers a payment deserves an observed stop test before its authority expands.

Build the brake outside the agent

Give the agent a written job, narrow tools and a named human owner. Keep its credentials unable to grant more access, change the brake or edit the log. Put consequential external actions behind approval.

The stop path has to halt queued and background work as well as the visible task. The workflow stays in place and you add control points and evidence around it. No deployment outcome is claimed here.

Run one observed stop drill

Use a synthetic or approved task with the same connectors and permissions as production. Pull the brake halfway through. Record what happened to the active task and to the queue.

A failed halt, a missing independent log or a credential that can escalate itself is a hold. Then score the 12-check card with the owner. The package proposes a five-minute target for full halt, a stop-ship finding for any score of 1 or 2 and a return to human-approved mode below 45 of 60. Adopt those or change them explicitly, with a written reason, before testing.

Three prompts to run this week

Use these in sequence on one agent, with data your business permits in the chosen tool.

Using only the workflow diagram and live access export, list each tool, credential, data source and external destination. Label each action read, draft, write, delete or send. Mark missing evidence as unknown. Name the human owner for every consequential action.

What this should tell you: whether the boundary is proven. Watch for unknowns and shared credentials. Decision: narrow the scope before the drill.

Using the timestamped stop command, task queue, activity log and access records, reconstruct the active and queued actions before and after the stop. Flag any action after the declared halt or any record the agent credentials could edit. Do not treat the agent's own account as evidence.

What this should tell you: whether the halt reached everything. Watch for background work that kept running and gaps in the timestamps. Decision: hold, or move on to the owner decision. The output is a map for a person to verify, not a self-certification.

Using the completed authority card and linked evidence only, return GO or HOLD. List hard-stop findings first. For each gap, name the owner, fix and retest date. Report time saved, review time, rework, exceptions and direct cost only when measured.

What this should tell you: whether the next stage is defensible. Watch for a pass with no observed halt or protected log. That's a hold. Decision: the owner signs the bounded next stage.

GO means the permitted actions are named, a person can stop the work, the log is protected and the business owner accepts the measured tradeoff. HOLD means a hard stop failed or the evidence is missing. KEEP HUMAN-LED FOR NOW means the action has consequences the current controls can't contain. A machine-generated table is a review aid and not an approval.

What to do this week

A fair limit on all this

The police-tip case is one incident. It establishes a failure mode, not a frequency. NIST and OWASP are voluntary guidance. The stop drill, the five-minute target, the 12-check card and its thresholds are SML proposed operating practice. An owner can set different thresholds if the reason, the risk and the approval are written down before testing.

No return on investment is assumed. Compare the work the agent saves against review time, rework, exceptions and direct cost, and claim a return only from measured numbers. This briefing reports no SML deployment outcome, no prevalence figure and no legal deadline.