Research · v1.0.1

Deterministic Chains: An Operations Agent That Earns Production Access

Lars Shakya Buch-Jepsen

v1.0.1Latest stable release: 2026-11-20

Introduction

The industry is busy wiring large language models into operations. The pitch is seductive: an agent that reads the alerts, decides what to do, and does it. Production systems quietly disagree. An operations agent that fixes things with a model inside its decision loop is an agent whose actions cannot be replayed, whose behavior drifts with a model update, and whose worst failure mode is a confidently wrong irreversible action at 3 a.m.

This piece states a position we have committed to in an operating platform, together with the architecture that makes the position enforceable. It can be compressed into one rule:

A model may propose. A chain must be deterministic.

Everything else – the promotion pipeline, the approval records, the operator’s shape – follows from refusing to put a model inside an executed action.

The position, and why it is not the obvious one

The obvious engineering response to “LLM-powered ops” is guardrails: sandboxes, confirmations for dangerous commands, dry-run modes. Guardrails are necessary and insufficient. The deeper problem is non-determinism as an operational property: if the same incident can produce different remediation on different nights, then no test proves anything about production, no audit reconstructs what was done, and no promotion of “this remediation is safe” can mean anything – there is no this to promote.

The alternative is to make the executed thing deterministic by construction. An operational change is expressed as a chain: an ordered sequence of steps such that the same input produces the same steps, in the same order, with the same parameters – every time. Chains are records, versioned, reviewable, diffable. A chain is a program, not a behavior.

This reframes what the model is for. In the agent-with-guardrails picture, the model chooses actions. In the chain picture, the model is a draft author and a novel-case escalation target: it may draft a chain candidate, it may explain one, it may propose runbook edits – but a draft is worth exactly nothing until it has been run in test, journaled, mined, validated, promoted, and approved, and by then what executes is a deterministic chain with a hash, not a model call.

The mechanics

Chains as records, test as the only first environment

Chains are declared as versioned records – a task record (one step), a chain record (ordered steps with parameters), and a policy record (what may run where). Every chain is environment-scoped, and test comes first, always: an unproven chain does not exist as far as production is concerned. Steps carry declared idempotency keys (re-running a chain after partial failure is a defined operation, not an incident), precondition checks (the chain asserts the world it expects before touching it), and rollback invocations.

The chain runtime produces a full replay log per run: every step, its input, its idempotency key, its outcome, its duration. An operations change you cannot replay is a change you cannot investigate; the replay log is not telemetry garnish, it is the artifact that makes the chain a legitimate unit of production work.

Promotion is earned, then approved

Chains earn production access through the same evidence ladder our other systems use, specialized to operations:

  1. Journal. Advisory chain proposals and their outcomes are recorded as fingerprinted decisions – never raw payloads, never secrets.
  2. Mine. Repeated change classes surface as candidates with conservative small-sample statistics; a chain class needs a minimum number of executed decision points before it is even a candidate.
  3. Validate. A candidate chain replays clean on fixtures and recorded incidents – 100% of its constraint checks, because the next stage is production.
  4. Promote. A signed, content-hashed promotion batch records the class’s elevation. The batch is the unit of change and the unit of revert: one step, atomic, journaled.
  5. Approve. A human flips the class to production in the policy record. Promotion and approval are separate acts – the system earns the right to be considered, the operator confers the right to execute. Earned-but-unapproved is a supported, visible, permanent state.

Once a class is both promoted and approved, the operator auto-executes it in production when its trigger fires – with full provenance from trigger to batch hash to per-step replay log. Decay demotes a chain class whose statistics have aged or drifted; a decayed chain demotes to test-only automatically and re-earns. The fleet never quietly keeps permissions it cannot justify.

Irreversible is never automated

The rule is absolute and simple: irreversible blast – destructive operations, production infrastructure applies – never routes to auto-execution, regardless of how much evidence a class has accumulated, and requires human approval per occurrence. Forever. Autonomy over the reversible; accountability for the irreversible. No amount of journaling converts a deletion into a reversible act, and the design does not pretend otherwise.

The operator is small, dumb, and auditable

The executing component – the operator – deserves its own paragraph, because the temptation is to make it smart. The operator in this design does four things: it reads the chain records, it checks the policy record, it executes approved deterministic chains, and it writes outcomes back. It contains no model, no heuristics, no initiative. If the operator were replaced with a shell script tomorrow, an auditor should find nothing surprising in the diff. All the intelligence in the system lives in what gets proposed and what gets promoted – the recorded, reviewable, revertible parts – and none of it lives in the part with its hands on production.

The reducer framing, one more time

It is worth naming the mistake this architecture is usually filed under. Read from the outside – “the LLM drafts it once, a deterministic chain runs it many times” – the system looks like a call-reduction scheme, and the descending model-call graph looks like the product. It is the receipt, not the product. What the system is actually doing is taking over the routine part of the operations decision function: each earned chain class is ground the platform permanently governs locally, deterministically, and auditably, with the model reserved for genuinely novel cases and human review. The companion research article in this collection makes the general argument; operations is simply the domain where the takeover framing has the shortest path to obviousness, because a non-deterministic takeover of production is unthinkable and a deterministic one is merely work.

Operations literature has long argued that reliability comes from boring, repeatable, reviewable practice [1] and that the unglamorous machinery around automation determines whether it can be trusted [2]. Chains are that argument taken literally: automation you can read, replay, revert, and refuse.

Evaluation status

Per the collection’s honesty discipline: the evidence ladder’s machinery and its measured gates in the originating domain (a code-transformation agent) are reported in the companion article. In the operations context, the chain runtime, the three record types, the approval records, and the operator exist as an approved specification in implementation; the plugin that journals ops telemetry and proposes advisory chains is likewise in implementation on the platform’s job scheduler. The first reversible chain’s promotion-to-production graduation, including a demonstrated kill-switch revert in production, is the acceptance milestone. No measured operational claims are made in this piece – no reduced incident counts, no time-to-remediation numbers, nothing. The architecture is the claim, and the piece is final in that position: every operational outcome remains future work, to be reported when the milestones above are met and not before.

Limitations

Runbook coverage is the honest limit of determinism: chains exist for the repeated and the foreseeable; the genuinely novel still requires a human with console access, and no amount of chain-building removes that – it only shrinks the fraction of the night it occupies. Environment drift means a chain validated yesterday is validated against yesterday’s world; precondition checks and the decay-re-earn loop mitigate this and cannot eliminate it. Approval latency is real: separating earned from approved inserts a human step into a pipeline that, under the reducer framing, would be fully automatic – this design keeps that human step on purpose and does not apologize for it. And the position is argued from operating experience and reading, not from a measured comparison against a model-in-the-loop operations agent; we do not intend to build the control arm.

Conclusion

The industry question “how much should operations agents be allowed to do?” is usually answered with thresholds. Our answer is structural instead: everything, provided what executes is deterministic, earned, approved, and revertible – and nothing irreversible, ever, without a human per occurrence. The chain is the unit that makes the answer enforceable: a model drafts, test earns, a human approves, the operator executes, provenance runs from trigger to hash, and a decayed chain walks itself back to test without being asked. Operations agents should not be trusted less than other automation. They should be auditable more – and the chain is the shape that survives the audit.

[1]
B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Site reliability engineering: How google runs production systems. O’Reilly Media, 2016.
[2]
D. Sculley et al., “Hidden technical debt in machine learning systems,” in Advances in neural information processing systems, 2015, pp. 2503–2511.