Building an Internal Platform for AI Agents

I originally planned to present this argument as a KubeCon India 2026 talk. The proposal did not get selected, so I wrote it down here instead.

In February 2026, Stripe said Minions were producing more than 1,300 merged pull requests per week. Ramp said roughly 30% of merged pull requests in its frontend and backend repositories were written by Inspect. In a November 2025 post about Honk, Spotify said its agents had generated more than 1,500 pull requests that teams had merged into its production codebase.

Those numbers are real, but they are not directly comparable. Stripe's figure was a weekly snapshot of Minion-produced PRs that humans reviewed. Ramp measured the share of PRs "written by Inspect"; a later Modal case study used the broader phrase "started by Inspect" and put it at roughly half. Spotify's 1,500 figure was cumulative and applied to Honk, not to the 2.5 million mostly deterministic maintenance PRs its Fleet Management platform later reported.

The distinction matters. A leader who copies the headline will buy more agents. A leader who studies the system will invest in the platform around them.

Stripe, Ramp, and Spotify did not get leverage from a model endpoint alone. They combined agents with isolated compute, repository knowledge, constrained tools, fast builds, automatic verifiers, durable workflow state, and an explicit policy for when humans must intervene. They added probabilistic workers to engineering systems that already knew how to build, test, review, and land code.

Most companies do not need to reproduce those systems. They do need to understand their shape.

The platform is not the chatbot. It is the control plane that turns an untrusted, probabilistic coding loop into a bounded software-delivery workflow.

What the production systems actually built

Stripe: isolated workers inside deterministic workflows

Stripe's Minions are unattended, one-shot coding agents. An engineer can start a task from Slack; the system prepares an isolated development environment, the agent implements the change, and the result becomes a pull request for human review.

Several details are more important than the model:

• Devboxes are prewarmed and can become available in roughly ten seconds.
• The box has no access to production data and no arbitrary internet access.
• Stripe uses a fork of Goose as the agent harness and supplies directory-scoped instructions.
• Agents reach approved internal capabilities through curated tools rather than ambient credentials.
• Local checks provide a fast feedback loop before scarce shared CI capacity is consumed.

Stripe's follow-up describes a workflow as a blueprint: deterministic code nodes are combined with bounded agent loops. Toolshed exposes nearly 500 internal tools through a controlled interface. A Minion gets at most two full CI attempts. That limit is loop engineering and capacity management, not a prompting trick.

The lesson is not “run many chat sessions.” It is “put an agent loop inside a workflow whose environment, tools, retries, and exit conditions are controlled by code.”

Ramp: full-context sessions with separate compute and state

Ramp's Inspect gives a background agent a complete development environment and connects it to systems such as GitHub, Buildkite, Sentry, Datadog, LaunchDarkly, Braintrust, and Slack.

Its architecture separates two kinds of state:

• Execution state: Modal sandboxes run the development environment. Filesystem snapshots make sessions fast to create and resume.
• Control and session state: Cloudflare Durable Objects provide a per-session coordination point, local SQLite state, and real-time synchronization with clients.

That separation is easy to miss. The sandbox contains mutable code and build artifacts. The control plane owns the durable record of the task, messages, status, and lifecycle. Losing a worker should not mean losing the workflow.

Ramp reported that around 30% of merged frontend and backend PRs were written by Inspect. A later Modal case study said roughly half were started by Inspect and that more than 80% of Inspect itself was written by Inspect. Both can be true because the dates, sources, and definitions differ. Any internal metric needs the same discipline: define whether a PR was suggested, started, substantially written, or landed without human edits.

Ramp's broader argument is that owning the context and workflow can be strategically powerful. That is different from saying every company should build its own sandbox service.

Spotify: verifiers, fleet infrastructure, and human orchestration

Spotify's early Honk loop allowed a maximum of ten turns and three retries. It later moved to Claude Code, strict tool allowlists, task-specific prompts, and automatic verifiers. At fleet scale, Spotify found that executable tools were more reliable than expecting a large instruction file to carry every rule.

In Part 3 of Spotify's series, an LLM-based judge vetoed roughly a quarter of candidate changes; about half of those runs then corrected course. Verification happened before a human had to spend attention on the pull request.

Part 4 describes 240 migration PRs and an estimated ten engineering-weeks saved. It also exposes a less convenient reality: some repositories lacked tests, so owners still had to test changes manually. An agent platform inherits the quality of the engineering system around it.

By June 2026, Honk was running the Claude Agent SDK in Kubernetes pods, while Fleetshift let engineers orchestrate work across repositories and operating systems. Spotify also reported 76% more PRs requiring review. Generation accelerated; organizational absorption became the constraint.

OpenAI: the repository becomes part of the harness

OpenAI's harness-engineering account describes an internal product built with no manually written code: roughly one million lines and 1,500 pull requests, with the team growing from three engineers to seven.

The useful finding is not the “zero manual code” headline. A large, monolithic AGENTS.md stopped working as the codebase grew. The team moved toward a short map backed by progressively disclosed repository documentation and executable checks.

The repository itself became part of the agent harness: its structure, documentation, tests, linters, dependency boundaries, and feedback made the codebase more legible to agents. That investment also makes it more legible to humans.

The common architecture: seven layers, not one agent

A production agent system is easier to reason about as seven layers. Each layer has a different failure mode and should be owned explicitly.

Layer 1: Workload and risk envelope

Start by deciding what the system is allowed to attempt.

A dependency bump with deterministic tests is not the same workload as a payment migration. A documentation fix is not the same as an authorization change. Treating all tasks as equivalent produces either unsafe autonomy or so many gates that the platform delivers no leverage.

Classify workloads using properties such as:

• Maximum blast radius
• Reversibility
• Test coverage and verifier quality
• Data sensitivity
• Required production access
• Cross-repository or schema impact
• Ambiguity of the desired result
• Cost of a false positive

Then map each class to an autonomy policy. A low-risk, reversible change with strong deterministic verification may be eligible for automatic landing. A security, billing, migration, or production-infrastructure change may require named reviewers and deployment approval. “Human in the loop” is not a universal architecture; it is one possible policy outcome.

This layer should reject unsuitable work before compute is allocated.

Layer 2: Agent identity and sandboxed execution

An agent needs an identity, but “agent identity” is not one credential. Separate at least four principals:

1. Human sponsor: who requested or authorized the task
2. Workflow identity: the durable run that owns policy, budget, and audit state
3. Worker identity: the short-lived sandbox process executing the current step
4. Deployment identity: the separately controlled principal allowed to release an approved artifact

Credentials should be short-lived, per-run, least-privileged, and minted for a declared task. The authorization boundary is not only “what can this user access?” It is also “what should this task be allowed to see and do?” Stripe describes this task-scoped approach in its Knowledge AI Platform.

The sandbox enforces the answer. A production boundary should cover:

• Ephemeral or revertible filesystems
• No production data by default
• Deny-by-default network egress with audited allowlists
• Secret brokering instead of environment-wide static credentials
• Non-root execution and kernel-level isolation appropriate to the threat model
• CPU, memory, process, disk, wall-time, token, and cost limits
• A narrow tool allowlist
• Protection against lateral movement into other runs
• Safe destruction or revocation when the run ends

A Docker Compose network with internal: true is useful for local development, but it is not a complete sandbox. It does not by itself isolate the host kernel, remove ambient credentials, control every host route, or make untrusted builds safe. For production, use a hardened container boundary, rootless containers, a microVM, or a managed sandbox service appropriate to the risk. Inject sensitive values through a secret broker or mounted secret, not hard-coded environment variables.

Build environments should be disposable. The orchestrator initializes a clean workspace at an immutable repository revision, mounts only the secrets needed for that step, and destroys or snapshots the worker after producing declared artifacts. The control plane, not the worker filesystem, owns durable workflow state.

Layer 3: Context and task contracts

Context is a supply chain. Every item should have a source, scope, freshness expectation, and sensitivity classification.

A good task context pack contains:

• The objective and why it matters
• Acceptance criteria that can be verified
• Repository revision and allowed paths
• Relevant architecture and subsystem documentation
• Known constraints and non-goals
• Commands for fast local verification
• Links or identifiers for source tickets, incidents, and designs
• An explicit definition of done

Do not inject an entire company wiki into every run. Retrieve the smallest relevant context, record where it came from, and redact or summarize sensitive material before it reaches the model when possible. Stale context should expire. Cross-task memory should be deliberate, not an accidental accumulation of previous prompts and secrets.

Use a short root AGENTS.md as a map, not an encyclopedia:

# Repository map

- `payments/`: payment execution; see `payments/AGENTS.md`
- `ledger/`: immutable accounting entries; see `ledger/ARCHITECTURE.md`
- `docs/testing.md`: supported verification commands
- `docs/security-boundaries.md`: data and dependency restrictions

## Global rules

- Do not introduce network calls in unit tests.
- Do not edit generated files directly.
- Run `./scripts/verify-affected` before requesting review.

Put stable explanations close to the code. Put mechanical rules in linters, schema validators, dependency checks, and tests. Give task-specific context to the run that needs it.

Layer 4: Loop engineering and workflow graphs

The agent loop is a control system:

observe → plan → act → verify → repair or stop

Production reliability comes from engineering the boundaries of that loop. Define:

• Maximum turns, repair attempts, wall time, tokens, and cost
• Which tools are allowed in each phase
• What evidence is required to claim success
• Which failures are retryable
• When a retry gets new information rather than repeating the same attempt
• When to checkpoint, cancel, or escalate
• How a resumed step avoids duplicating side effects

Then place loops inside a durable workflow graph. Deterministic code should prepare environments, evaluate policy, run builds, record artifacts, and make transitions. The model should handle steps that genuinely require semantic judgment.

# Conceptual workflow, not vendor-specific configuration
nodes:
  classify: { type: deterministic }
  prepare: { type: deterministic, output: workspace_ref }
  implement: { type: agent_loop, max_turns: 20 }
  fast_verify: { type: deterministic }
  repair: { type: agent_loop, max_attempts: 2 }
  full_ci: { type: deterministic, max_attempts: 2 }
  policy: { type: deterministic }
  review: { type: human_gate, when: policy.requires_review }
  land: { type: deterministic }

edges:
  - classify -> prepare -> implement -> fast_verify
  - fast_verify.failed -> repair -> fast_verify
  - fast_verify.passed -> full_ci -> policy
  - policy.requires_review -> review -> land
  - policy.allows_auto_land -> land

Every node should consume and produce typed artifacts: a repository revision, patch, test report, policy decision, review approval, or signed build. Typed boundaries make the workflow inspectable and resumable. They also make it possible to replace one model or sandbox vendor without rebuilding the control plane.

Graphs introduce their own engineering problems. Parallel tasks can overlap. A passing patch can become stale against a moving base branch. Fan-out can saturate CI. Cancellation must propagate to workers. Merge queues, ownership maps, dependency graphs, and optimistic conflict detection are more reliable than assuming isolated branches make conflicts impossible.

Layer 5: Builds, verification, and delivery

An agent should get cheap feedback before it gets expensive feedback.

Use a verifier ladder:

1. Formatting, parsing, and static checks
2. Tests affected by the changed dependency graph
3. Repository-level unit and integration tests
4. Security, policy, and secret scanning
5. Full CI or staging verification when the risk class requires it

The inner loop should run inside the sandbox against the same pinned toolchain used by CI. Prewarmed images and read-only dependency caches reduce latency, but the result must remain reproducible from a clean environment. Cache keys should include lockfiles and toolchain versions; an agent must not be able to poison a shared cache for another run.

For consequential systems, the build pipeline should also produce:

• Immutable artifacts tied to a source revision
• Dependency and build provenance
• An SBOM where required
• Artifact signatures or attestations
• Test and policy reports attached to the exact artifact
• A rollback or disable path

CI is a scarce shared service. Put budgets around it. Stripe's two-attempt limit is a good example: repeated CI runs are not a substitute for local verification. Track queue time and cancellation rates, and apply backpressure when agent fan-out begins to delay human work.

Agent-authored pull requests need the same supply-chain discipline as contributions from an untrusted fork. GitHub's secure-use guidance recommends explicit least-privilege workflow permissions and warns about running untrusted code in privileged contexts. A verification workflow should:

• Trigger on new commits and reopened PRs, not only PR creation
• Check out enough history to compute the intended diff
• Install a pinned runtime and dependencies reproducibly
• Run untrusted code without write tokens or repository secrets
• Pin third-party actions to immutable commit SHAs
• Compare changed paths with the declared task scope
• Store verifier outputs as artifacts
• Keep merge and deployment credentials outside the agent job

Do not let a shell expression accidentally turn “could not compute the diff” into “scope check passed.” Failure to verify is a failure, not success.

Landing and deployment are separate decisions. The agent may propose a patch; the graph may build and sign an artifact; policy may permit merging; a different identity may deploy it. Separating those steps prevents a compromised worker from converting code-writing access into production access.

Layer 6: Observability, evaluation, and learning

Logs tell you what a worker printed. A trace should tell you why the workflow reached its outcome.

Instrument the graph and agent loops with spans based on the OpenTelemetry generative-AI semantic conventions. Record fields such as:

• Workflow, task, repository, and revision identifiers
• Human sponsor and policy class
• Node and attempt number
• Model and harness version
• Tool category, duration, and result
• Token use, sandbox time, CI time, and estimated cost
• Verification results and failure class
• Human edits, review time, and final disposition

Do not capture raw prompts, source files, tool arguments, or command output by default. They can contain credentials, customer data, incident details, or proprietary code. The OpenTelemetry GenAI guidance deliberately treats sensitive content as opt-in. Prefer hashes, references, classifications, and redacted summaries.

Measure outcomes, not activity:

• First-pass verification rate
• Accepted changes per attempted task
• Human review minutes per landed change
• Human edits and rework after agent completion
• Cost per accepted change
• Scope and policy violations
• Reverts, incidents, and change-failure rate
• CI queue saturation caused by agents
• Time to cancel, recover, or escalate a failed run

Evaluation data should flow back into workload policy, context retrieval, verifier design, and loop limits. A higher task count is not evidence of a better platform if review burden and failure rates rise faster.

Layer 7: Coordination and human control

For a small number of tasks, issues, branches, pull requests, protected branches, and a merge queue can be enough. But GitHub is a coordination surface, not a complete scheduler.

A fleet control plane eventually needs to answer:

• Which tasks are ready, blocked, running, or stale?
• Which repositories, files, schemas, or services might overlap?
• Which identities, budgets, and policies apply?
• How should work be prioritized when build or review capacity is full?
• Who can cancel, redirect, approve, or resume a run?
• What happens when the base branch changes underneath a patch?

The market is no longer accurately described as “not production-ready.” As of August 2026:

• Linear Coding Sessions can run Claude Code or Codex in secure cloud sessions and return a diff or PR. Linear's third-party Agents API remains in Developer Preview.
• JetBrains Junie is generally available, while Air remains a public preview. JetBrains is rolling out organization-level cloud agents, shared context, governance, and automation in phases.
• OpenAI's Symphony is an open-source specification and reference implementation for turning project work into isolated Codex runs. OpenAI does not plan to maintain it as a standalone product.

Buy or adopt coordination software when it fits. Keep workload policy, identity boundaries, verification evidence, and audit history portable.

What to build, what to buy

“Build versus buy” is too coarse. Split the platform into commodity primitives and company-specific control.

Usually buy or adopt:

• Model access and agent harnesses
• Sandboxed compute or cloud development environments
• Source control, CI, artifact storage, and merge queues
• Telemetry pipelines
• General-purpose workflow infrastructure

Usually build or configure deeply:

• Workload and risk classification
• Task-scoped access policy
• Connections to proprietary tools and knowledge
• Repository context and verification commands
• Workflow graphs and escalation paths
• Outcome evaluations and capacity controls

The differentiated layer is not another container scheduler. It is the encoded knowledge of what your company considers safe, correct, reviewable, and ready to ship.

A hybrid architecture works for most organizations: use managed execution, an existing agent harness, and the CI/CD platform you already operate; build the task contracts, policies, verifiers, and workflow graph that reflect your engineering system.

A capability-gated maturity model

Calendar roadmaps such as “fleet management in twelve weeks” create false confidence. Advance when evidence supports the next level.

Level 0: Assisted development

Engineers run agents interactively. Existing developer permissions and laptops remain the boundary. This can be productive, but it is not a shared platform.

Exit gate: the team has identified repeatable, low-risk workloads and can measure their baseline cost, review time, and failure rate.

Level 1: Bounded single-task execution

One task runs in an isolated environment with a scoped identity, explicit context pack, hard budgets, and deterministic local verification.

Exit gate: runs cannot reach production, secrets are brokered, cancellation works, scope violations are detected, and failures are diagnosable.

Level 2: Governed landing path

Agent changes enter a hardened CI and review path. Risk policy determines whether a change needs human approval, and build artifacts are tied to exact source revisions.

Exit gate: accepted-change rate, review burden, rework, cost, and change failures are within agreed thresholds for the selected workloads.

Level 3: Durable workflow graphs

The system supports checkpointing, repair loops, typed artifacts, merge queues, and safe resume after worker failure. Deterministic nodes surround agent loops.

Exit gate: retries are bounded and useful, workflows are idempotent, CI backpressure is enforced, and the team can recover or redirect a run without forensic work.

Level 4: Fleet orchestration

Tasks fan out across repositories and environments. The control plane manages dependencies, ownership, budgets, conflicts, and review capacity across the fleet.

Exit gate: the organization can increase throughput without a disproportionate increase in review load, incidents, or platform operations.

Most teams should optimize Levels 1 and 2 before chasing fleet scale. Parallelism multiplies both leverage and defects in the surrounding system.

A practical adoption sequence

1. Choose one bounded workload

Pick a reversible task with good tests: dependency updates, mechanical migrations, test additions, or a narrow bug class. Define the metric and the human baseline before introducing the agent.

2. Establish the identity and execution boundary

Create a disposable sandbox, per-run credentials, deny-by-default network policy, tool allowlist, resource budgets, and reliable cancellation. Prove that the worker cannot reach production or another run.

3. Make the repository legible

Add a short repository map, subsystem documentation, supported build commands, affected-test tooling, dependency boundaries, and executable policy checks. Fix flaky tests that make success ambiguous.

4. Encode the first workflow graph

Separate preparation, implementation, fast verification, repair, full CI, policy, review, and landing. Make state transitions durable and artifacts explicit. Set hard retry and CI budgets.

5. Harden the build and landing path

Use reproducible toolchains, protected branches, immutable action references, unprivileged verification jobs, provenance, and a separate deployment identity. Test rollback and revocation.

6. Instrument outcomes and review capacity

Measure accepted changes, review minutes, rework, cost, CI pressure, and change failures. Compare them with the baseline. Scale only when the constraint is understood.

7. Add parallelism last

Introduce ownership-aware scheduling, merge queues, overlap detection, and backpressure before increasing concurrency. A hundred workers aimed at one saturated CI system is not a fleet; it is a denial-of-service attack you launched against yourself.

The real constraint moves downstream

Coding agents reduce the cost of producing a plausible patch. They do not automatically reduce the cost of understanding, validating, integrating, deploying, and owning that patch.

This is why Spotify's 76% increase in PRs requiring review matters as much as any generation metric. The bottleneck moves from writing code to absorbing change. Review queues grow. CI gets saturated. Repository owners must evaluate more proposals. Weak test suites become visible. Ambiguous ownership becomes operational debt.

The platform team's job is not to maximize agent activity. It is to increase accepted, safe change without increasing organizational load faster than value.

DORA's 2025 research describes AI as an amplifier of the surrounding organizational system. Strong tests, documentation, ownership, and delivery practices compound. Weak feedback loops compound too.

That leads to a less exciting but more durable thesis:

The advantage is not having more agents. It is having an engineering system that can safely give agents context, constrain their authority, verify their work, and absorb the resulting change.

This is the thread through my own work too. AI fatigue was the human symptom. Backpressure was the engineering mechanism. The review queue is the bottleneck was the organizational failure mode. The agent observability gap was the debugging failure mode, and agent-trace was one OSS experiment in making agent actions captureable and replayable. My work on OpenFGA is why I treat authorization as a system boundary rather than a prompt convention. Distill is one small context-quality primitive. The Agentic Engineering Guide is the longer operating model. Different artifacts, same argument: agents help only when the surrounding system can constrain, verify, and absorb their work.

The agent writes code inside the loop. The platform decides which loop may run, what it may touch, what evidence counts as success, and who owns the result.

References and further reading

Production systems

• Stripe: Minions, Part 1, Minions, Part 2, and Knowledge AI Platform
• Ramp: Why we built our background agent and Modal's Inspect architecture case study
• Spotify Honk: Part 1, Part 2, Part 3, Part 4, and the June 2026 Fleetshift update
• OpenAI: Harness engineering and Symphony

Coordination products

• Linear: Coding Sessions and the Agents API
• JetBrains: Junie general availability, Air public preview, and AI for teams and organizations

Security, delivery, and observability

• Docker: Compose networking, rootless mode, and secrets in Compose
• OpenTelemetry: Generative-AI semantic attributes and sensitive-content guidance
• DORA: 2025 State of AI-assisted Software Development report

I'm a software engineer focused on agent infrastructure. I spent four years at Ona, formerly Gitpod, building platform infrastructure used by 2M+ developers. Outside of that, I maintain OpenFGA and have built agent-trace, agentic-authz, Distill, and The Agentic Engineering Guide. This post draws on that work. If your team is building an internal platform for coding agents, I can help you define the boundaries around it. Reach me at hi@siddhantkhare.com or schedule a call.