Research note · Reza Ilham · 2026

Human Gates in Agent Pipelines

About this note

Where should a person approve an AI coding agent's work, and where does a gate just slow the team down? I built three setups at different levels of autonomy to measure what survives code review.

Chapter 1

The question

Most coding benchmarks ask whether a model can close a ticket in one shot. On a banking or healthcare app with 50M+ users, a passing test run is only half the job. When a subtle regression ships on Friday night, a person still owns the pager.

Put human gates too late in an agent pipeline and an engineer spends two hours untangling a 1,500-line pull request they did not write. Put gates on every step and the pipeline turns into paperwork. I want to know which approval points actually improve code survival and reduce bugs after merge.

Chapter 2

Three setups I built

Over the past year I shipped three projects that place the human at different points in the loop:

  1. No code gates (SliceBill). The baseline. I described the product and let an agent pick the stack, write the logic, and ship the web app without reading or editing a line of code. It works in production, though fixing a bug later means reading an unfamiliar codebase from scratch.
  2. One engineer at the diff (CatBoard). An engineer shapes a batch of tickets through MCP skills, runs several agents in parallel across separate git worktrees, and inspects each diff before merging into the main checkout.
  3. Role gates before code (CatAgent and Figma → Dev Spec). A team pipeline on top of Jira. Humans approve the story breakdown, the cross-platform API spec, and the TDD test suite before an engineer agent writes implementation code on a branch.
Explore the three setups
Filled nodes are the signals I measure. Hover or tap any node to inspect the boundary.
Three setups · human gate placement SliceBill: no gates CatBoard: diff gate CatAgent: staged gates Branch pipeline/KEY · commit markers Code survival & review time Bug origin (agent vs. human) Three setups · human gate placement SliceBill: no code gates CatBoard: worktree diff gate CatAgent: staged spec & TDD gates Branch pipeline/KEY · commit markers Code survival & review time Bug origin (agent vs. human)

Setup Select a node above to see where a human steps in and what gets measured.

Chapter 3

What I expect to find

Comparing these three setups lets me test three claims in a research master's:

  1. Gating the spec and tests raises code survival. When a human tech lead approves the technical spec and TDD tests before the code stage runs, a higher share of agent-written lines should survive code review untouched than when humans only review the final diff.
  2. Upfront gates cut total review time. Reviewing a short spec and test plan takes less time than rewriting a large implementation branch. Total engineer hours and fixup commits per story should drop even with the extra sign-off step.
  3. Bugs cluster at the handoff. Separating untouched agent commits from human review edits makes it possible to see whether post-merge bugs come from the agent's initial pass or from rushed human edits during review.

Chapter 4

What gets measured

Because CatAgent pushes to pipeline/<TICKET-KEY> with its own commit marker before an engineer checks out the branch, I can pull four signals straight from Git, Bitbucket, and Jira:

  1. Code survival rate. The line-level diff between the agent's initial commit on pipeline/<TICKET-KEY> and the merged pull request, checked again 30 days after release.
  2. Test-suite edits. How often the reviewing engineer rewrites the TL agent's TDD tests versus only editing the implementation files.
  3. Review time. Time in status from branch push to merge approval, compared against manual tickets of the same size in the same repo.
  4. Bug origin. Tracing QA and production bug fixes with git blame back to untouched agent lines versus lines a human rewrote during review.

Chapter 5

Where this goes next

I built the tools first because you cannot measure how a team reviews agent code without a pipeline they actually use. The next step is running this study formally in a software engineering master's program. If your lab works on human-in-the-loop software engineering or coding agents, send me an email.