Chapter 1
The question
Most coding benchmarks ask whether a model can close a ticket in one shot. On a banking or healthcare app with 50M+ users, a passing test run is only half the job. When a subtle regression ships on Friday night, a person still owns the pager.
Put human gates too late in an agent pipeline and an engineer spends two hours untangling a 1,500-line pull request they did not write. Put gates on every step and the pipeline turns into paperwork. I want to know which approval points actually improve code survival and reduce bugs after merge.
Chapter 2
Three setups I built
Over the past year I shipped three projects that place the human at different points in the loop:
- No code gates (SliceBill). The baseline. I described the product and let an agent pick the stack, write the logic, and ship the web app without reading or editing a line of code. It works in production, though fixing a bug later means reading an unfamiliar codebase from scratch.
- One engineer at the diff (CatBoard). An engineer shapes a batch of tickets through MCP skills, runs several agents in parallel across separate git worktrees, and inspects each diff before merging into the main checkout.
- Role gates before code (CatAgent and Figma → Dev Spec). A team pipeline on top of Jira. Humans approve the story breakdown, the cross-platform API spec, and the TDD test suite before an engineer agent writes implementation code on a branch.
Explore the three setups
Setup Select a node above to see where a human steps in and what gets measured.
Chapter 3
What I expect to find
Comparing these three setups lets me test three claims in a research master's:
- Gating the spec and tests raises code survival. When a human tech lead approves the technical spec and TDD tests before the code stage runs, a higher share of agent-written lines should survive code review untouched than when humans only review the final diff.
- Upfront gates cut total review time. Reviewing a short spec and test plan takes less time than rewriting a large implementation branch. Total engineer hours and fixup commits per story should drop even with the extra sign-off step.
- Bugs cluster at the handoff. Separating untouched agent commits from human review edits makes it possible to see whether post-merge bugs come from the agent's initial pass or from rushed human edits during review.
Chapter 4
What gets measured
Because CatAgent pushes to pipeline/<TICKET-KEY> with its own commit marker before an engineer checks out the branch, I can pull four signals straight from Git, Bitbucket, and Jira:
- Code survival rate. The line-level diff between the agent's initial commit on
pipeline/<TICKET-KEY>and the merged pull request, checked again 30 days after release. - Test-suite edits. How often the reviewing engineer rewrites the TL agent's TDD tests versus only editing the implementation files.
- Review time. Time in status from branch push to merge approval, compared against manual tickets of the same size in the same repo.
- Bug origin. Tracing QA and production bug fixes with
git blameback to untouched agent lines versus lines a human rewrote during review.
Chapter 5
Where this goes next
I built the tools first because you cannot measure how a team reviews agent code without a pipeline they actually use. The next step is running this study formally in a software engineering master's program. If your lab works on human-in-the-loop software engineering or coding agents, send me an email.