The agentic SDLC: steal the eval, skip the five agents

The agentic SDLC is having its moment, and most of the coverage focuses on the wrong part of it. On the Stack Overflow Podcast, Ryan talks to Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, about a five-agent end-to-end pipeline built on MCP servers.

My reaction was not "I want five agents." It was: the two smallest ideas in that episode are the only ones a two-person team in Colombo can actually run, and they're the ones nobody is copying.

---

🔍 The agent count is the least interesting number

Five agents wired across the lifecycle is an org-chart diagram more than an engineering insight. It tells you Motorola Solutions has enough scale to justify a dedicated stage per phase. It does not tell you that five is correct, or that four fails.

What the shape actually admits is more useful:

• Agents are placed at handoffs, not at tasks. Design → spec, spec → code, code → test.
• Handoffs are where requirements quietly get lost, so that's where the cost sits.
• MCP is the plumbing choice, not the idea. It's what makes each stage able to read the ticket, the repo, and the test results without a bespoke integration each time.

If you take one structural lesson: agents earn their keep at the seams between phases, not inside them. Automating "write the code" is crowded. Automating "did the spec survive the handoff" is not.

---

🛠️ Specification enrichment is the cheapest stage in the whole pipeline

The technique I'd implement tomorrow is specification enrichment: an extra stage that runs immediately after design, before anyone writes implementation code. QA moves left, into the requirements themselves.

Concretely, it's one prompt against your design doc that asks the questions a good tester would ask in review:

1. What are the boundary values, and what happens at each one?
2. Which inputs are unspecified, and what does the system do when it gets them?
3. What's the failure mode when a dependency is down?
4. Which acceptance criteria are untestable as written?
5. What did this spec assume without saying?

For a Sri Lankan tool like an EPF or tax calculator, that stage is the difference between shipping and shipping wrong. "Calculate the tax" passes review. "What happens at exactly the bracket boundary, at zero income, at a negative deduction, when the effective date falls mid-year" does not, until you answer it.

Key takeaway: the bug you catch in the spec costs one prompt. The same bug caught after a user files it costs a rebuild, a redeploy, and your credibility on a page Google already ranked.

You don't need an agent framework for this. You need a checklist prompt and the discipline to run it before you open the editor.

---

📊 Cohen's kappa is the part I'd steal first

The genuinely technical idea in the episode is using Cohen's kappa to compare multiple LLMs acting as judges. This matters because "LLM-as-a-judge" has a silent failure mode: your judge agrees with you often enough to feel right, and you never check how much of that agreement is luck.

Kappa corrects for chance agreement:

κ = (p_o − p_e) / (1 − p_e)

p_o = observed agreement
p_e = agreement expected by chance

Worked example, my numbers, not the episode's. You hand-label 100 outputs pass/fail and 80 are passes. Your judge also calls 80 passes, and agrees with you on 84 of the 100 items.

Quantity	Value
Observed agreement (p_o)	0.84
Expected by chance (p_e)	0.80×0.80 + 0.20×0.20 = 0.68
Cohen's κ	(0.84 − 0.68) / (1 − 0.68) = 0.50

84% accuracy reads like a working judge. A kappa of 0.50 says half the headroom above chance is unexplained. On an unbalanced dataset, raw accuracy flatters you badly.

The conventional reading of kappa (the Landis and Koch bands, widely used in inter-rater work):

κ range	Interpretation	Would I ship on it?
< 0.20	Slight	No
0.21 – 0.40	Fair	No
0.41 – 0.60	Moderate	Only with human spot-checks
0.61 – 0.80	Substantial	Yes, with sampling
0.81 – 1.00	Almost perfect	Yes

Run this across two or three candidate judge models and you get something an accuracy score can't give you: evidence about which model to trust, and whether your rubric is the actual problem. If every model scores fair, the rubric is ambiguous, not the models.

---

💰 Running this on a learning budget

The gap between a Motorola Solutions pipeline and a solo build in Sri Lanka is not intelligence. It's tokens per change. Five agents on every commit is a monthly bill; two agents at the two highest-leverage points is a rounding error.

Where I'd spend, in order:

Stage	Worth it solo?	Why
Spec enrichment	Yes, first	One call, catches requirement bugs before code exists
Judge / eval agent	Yes, second	Tells you whether anything else you automate is working
Code generation	Already have it	Your editor does this
Test generation	Maybe	Useful once specs are enriched, weak before
Release / deploy agent	Not yet	Needs infra maturity you probably don't have

Two practical notes. First, a hand-labelled set of 50 to 100 examples is enough to compute a meaningful kappa, and you can build one in an afternoon. Second, before you commit to a judge model, price the workload: our AI token counter shows how much context each eval call actually burns, and the model comparison tool lists input and output pricing side by side. Judges run on every candidate output, so per-token cost compounds faster than you'd expect.

---

💡 What this means for you

Read the source episode as a report from a large test-engineering org, not a blueprint. The transferable parts are small and cheap:

• Put an agent at the handoff, not the task. Design-to-spec is the highest-yield seam.
• Enrich the spec before writing code. Five questions, one prompt, before the editor opens.
• Never trust a judge you haven't measured. Compute kappa on a hand-labelled set. Accuracy alone will lie to you on unbalanced data.
• Buy the two stages that pay, skip the three that don't. Agent count is not a maturity score.

Bottom line: an agentic SDLC isn't a fleet of agents. It's the two places where you stopped guessing.

I'm commenting on the Stack Overflow Podcast episode linked above, not reproducing it. The five-agent pipeline, MCP plumbing, Cohen's kappa evaluation and specification enrichment are theirs; the worked numbers, budget ordering and opinions here are mine.