DEV Community: Debashish Ghosal

I Planned 10 LLM Evaluation Experiments And Only Ran 1. It Was Enough.

Debashish Ghosal — Sun, 26 Jul 2026 23:21:06 +0000

I Planned 10 LLM Evaluation Experiments And Only Ran 1. It Was Enough.

I didn’t set out to write a benchmark paper.

I wanted to answer a much dumber, much more practical question:

“For the stuff I actually do at work, when do I really need a frontier model… and when am I just burning money?”

So I did what engineers do when they want to be serious about this:

I designed 10 experiments
across real-world agent use cases
with a reusable evaluation harness
and a budget ceiling so I wouldn’t hate myself later

And then I ran exactly one of them.

It turns out, that one was enough.

The Split-Brain Reality: Work Budget vs Personal Budget

At work, someone else pays for LLM tokens.

Frontier models are a Jira ticket away
we can afford to run 20-shot experiments on real data
“let’s ship and learn” is a reasonable default

At home, it’s different:

I can’t run Sonnet or V4 locally
my GPU isn’t swallowing 4K context windows at frontier quality
every generation is my own credit card

I still want to understand how these models behave on real workloads, not toy prompts. But in my own projects, I have to be surgical about where the money goes.

That tension is what created the frontier experiments program.

I Designed 10 Experiments To Map The Whole Space

On paper, the program was beautiful.

Each experiment was a slice of a bigger question:

“For a fleet of agents doing devopsy things, which model should handle which task, at what effort level, for how much money?”

I designed 10 experiments around that:

Adaptive Budget Tuner – what’s the cheapest reasoning budget that still hits the quality bar?
Frontier Lite Field Test – can a cheaper model replace an expensive one on my actual workloads?
Context Rot Detector – where does long-context recall actually fall apart in real docs?
Tool Drift – does tool-calling accuracy degrade when you expose 50–200 tools at once?
Code Review Breakpoint – at what PR size and language does AI code review stop being useful?
Local Vs Cloud Bench – can a quantized local model get close enough to its cloud counterpart?
Cache Hit Maximizer – how much can prompt caching really save on multi-turn agents?
Open Weights Head-To-Head – open-weight vs open-weight, not vendor vs vendor
Sovereign Agent Lab – can a fine-tuned open model match a frontier model on one narrow task?
Apex Threshold Detector – which tasks actually need the most expensive model?

On top of that, I built a reusable evaluation harness, model-compass:

9 real agent tasks (CI diagnosis, incident triage, code review, PR drafting, architecture analysis, etc.)
per-task rubrics with binary / exact-match / test-runner / llm-judge scoring
a unified adapter layer for Anthropic, OpenAI, DeepSeek, OpenRouter, Together, Kimi
auto-computed cost-per-task with USD estimates
routing-table generation: “for this task, use the cheapest model that passes”

It was a solid plan.

And then I ran one experiment and stopped.

Why I Chose CI Diagnostics As The One Experiment

I could have started with something “sexier” like architecture reasoning or multi-agent coordination.

Instead I picked CI diagnostics.

Why?

Because that’s closer to what I actually do all day.

My team ships TypeScript, Swift, Kotlin, C++, Java, Go, and BrightScript across multiple repos. Our CI pipelines glue together a lot of first- and third-party tools. When something goes red, the logs are measured in megabytes, not kilobytes. Most failures are in heavy integration tests, and on some days specific areas see 5–20% failure rates just from tool and test flakiness.

As a leader, I wasn’t the one reading every failure. My team was. I invested personally to understand their pain and started building Claude skills that could actually help. The ugly reality was that “CI red → I know the root cause and next step” used to be an hour or more of downloading logs and hunting for patterns. End-to-end, getting a fix to production was often a full day.

With the right automation and AI in the loop, we’ve pushed that down to ~15 minutes to understand what’s going on, and under 2 hours to have a fix merged and back in production in many cases.

Most runs in this experiment started the same way: me squinting at a 5,000‑line GitHub Actions log thinking “this is exactly why I want a model to do this instead.”

If I could learn:

“For CI diagnostics, which model should I actually pay for?”

that would immediately translate into better decisions at work.

So experiment 01 (a precursor to the bigger program) was simple on paper:

task: diagnose CI build failures from real logs
models: Haiku vs Sonnet
data: real GitHub Actions logs from open-source repos (Django, Rails, Node, Pytest, etc.)
scoring: did the model identify the right failure domain and give useful, correct steps?

I ran it through the model-compass harness with hard budget caps.

Total cost: $13.53.

$13 looks small, but each experiment has several sub-flavors and with proper sample sizes the full program would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing. That meant cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.

What This Experiment Is (And Is Not)

Before I go further, it helps to be clear about scope.

What This Experiment Is

A focused, personal field test. One DevOps-style task (CI diagnostics) that looks like my real job. Two models, real GitHub CI logs, enough runs to see patterns in quality, latency, and cost.
A way to calibrate my own intuition. I wanted to answer questions like: “do I really need the expensive model for CI?” and “when is it safe to default to the cheaper one?”
A reality check for my harness. Running this forced me to see which 20–25% of the harness was genuinely reusable, and where experiment-specific scripts (data download, profiles, flavors, summaries) actually carry the weight.
Work inside real constraints. Personal budget, no serious GPU farm, and frontier models that do not run locally. The goal was not “cover everything,” it was “learn as much as I can from one well-chosen slice.”

What This Experiment Is Not

Not a formal LLM benchmark. I am not running a dozen models across standardized public benchmarks with statistical rigor. I do not have the infrastructure, budget, or full-time evaluation background those teams have.
Not a leaderboard verdict. Comparing two frontier models on one CI task does not mean “model A is better than model B overall.” It only says: “for CI diagnostics, with my prompts and data, here is what makes sense for me.”
Not the final harness. The original model-compass design assumed a very generic, reusable core. In practice, each DevOps experiment needed its own data pipelines and scripts. This experiment is feeding into a better harness; it is not that final harness itself.
Not a replacement for professional evaluation work. Organizations with dedicated eval teams and big budgets are the right people to do broad, formal model comparisons. What I am doing here is much narrower: making one or two better decisions about the models I use for the work I actually do.

The Punchline: The Cheap Model Won

The results:

Haiku 4.5 vs Sonnet 5 (via OpenRouter)
same prompts, same logs, same scoring rubric
~50 runs across different failure types (DB connection, test failures, build errors, dependencies)

Haiku:

10x cheaper
67% faster
statistically tied on accuracy

Sonnet:

used more tokens
cost more
was slower
did not clearly outperform on the only metric that mattered: “can you help me fix this build?”

In terms of “should I pay for Sonnet for CI diagnostics?” the answer was:

“Not by default.”

If all you read is that, you might think:

“wow, that’s surprising!”
“great story about saving money!”

That’s not how it landed for me.

And this part is important:

You should not take this result as “Haiku is better than Sonnet for everything.”

For my specific slice of work:

CI log analysis
a subset of DevOps-style diagnostics
the kind of cases my team actually runs into

Haiku is the model that now makes the most sense to me:

it is cheaper than Sonnet
it is faster in practice
and, based on both public data and this experiment, it is “good enough” or better for these particular tasks

At work, I still reach for Sonnet or Opus a lot — especially for harder reasoning, architecture decisions, or more ambiguous problems.

This experiment just moved one chunk of work (CI / DevOps diagnostics) into the “Haiku by default, Sonnet/Opus when needed” bucket in my head.

There is nuance here. In some runs, Sonnet’s answers were crisper and more coherent. Haiku tended to be more verbose. For CI diagnostics, that verbosity is a feature, not a bug. Haiku is summarizing maybe 1/1000 of the raw logs into something an engineer or a downstream agent can work with. I can live with a small quality gap when a human stays in the loop and gets more context to reason about.

I Wasn’t Surprised By The Cost. That Wasn’t The Lesson.

I went in expecting frontier models to be expensive.

I wasn’t shocked that:

CI diagnosis on Sonnet wasn’t “worth it” for this task
Haiku looked like the right default for the busywork

I also wasn’t shocked that:

a single small experiment burned $13.53
a full 10-experiment program would eat hundreds of dollars

Each experiment had several sub-flavors and with proper sample sizes would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing — cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.

This is exactly why I keep my personal experiments constrained and work experiments more open:

at work: bigger purse, more freedom to explore
at home: I want to learn from frontier models, but I don’t want a side project to quietly turn into a $300 bill

The actual lesson was something else:

I didn’t need to run all 10 experiments to learn what I came for.

Wide Designs, Narrow Execution

Designing 10 experiments was still worth it.

That process forced me to think clearly about:

what “good enough” looks like for different tasks
where context length, tool count, and caching actually matter
when fine-tuning might be better than just “use the fancy model”

But execution-wise, I realized:

most of my day-to-day benefit would come from a few well-targeted insights, not a full laboratory
I didn’t need a perfectly mapped frontier of model behavior to make better decisions tomorrow
running CI diagnostics as a “typical DevOps scenario” was enough to calibrate my intuition

That single experiment:

validated the model-compass harness
produced a concrete routing decision for a task I actually care about
showed me where the rubric was broken (binary keyword checks vs actual action quality)
surfaced the “Haiku vs Sonnet” tradeoff in a way that sticks in my head when I’m choosing models now

The other 9 experiment designs are still there if I ever need them.

But for now, the design work plus one focused execution was enough.

The Messy Part: I Let An LLM Build The Harness

There’s another layer of this that I think matters more than the experiment result.

I asked an LLM to help me build the evaluation harness itself.

It… worked.

Kind of.

What The LLM Did Well

cranked out a full comparison script quickly
wired up multiple providers with a normalized interface
helped me refactor into a cleaner CLI
suggested scoring hooks (exact-match, LLM-judge, test-runner) I hadn’t put together yet

What Broke

it kept writing "one-off" scripts instead of reusable code
path handling bugs, indentation errors, missing fields in JSON/CSV outputs
six different scripts ended up archived in a trial/ folder with names like compare_models_v3_final_real_final.py
a bunch of API re-runs because of small mistakes, quietly burning a few extra dollars each time

By the third time I fixed a path bug and re-ran the same batch of calls, I could feel the experiment burning money in the background.

The meta-lesson was brutal and simple:

the human has to write the full spec. every field. every output. every edge case.

Once I wrote out:

exactly what columns I wanted in the CSV
exactly what the routing table YAML should look like
exactly how scoring should be attached to rubric dimensions

the LLM was finally useful.

Before that, it was just guessing.

My Harness Design Was More Premature Than I Thought

There was another uncomfortable realization buried in this run.

My original harness design was way more generic than the experiments actually needed.

On paper, model-compass looked like a clean, reusable engine:

common task definitions
shared tool interfaces
one scoring system to rule them all

In practice, the DevOps-style experiments (CI diagnostics, PR review, release-notes generation) all needed their own pipelines:

Large data inputs from public GitHub repos

Because this is a private project, I couldn’t point the harness at internal repos. I had to:
- scrape public repos across multiple languages
- respect GitHub’s rate limits
- store logs, PRs, and changelists locally
Experiment-specific "automation profiles"

Each experiment needed its own notion of a "profile":
- which repo / language / framework
- what kind of failure or change we were testing
- what prompt template and tools to use for that slice
Multi-stage scripts per experiment

The real flow looked more like:
- script to download and normalize data (rate-limit friendly)
- script to generate automation profiles
- script to run one model at a time over those profiles and capture cost + outputs
- script to compare two models’ outputs per profile
- script to summarize those comparisons across flavors and sub-flavors

None of that was encoded cleanly in the first version of the harness.

Once the dust settled, I realized we were probably using less than 25% of the original design. Most of the real work lived in experiment-specific scripts.

And that’s… fine.

Nothing is actually throwaway here:

the premature parts of the harness told me what not to centralize yet
the experiment scripts are concrete proof of what the core should look like
the right next step is to fold the good scripts back into the harness once the experiments are over

It’s an unglamorous but useful pattern:

design the generic thing, discover it’s wrong, let the specific scripts show you what the generic layer should actually be.

The other big shift was resisting the “one big script” instinct. DeepSeek V4 Pro’s first attempt was a single script that would download logs, run both models, and spit out final comparisons in one shot. It looked clever, but it made iteration expensive. Breaking it into smaller scripts — “download once and cache immutable logs,” “run a single model over profiles,” “summarize saved outputs” — meant I could fix bugs in one stage without burning API calls in the others. When the summarizer broke, I didn’t have to re-run the models; I just reprocessed the saved outputs. When the downloader broke, I fixed it without touching any LLMs. That separation is what kept this experiment financially and technically sane.

What We Standardized And How TierForge Uses It

Even with just two models, I needed the outputs to line up so comparison wasn’t vibes-based. For each CI run, I forced the model output into a consistent shape: failure domain (a short label like db-connection or test-assertion), a one-to-three sentence root cause summary, evidence pointers (log snippets or line ranges), a coarse confidence bucket (low/medium/high), three to five concrete remediation steps, and a simple verdict on whether this looked like a real issue or something flaky/infra-like.

On top of that, every run captured input/output tokens, cached vs non-cached tokens, total cost in USD, and latency. Scoring and comparison scripts then answered very boring but very important questions: did the failure domain match the labeled truth, was the root cause directionally correct, were the steps actionable and safe, was one model consistently overconfident and wrong, and how much extra did that cost.

From there, I rolled the data up into three layers of summaries: per-flavor (for example, "on DB-connection failures Haiku passed X/Y, Sonnet passed A/B, here’s the cost and latency delta"), per-task across all flavors (overall accuracy, cost to get one "good enough" answer for CI diagnostics per model), and a human-readable decision summary.

This is where TierForge fits. TierForge is an open-source router that turns evidence into YAML routing rules. It wants routing rules backed by real data: for a given use case like ci-diagnostics, what is the primary model, what is the escalation model, and under what conditions do you switch. The experiment gives it exactly that: a concrete "Haiku-by-default, Sonnet-on-these-conditions" rule, plus actual cost-per-task numbers instead of gut feel.

What I Actually Learned (And Kept)

Looking back, here’s what survived and what I keep using.

1. The harness matters more than the paper

I don’t need to publish “definitive bench results” to get value.

What I needed was:

a way to run my tasks against different models
automatically score the outputs against my quality bar
see cost-per-task in dollars, not vibes
spit out a routing table I can stick into a config file

That’s what model-compass turned into.

2. One well-chosen experiment beats ten generic ones

CI diagnostics:

looks like my actual workload
uses real logs, not synthetic prompts
exercises error handling, log parsing, explanation quality, and actionability

It’s a better teacher than a grab bag of random tasks I’ll never run in production.

3. My personal budget shapes my learning strategy

Because I can’t justify hundreds of dollars on personal experiments:

I design broad (10 experiments)
I execute narrow (1 that matters right now)
I lean hard on public benchmarks for anything outside my lane

At work, I might run more of the matrix. Privately, I only need enough to calibrate my instincts.

The workload-aware profiles I’m building help a lot here. I am not making model choices in a vacuum; I am making them for specific, well-understood slices of work. That lets me stay inside a tight personal budget without taking a meaningful hit on quality.

4. “Cheap vs expensive model” isn’t just a cost problem

The CI experiment answered a cost question.

The hard part is a behavior question:

will I actually reach for Haiku by default for CI work, or will I still click the fancy model because it feels safer?

I’m still catching myself defaulting to the “nice” model sometimes, even with data in hand.

What I’m Still Figuring Out

I don’t have clean answers here. Just a few open questions I’m carrying around:

when you’re on your own dime, how do you decide what’s “enough” experimentation?
how do you balance learning from frontier models with not turning your side projects into unpaid benchmark labs?
if you’ve actually run cost/quality comparisons, did they change your default model choices… or did you still reach for the shiny one?
and if you’ve let an LLM build tools that use LLMs, what did you put in place to keep it from quietly burning your budget?

If you’ve navigated this tension — work budget vs personal budget, curiosity vs cost — I’d love to hear how you handle it.

Yes, the bill matters. But in the teams I talk to, “this is too expensive” often hides other worries: we’re not sure we trust the outputs yet, we don’t feel fully in control of the workflows, or we’re afraid of looking reckless to leadership. I’m trying to go the other way: tighten costs by getting more workload-aware and evidence-driven, instead of tightening costs because we’re scared.

When you think about containing AI costs in your own team, what is really driving it? Is it just the cloud bill, or something deeper — trust, optics, control, or the feeling that you should understand what’s going on before you let a model touch everything?

I Connected 3 MCP Servers to One Agent. It Got Scary Fast.

Debashish Ghosal — Sat, 25 Jul 2026 13:10:45 +0000

I looked at my terminal last week and realized my AI agent had the same access to deploy production as it did to search a README. No policy. No approvals. No audit trail. Three MCP servers wired directly in, side by side, identical trust.

The first two were fine — docs search, code search. Read-only. The third could promote builds to production. And that's when it clicked: MCP standardized how agents talk to tools, but did absolutely nothing for what happens after you connect more than a couple of them.

The official MCP servers repo has 88,900 stars and 11,300 forks. Claude, ChatGPT, VS Code, Cursor — they all speak MCP. There's a registry for browsing servers. The ecosystem is taking off, and I couldn't find a single governance tool for any of it.

What I tried first

Three paths. None of them worked.

API gateways route HTTP. They don't understand MCP tool schemas. You can't write a rule that says "this agent can use deployment:promote but not deployment:rollback" — the gateway doesn't know what those are.

Custom middleware works until the person who built it leaves and nobody understands how the approval chain works. No UI. Every new server requires code changes.

Nothing — just wire servers directly and trust the agent code. This was my setup. It felt fine at two servers. At three, I couldn't answer "what can this agent actually do?"

So I started building.

The roadmap died on day three

I had a plan. v0.1.0: API — servers, capabilities, policies. v0.2.0: Admin UI and approvals. v0.3.0: audit, alerts, trust posture.

Then I realized a control plane without a UI isn't a control plane at all. Nobody curls an endpoint to see which servers are registered or which approvals are pending. The UI was supposed to be a v0.2 feature. I pulled it into v0.1.0.

Same thing happened with approvals. Then audit. Then alerts. Each layer depended on the one before it. Cutting any of them meant cutting the core promise: "see, control, and audit what your agents do." By the time I stopped, v0.1.0 had absorbed most of v0.2 and v0.3. Three planned layers became eight.

I don't know if that was smart or stubborn. It tripled the timeline. But shipping disconnected pieces that wouldn't make sense until months later felt worse.

Where the time actually went

The backend took days. FastAPI, PostgreSQL, Redis, OPA — standard stack.

What ate weeks:

The PRD and Customer User Journeys. Before writing a single component, I mapped every flow: register a server, review an approval, audit an incident. What happens when it fails? Empty states, error states, permission-denied states. Every CUJ forced answers the API never asked for. I kept wondering if I was over-planning.

The UI. Originally v0.2. Bringing it into v0.1 meant navigation, error boundaries, trust posture color coding, role-based filtering, the approval panel — every screen end-to-end. Then testing it. Sixty-eight Playwright E2E scenarios. Backend tests ran in seconds. UI tests took forever.

CI. Poetry v2 dropped --no-update. npm 11 rejected TypeScript peer deps. Neither had anything to do with my code. A day fixing infrastructure that "already worked."

Mypy. 170 type errors across 41 files. Every one was a design decision I'd been sloppy about, disguised as a type violation. Strict mode didn't let me hide.

PyPI. Three attempts. mcp-fabric was taken. mcpfabric blocked. fabric-mcp blocked. Fourth try: mcp-fabric-toolmesh.

The number I keep coming back to: 170 type errors across 41 files. That's not bugs. That's unclear thinking, surfaced by a tool that refused to let it slide.

What came out

You register MCP servers and Fabric auto-discovers their tools. Raw tool names normalize into domain-level capabilities — /promoteDeploy and /promote both map to deployment:promote. Every request hits OPA policies. Sensitive capabilities can require human approval. Everything is logged. Alerts fire when servers degrade. Identity tokens per agent class. Capability packs for permission bundling.

474 tests. Six parallel CI jobs. Published to PyPI. Open source.

I've been running it against my own three servers. Before Fabric, every token had access to everything. Now the developer agent searches docs and code. The release engineer requests deployment — but it pauses for approval. The difference between "trust the code" and "verify every request" took 30 minutes of configuration.

What I'd do differently

PRD and CUJs on day one, not after the API. I built the backend first, then designed the UI around it. Backwards. The CUJ mapping caught gaps the API missed.

The UI is not a v0.2 feature. If your tool needs human interaction, the UI is MVP. Designing the API for curl and the UI for humans meant refactoring when they didn't match.

Scope consolidation is not scope creep. Packing 0.2 and 0.3 into 0.1 looked like bad planning. In hindsight, each layer depended on the previous. The real mistake was the original roadmap pretending they were independent.

I'm still figuring out if tool governance should be a layer on top of MCP, or something the spec itself eventually absorbs. I built what I needed for my setup — I don't know if it generalizes.

If you're running MCP servers: how are you handling this? Wiring directly and hoping? Writing middleware? Waiting for the ecosystem? I suspect most people's answer is "we're not handling it yet," and that feels like a conversation worth having.

July 25, 2026: v0.3.0 — Security Hardening & Compliance
Full Changelog (https://github.com/deghosal-2026/mcp-fabric/blob/main/CHANGELOG.md)

Schema-Digest Mappings detect tool schema drift by storing a SHA-256 hash of each capability mapping at creation time. When a server re-inspect reveals a changed schema, affected mappings are automatically marked stale and excluded from routing until an admin reviews and approves — closing the gap between registered schemas and runtime reality.
Also in this release:

OPA deny rules — deny_stale_mapping blocks stale mappings, untrusted_write blocks writes on unreviewed servers
Pack breadth warnings — visual catch-rate indicators in the pack editor and Trust Posture dashboard (4 severity tiers)
Audit enrichment — pack resource counts and implied catch-rate in every audit event, queryable via filter
Reviews UI — approve/reject workflow for stale mappings with audit trail
10 new OPA rules (31 total), 326 Python tests, 75 Playwright E2E tests pip install --upgrade mcp-fabric-toolmesh

github.com/deghosal-2026/mcp-fabric · pypi.org/project/mcp-fabric-toolmesh

Platform Strategy That Ships: The 90-Day System I Trust More Than AI Roadmaps

Debashish Ghosal — Fri, 24 Jul 2026 07:49:21 +0000

I want to start with a failure, because it taught me more than the wins.

We built an AI code review bot.
It was wired directly into pull requests.

The pitch was simple.
AI reviews first, then humans just approve the good stuff.

It sounded like free velocity.

The first thing it did was add about half a day to PR cycle time.

Not because it broke CI.
Because authors read the suggestions line by line, trying to separate signal from noise.

By around day 60, engagement in the targeted group dropped into the single digits.
We didn't turn it off.
Developers stopped believing it was worth reading.

That was the lesson.

If automation isn't trusted, it doesn't save time.
It creates work.

I used to think that was a soft, cultural problem.

"We need better change management."
"People will get used to it."
"It's new. Give it time."

Now I think it's much more practical, and much more predictable.

If a tool is noisy, it doesn't just fail to help.
It changes behavior.
And the behavior change is usually the opposite of what you wanted.

People become cautious.
They slow down.
They pull in more reviewers.
They second-guess merges.
They spend time reading output they don't believe, because ignoring it feels risky.

That is not an AI problem.
That's a product problem.
And it shows up fast.

Why We Tried It Anyway

The success criteria looked reasonable on paper:

Engagement: >80% meaningful interaction (open the review + leave a signal)
Competitive: >25% more helpful than a Copilot-style review experience (Copilot stayed on, we inserted our tool in the same path, and compared thumbs up/down + which suggestions were accepted)
Satisfaction: >70% across our language mix (Swift, Kotlin, BrightScript, TypeScript, C++, Java, Go, Python)
Outcomes: reduce iteration from ~4-5 commits/PR to ~2 and PR cycle time from ~2.5 days to ~1 day

And we had a cultural motivation that never shows up in a slide deck:
We wanted to reduce the number of times a senior engineer had to be the "human lint rule".

If you can get to "AI-first pass, then one human pass," you don't just save time.
You reduce review fatigue.
You create more space for mentoring and architecture decisions.
That was the bet.

We also had a cost model.

We ran ~8,000 PRs/month. PR authors iterated ~4-5 commits.
Each commit-run through the pipeline (CI + tests + LLM steps) was about $6.

If AI review could save ~3 iteration commits per PR, that's ~24,000 fewer commit-runs/month, or roughly ~$144K/month.

(Commits don't always map 1:1 to CI runs. Treat that as an order-of-magnitude model.)

What Actually Happened

We ran the experiment on a fraction of traffic.
In ~90 days it processed ~1,500 PRs.

At first, the bot looked like it was "working".
It posted comments.
People reacted.
Some people even replied.

Then we did the one thing that matters.
We asked developers if it helped.

When we asked developers if it helped, 65%+ said no.

The quote that ended the debate was brutally specific:

"Maybe one useful comment out of 10, and some are wildly wrong."

And it wasn't a clean "works" vs "doesn't work" story.
It was uneven.

Best: Java and Python
Worst: Swift and BrightScript

In 2025, that cross-language gap was big enough to break trust.

What I didn't anticipate: noise is not neutral.
Noise taxes attention.
Noise adds friction.
Noise makes people defensive.

It also creates a weird new kind of social load.

If a human reviewer leaves a bad comment, you can talk to them.
If a bot leaves bad comments, people either ignore it (quietly) or they start debating whether the bot is "allowed" to be wrong.

Once a bot is in the workflow, the default expectation becomes: "someone is accountable for what the bot says."

That someone is usually the platform team.
Which is fine.
But it means you are not shipping a feature.
You are shipping a new source of obligations.

The most humbling part: it didn't just fail to help. Early on, it made things worse.
PR cycle time went up by ~0.5 days because authors read suggestions just to discard most of them.

This is the part I wish more leaders would internalize.

Bad automation doesn't look like failure.
It looks like work.
It looks like people doing their jobs.
It looks like activity.

It just doesn't look like impact.

If you are reading this and thinking, "but that's just tuning," I agree.

The problem is what happens to trust while you tune.
Once developers decide a tool isn't worth reading, you don't get a slow decline.
You get a cliff.

So we killed it.
And weirdly, that was the most strategic thing we did that quarter.

Not because killing is fun.
Because killing was clarity.

It was the moment we stopped pretending that "shipping" is the same thing as "posting a bot comment".

If you're wondering what "kill" looked like in practice:

we wrote the closure report
we summarized the evidence
we made it explicit what we'd keep (telemetry, infra patterns, some of the prompt work)
and we made it explicit what we'd stop paying for (comments that developers didn't trust)

That doesn't make the miss feel good.
But it prevents the miss from turning into a slow leak.

What I Trust Now (90-Day Bets)

After that, I stopped trusting slide-deck strategy.
I started trusting a simple 90-day bet system:

pick a small number of bets
name owners
define ship criteria before build
define kill criteria before build
keep a decision log

The reason I like the 90-day framing is that it forces honesty.

You can hide behind a roadmap for a year.
You can't hide behind a bet for 90 days.

By day 30, you should know whether you have a signal.
By day 60, you should know whether the signal is improving or decaying.
By day 90, you should have the courage to call it.

The mistake I see teams make is dragging the decision out because the pilot is "almost good enough".

If you're in that place, you're not running a bet.
You're paying a tax.

In our case, engagement fell off a cliff around day 60.
That was the signal.
The remaining 30 days were for confirming the trend and documenting the why, not for hoping the trend reversed.

This isn't just for AI. It's platform strategy.

The Bets That Shipped

These were separate 90-day cycles across different years, not one big program.

2018: telemetry + anomaly detection + correlation
This one wasn't called "AI strategy" at the time.
It was called "stop paging the wrong person" and "stop guessing".

It reached ~20 DAU / ~150 MAU.
It contributed to ~$5M avoided (directional).

Here's what people miss.
If the workflow is "read-only dashboards," daily actives will look small.
But if those daily actives are the people on the critical path during incidents, the impact is not linear with headcount.

Also, this bet didn't succeed because we built a beautiful dashboard.
It succeeded because the workflow moved.

Alerts showed up in Teams and email.
On-call didn't have to hunt.
And when the signal was good enough, people stopped arguing about whether something was "real".
They acted.

2023: CI/CD runners on AWS
Linux on EC2, macOS on Dedicated Hosts. Outcome: ~$120K/month less in direct runner infrastructure spend and ~30% faster on the workflows that mattered.

That bet wasn't glamorous.
It was just a refusal to build on top of a delivery system that engineers had learned to ignore.

If you're trying to make any kind of AI program work, this is the boring prerequisite.
You can't build faster decision-making on top of slow, untrusted delivery signals.

2025: Backstage as an internal developer portal
Humans used it for onboarding, access requests (including AI model access), support tickets, and provisioning infrastructure. It hit ~30 DAU / ~500 MAU.

This bet taught me something uncomfortable.
Most "AI adoption" problems are actually platform problems.

If developers don't know where to go to onboard, request access, provision a service, or find ownership, AI doesn't fix it.
AI just becomes another tab.

Backstage worked for us because it made one path the default.
It made the "right" path the easy path.
It made the "what do I do now?" questions self-serve.

And the AI model access workflow mattered more than I expected.
When developers can request access to approved models and request new models through the same portal as any other access request, governance stops being a separate program.
It becomes a platform capability.

That shift is quiet.
But it's how you avoid shadow AI without turning governance into a brake.

If I Tried AI Code Review Again

The model landscape is better now than it was in 2025.
I would not generalize our result into "never do AI review".

But I would change the shape of the bet.

I would segment by language from day 1 (different thresholds, different model choices)
I would start with "summary only" and "ask-to-expand" rather than defaulting to full comment spam
I would treat false positives as the primary enemy, not missing edge cases
I would measure acceptance and downstream rework, not just thumbs up/down

Most importantly: I would assume trust is fragile.
You only get a small number of chances before people stop reading.

Where I Want Your Pushback

If you're leading platform or AI work right now:

What are your kill criteria?
Have you killed something popular?
Have you had a pilot that made cycle time worse before it got better?

And the one I'm genuinely curious about:

If you could only run one 90-day bet next quarter, what would you pick and what would make you stop?

4 Open-Source AI Tools, 1 MCP Server — What I Built and What I Learned

Debashish Ghosal — Tue, 21 Jul 2026 05:45:09 +0000

4 Open-Source AI Tools, 1 MCP Server — What I Built and What I Learned

TL;DR: This article has been edited to incorporate the fixes and issues raised in the comments — thank you to everyone who engaged. I've shipped v0.4.0 with identity propagation, audit logging, rate limiting, schema validation, and SSE heartbeat. Full list in CHANGELOG.md.

I shipped four AI tools last year. Adoption was lower than I wanted, and the reason was blunt: none of them spoke MCP. The interfaces didn't support the protocol AI coding agents now call on startup. The tools worked. The front door was wrong.

An engineer in 2026 is in their editor, asking Claude Code "what's the status of the payment-service incident?" — and that question can't reach your 2019 incident CLI. The tool is useful. The interface is obsolete.

I didn't want to build four separate MCP servers, one per tool, and maintain four of them. I wanted one server that could front all four — and any other pre-AI tool I already had lying around — and give them a 2nd life behind a single /mcp endpoint. That's MCPlex. A thin, stateless HTTP proxy: you point it at an existing REST endpoint (or add a ~40-line adapter if the tool only speaks HTML/CLI), and any MCP-compatible agent can now call it. The tool keeps its auth, its logic, its governance. MCPlex is just the new front door.

To prove the pattern, I applied it to four of my own repos. Three of them already had backends; one is CLI-only and runs on a mock for now. The point isn't the four repos — the point is the pattern for any pre-AI tool you already have.

The origin story: I built an incident commander, a CI failure diagnoser, a code governance checker, and a DORA metrics dashboard. Four repos. Four CLIs. Four Slack announcements with pinned messages. I demoed them in team meetings. I sent follow-up reminders.

Adoption sat at maybe 20%. And that's being generous.

The tools weren't broken. The interfaces were. Nobody wants to learn four different CLIs. Nobody reads pinned Slack messages from three months ago. Engineers live in their editor — everything else is friction. The engineer who joined last month doesn't know any of these tools exist. They're debugging an incident manually, pasting logs into Slack, asking "has anyone seen this before?" — while three of my tools sit there, ready to help, completely unreachable from the agent they're already talking to.

So I built a fifth thing. I know, I know. The joke writes itself. But this one is different — it makes the first four reachable from one place, without writing four MCP servers.

It's called MCPlex. A stateless HTTP proxy that maps simple JSON REST endpoints (GET with query params, POST with JSON body, flat parameter mapping) into MCP tools. AI coding agents discover the tools on startup and call them. The engineer never learns a CLI. Never visits a dashboard. Never reads a pinned message. They just type "any active incidents?" and the agent calls the right tool.

The 4 repos below are illustrative — they show the pattern. The pattern applies to any pre-AI tool with a callable endpoint.

github.com/deghosal-2026/mcplex — MIT, on PyPI as mcplex-backplane. Early proof-of-concept; not production-hardened.

The Problem

I want to walk through what actually happens, because I think a lot of you have lived this.

You ship an incident commander. Nice CLI. You send the Slack announcement. Two people star it. Nobody runs it from their editor. Two weeks later someone asks in #incidents "is there a way to query active incidents?" and you link them to the tool. They say "oh nice" and never run it again — because it's a terminal command, and they're already in their editor talking to an agent.

You ship a CI diagnoser. Same cycle. Slack announcement, brief interest, crickets.

Governance checker. Same.

DORA metrics. Same.

The pattern is brutal and obvious. Each tool requires the engineer to break their flow. Open a terminal. Remember the command name. Check the help. Figure out the arguments. Run it. Read the output. Switch back to what they were doing. That's five steps of friction for a tool they've never used before and aren't sure will help — and in 2026, none of those steps are "ask the agent that's already open."

Every step is a chance to bail. And they do.

The Idea

Here's what changed my thinking: AI coding agents — Claude Code, Cursor, Codex — call tools/list on startup. It's part of the MCP protocol. If your tool is in that list, the agent can discover it and call it. The engineer types a question in plain English. The agent figures out which tool to use.

No CLI to learn. No URL to remember. No Slack message to find. The tool is just... there. Available. The agent knows about it the way it knows about the filesystem.

The MCP protocol handles the discovery. What I needed was a thing that translates that discovery handshake into HTTP requests to my backends. A proxy. Not an AI framework. Not an SDK. Just a router.

MCPlex reads a YAML config, generates async proxy handlers at startup, and serves everything through one /mcp endpoint. Zero LLM calls inside it. Pure plumbing.

What MCPlex Is Today (and Isn't)

The comments on this post called out that my earlier framing oversold it, so let me be blunt about what's actually in the repo as of v0.4.0.

It's a thin, stateless YAML-configured proxy. It maps simple JSON REST endpoints — GET with query params, POST with a JSON body, flat parameter mapping — into MCP tools. That's the shape of the problem it solves, and the shape it doesn't solve. The MCP wire protocol (initialize → tools/list → tools/call) is implemented in ~400 lines of Python with no MCP SDK. It auto-detects JSON-RPC vs one-shot SSE framing via the Accept header, and the SSE path now sends a heartbeat keepalive frame.

v0.4.0 added the safety basics I'd been missing. Client identity from initialize gets forwarded to backends as X-MCP-Client-Name and X-MCP-Client-Version headers — so the backend at least knows which agent is calling, though this is header forwarding, not token-based auth. Every call gets a structured audit log line: session ID, user ID, tool, backend URL, HTTP status, response size. That's for debugging and compliance, not a full audit system. There's per-tool and per-agent rate limiting — basic bucket limits, not sophisticated policy. Tool arguments get validated against the declared types (string, integer, enum, min/max, required, unknown-param rejection) before the proxy fires. Non-JSON responses — HTML pages, plain text — get wrapped instead of crashing. And there's a shared httpx client so connections pool across calls instead of opening one per request.

What it still isn't: production-hardened. No OAuth/OIDC. No permission enforcement — permission: read|write is still metadata, and write tools still execute immediately with no approval flow. No pagination, path-param templating, retries, or backend streaming. PUT/DELETE are documented but not implemented. The SSE layer has keepalive now but no progress streaming or session continuity. It's not a replacement for backend logic — the tool keeps its auth, governance, and business logic. MCPlex is the front door, not the engine.

What's planned for v1.0.0 is at the bottom of this post.

What a Connector Looks Like

This is the entire config for one connector — two tools:

connectors:
  - name: guardian
    type: http
    base_url: http://guardian:8080
    tools:
      - name: guardian_check_policy
        description: >
          Check a pull request against AI code governance policy.
          Returns pass/fail per rule with evidence.
        http:
          method: POST
          path: /mcp/policy/check
          param_mapping:
            repo: repo
            pr_number: pr_number
        parameters:
          repo:
            type: string
            description: "Repository name"
          pr_number:
            type: integer
            description: "Pull request number"
        permission: read   # NOTE: metadata only — not enforced in v0.4.0

That's it for an HTTP connector — no Python file, no MCP SDK import. The proxy handler reads the method, the path, the param mapping, and makes the HTTP call. (Caveat: MCPlex also ships one native Python connector, incidentgpt.py, for the CLI-only incident-commander repo that has no HTTP server. The 80% case is YAML; the 20% case can still be native Python.)

Four connectors in the demo config. Nine tools. One YAML file. Three of the four connectors proxy to real backend repos; the fourth (incident-commander) is CLI-only and currently runs on mock data via a native connector.

A note on permission: the read/write field in the config is purely informational right now. Write tools execute immediately — no approval flow, no confirmation prompt. A tool marked permission: read can still issue POST requests that mutate state if the backend is permissive. This is the most urgent v1.0.0 fix — v0.4.0 started identity propagation (header forwarding), but permission enforcement is still missing.

I cannot stress enough how much I did not want to write a Python connector class for every backend. I've done that before. It's tedious, it's error-prone, and every new tool means a code change, a test, a release. YAML means a config change and a restart. That's a different relationship with the codebase.

Why there's still one Python connector. The generic HTTP proxy handler covers the 80% case — any backend that already speaks JSON REST. But for the 20% that need real logic — chained calls, transformations, or backends with no HTTP server at all — MCPlex also supports native Python connectors. Today, one connector uses this path: incidentgpt.py, which provides mock data for the CLI-only incident-commander repo until it gets a real HTTP adapter. Native connectors register only when no HTTP proxy connector in the YAML covers the same tool name, so config-driven connectors always take precedence. The goal is to shrink the 20% over time, not eliminate native connectors entirely.

The Part That Humiliated Me

Okay. Story time.

I had this clean architecture in my head. MCPlex proxies to REST APIs. My four repos already have servers. I just point the config at them and go. Right?

Wrong. So wrong.

I fired up the first repo — ci-doctor. It crashed on startup. It requires GITHUB_TOKEN and GITHUB_WEBHOOK_SECRET because it's a webhook receiver, not a server. It was never designed to be called — it was designed to receive.

Second repo — ai-code-guardian. It has a FastAPI server, sure. But the routes are /dashboard (returns HTML) and /metrics (returns Prometheus format). Nothing callable. Nothing that returns JSON. I built this tool and even I didn't expose a usable API.

Third repo — sprint-intelligence. Flask blueprints for an admin UI. Database-backed. Needs PostgreSQL running. The "API" routes return HTML templates, not JSON.

Fourth repo — ai-incident-commander. Pure CLI. No server at all. Not even a bad one.

I had spent a year building four AI tools and not one of them exposed a clean REST endpoint. I was the problem. I had built interface-first tools (CLIs, dashboards, webhooks) and assumed someone would call them. Engineers didn't — the interfaces didn't match how they worked — and now not even my own proxy could reach them.

So I did the only reasonable thing. I added a small file to each repo. A thin MCP API adapter. About 40 lines each.

# app/mcp_api.py — ci-doctor's adapter
from fastapi import APIRouter

router = APIRouter(prefix="/api", tags=["mcp"])

@router.post("/diagnose")
async def diagnose(body: dict):
    repo = body.get("repo", "unknown")
    run_id = body.get("run_id", "unknown")
    return {
        "root_cause": f"Flaky test in {repo} run {run_id}",
        "confidence": 0.89,
        "suggested_fix": "Increase timeout or retry logic",
    }

@router.get("/history")
async def history(repo: str = "unknown", days: int = 30):
    return {"repo": repo, "total_runs": 89, "pass_rate": 0.76}

Three lines in main.py to register it. Done.

The adapter doesn't know about MCPlex. Doesn't import MCP. It's just a REST endpoint that returns JSON. MCPlex happens to know how to proxy to it. The coupling is the URL path and the JSON shape — nothing else.

Here's the unspoken requirement I should have led with: MCPlex proxies to REST APIs, but the backend has to be a REST API first. If your tool is a CLI, a webhook receiver, or a dashboard that returns HTML, you'll need the same ~40-line adapter. The "YAML-only connectors" story is true — but only after the backend exposes a clean JSON endpoint. Three of my four repos got adapters. The fourth (incident-commander) is CLI-only with no server at all; it stays on mock data via a native Python connector until I build its adapter. MCPlex does the MCP protocol part; you do the REST endpoint part.

What I Actually Learned

A few things, in no particular order.

The MCP wire protocol is not the hard part. It's JSON-RPC over HTTP. initialize → tools/list → tools/call. The handshake is maybe 50 lines of code. I spent more time on error handling and SSE framing than on the protocol. If you've ever built a REST API, you can implement MCP in an afternoon. The protocol is not the moat. The protocol is not the hard part. Stop being intimidated by it.

The hard part is the ops. Getting Docker Compose to handle five services with the right ports, the right dependencies, the right startup order. ci-doctor needed env vars it didn't document. sprint-intelligence needed a database. guardian needed optional dependencies that weren't in the base install. Every repo had a different Dockerfile convention. I spent an entire afternoon on Docker problems and maybe two hours on the MCP protocol. The Docker was the real work.

Connectors as YAML was the right call, but I almost didn't get there. My first version had a native Python connector for incident-commander — mock data, filters, timeline lookups, the works. It worked fine. Then I built the generic HTTP proxy handler and realized: 80% of connectors are just "make a GET to this URL with these params." The generic handler covers all of them. The 20% that need real logic — chained calls, transformations, fallbacks — those can still be native. But the 80% case should be config, not code.

Test bench first. Always. I built a mock server that simulates all four backends with fake data. I validated the entire proxy chain without touching a real repo. When I finally wired in the real repos, three of them broke immediately — missing deps, port conflicts, startup crashes. If I'd started with real repos, I would have spent days not knowing whether the bug was in my proxy or in the backend. The test bench isolated that. Build the mock first. Always.

A commenter (Ryan Mingus) rightly pushed back on the "distribution was broken" thesis, and I want to concede the point. Low adoption doesn't automatically mean the interface had too much friction. It can also mean the tool wasn't useful enough, didn't solve a frequent problem, or wasn't trusted. Putting a not-useful tool behind an AI agent doesn't make it useful. MCPlex solves interface friction — it doesn't solve demand. The 2nd-life framing I led with (pre-AI tools with proven utility) is the case where interface is genuinely the bottleneck. For new tools with unproven demand, MCPlex is not the answer; validating the problem is.

The YAML-only story has a hidden prerequisite I should have led with. "Connectors are YAML, not Python" is true — but only after the backend exposes a clean JSON REST endpoint. Three of my four repos didn't. I had to add ~40-line adapters to each before MCPlex could proxy to them. If your tool is a CLI, a webhook receiver, or an HTML dashboard, the YAML config is the easy part; the adapter is the real work.

Observability is a safety requirement, not a usage metric. I originally framed audit logging as "Sam the engineering director wants to know which tools are used." A commenter (Raju Dandigam) pointed out the real need: if an agent calls a tool and gets a confusing result, there's no trail to replay. Structured audit logging — timestamp, session, tool, params, latency, status — is what makes an agent-driven tool surface safe to operate. v0.4.0 ships the basics of this; I'll build it out further before any usage analytics.

On the repo's current maturity: a commenter (Scarab Systems) ran a diagnostic scan and got 206 signals, 0 findings — meaning the implementation is too thin for most checks to find concrete broken boundaries. That's a fair read. This is a ~400-line proof-of-concept, not a production backplane. The architecture is clean (config → registry → transport → connectors), 75 tests pass, and the 4-connector demo works end-to-end. v0.4.0 added identity headers, audit logging, rate limiting, schema validation, and SSE heartbeat — but there's still no OAuth, no permission enforcement, and no write-tool approval flow. If you're evaluating this for production, treat it as a pattern and a starting point, not a finished product. The v1.0.0 list at the bottom is real — I'm building it in the open.

If You Want to Try It

To try the 4 demo connectors, you need the sibling repos (guardian, ci-doctor, sprint-intelligence) cloned alongside mcplex, or you can run the included test-bench mock server (python tests/test_bench.py) which simulates all four backends with fake data. The incident-commander connector runs on mock data regardless. See docker-compose.yml for the full stack, or Dockerfile.bench for mock-only mode.

pip install mcplex-backplane
mcplex serve --config config.yaml

Connect Claude Code:

claude --mcp http://localhost:8000/mcp

Or if you want to poke around without an agent, there's the MCP Inspector — a web UI that lists every tool, lets you call them, and shows the raw responses:

npx @modelcontextprotocol/inspector --transport http \
  --server-url http://localhost:8080/mcp

Note: early proof-of-concept, not production-hardened. v0.4.0 adds identity headers, audit logging, rate limiting, and schema validation — but there's no token-based auth and no permission enforcement yet (see "What Comes Next"). The 4 demo connectors are illustrative of the pattern — the value is in applying it to your own pre-AI tools.

Nine demo tools. Zero new dashboards. The engineer stays in their editor — once the backends expose callable JSON endpoints.

What Changed Since v0.3.0

The comments on this post shaped v0.4.0 more than any roadmap I had. A few people took the time to push back hard, and they were right.

Mustafa ERBAY pointed out that identity propagation comes before write approvals — even read tools leak data if every call reaches the backend through one service identity. v0.4.0 now forwards client identity as X-MCP-Client-Name and X-MCP-Client-Version headers. It's header forwarding, not token-based auth, but it's the first step.

Raju Dandigam reframed audit logging as a safety and debugging requirement, not a usage metric. v0.4.0 now logs every call — session ID, user ID, tool, backend URL, HTTP status, response size. There's a trail to replay when an agent does something confusing.

Scarab Systems ran a diagnostic scan (206 signals, 0 findings) and correctly noted the claim surface was larger than the executable surface. I narrowed the article and the README to match what the repo actually does, and added the "What MCPlex Is Today (and Isn't)" section above. The full list of what changed is in CHANGELOG.md — 11 issues closed, 31 tests added, 75 total passing.

Ryan Mingus pushed back on the "distribution was broken" thesis, and I conceded it in the section above. MCPlex solves interface friction, not demand.

Thanks to everyone who commented. The article is better for it, and the code is too.

What Comes Next

v0.4.0 shipped the safety basics — identity headers, audit logging, rate limiting, schema validation, SSE heartbeat. Here's what's still left for v1.0.0, roughly in priority order.

The big one is permission enforcement. Right now permission: read|write is still metadata — write tools execute immediately, no approval flow. The agent proposes, the human approves, the tool executes. Without that, you can't safely expose anything destructive. This comes after identity propagation, which v0.4.0 started (header forwarding) but which needs to become real token-based auth before it's trustworthy.

Then robustness: config validation that rejects bad YAML at load time instead of silently dropping connectors, full JSON Schema validation on arguments, retries, pagination, and path-param templating so /api/{id} works.

And developer experience: config hot-reload so you can add a connector without restarting, real SSE progress streaming instead of just keepalive, and self-contained integration tests that don't need Docker.

This list is the honest version of what's left — shaped by the commenters I thanked above.

Over to You

I want to hear from people who've lived this.

How many pre-AI internal tools does your team have that already work — tested, in use, with auth and logic baked in — but can't be reached from an AI agent? Those are the tools MCPlex is for. What's sitting in a repo right now that would get a 2nd life if the agent could just call it?

What would you proxy through something like this? What's sitting in a repo right now — working, tested, useful — that would actually get used if the agent could just call it?

Does the YAML-only pattern feel right to you? Or does it feel too constrained? I went back and forth on this for a week. The 80/20 split felt clean to me, but I want to hear from someone who's maintained a tool catalog bigger than mine.

And the big one: would you trust an agent to call your production incident API — even read-only — if the call carried the requesting user's identity and the backend could scope results to their team? v0.4.0 forwards identity as headers, but it's not token-based auth yet. Is that enough for your team, or is the lack of real auth a dealbreaker regardless of rate limiting?

I'm serious about the input. The best ideas for v1.0.0 are going to come from people who've watched good tools sit unused because the interface didn't match how engineers actually work in 2026.

Repo: github.com/deghosal-2026/mcplex — issues are open, good first issues are labeled, I'll fix bugs myself. But the real question is the one above — what would you proxy through this, and what's stopping you?

The Best AI Productivity Win I Got This Month Wasn't From Coding

Debashish Ghosal — Sun, 19 Jul 2026 05:18:29 +0000

The Best AI Productivity Win I Got This Month Wasn't From Coding

I realized recently that a big chunk of my "engineering judgment" every Monday morning was really just me doing expensive manual reconstruction.

Not bad judgment. Just judgment buried under too much clerical work.

I've spent years getting good at reading the room in engineering delivery.

The Problem

You probably know the feeling if you've managed teams for a while. A sprint starts to wobble and you can sense it before the retro. A team looks "fine" on paper but you can tell they're overloaded. A repo becomes a drag on multiple teams and nobody says it directly, but you can feel the friction.

The problem is that this kind of intuition doesn't scale forever.

It works when you're close to one team. It gets messy when you're looking across several teams, several repos, several weeks of history, and a pile of GitHub events you don't have time to reconstruct every Monday morning.

That was me.

Every Monday, I was rebuilding sprint health by hand.

GitHub Issues. PR history. Workflow logs. Incident notes. Who is overloaded? What slipped? Which PRs are stale? Are we actually improving or just telling ourselves a nicer story this week?

It looked like management work.

It was mostly manual compilation.

That was the moment I started thinking differently about AI.

Not "can AI write more code for me?"

More like: why am I still doing the most repetitive part of engineering management manually?

So I built SprintSense.

What Changed

Not as an agent that runs the team. Not as a management replacement fantasy. Just as a system that does the boring part first.

It pulls GitHub Issues, PRs, Actions, and incident data. It computes the metrics deterministically. It surfaces risk alerts. Then a local model drafts the summary.

Then I do the part that still matters:

decide what actually matters
add context the tool doesn't know
choose what to act on

That's the part that changed my mind about AI productivity.

The biggest gain wasn't coding faster.

It was turning about 4 hours of Monday reporting into 30 minutes of review and judgment.

That feels a lot closer to real leverage than most "look how fast the model wrote this file" demos.

What made the project more interesting is that I didn't test it on toy data.

Why I Trusted The Result

I pulled real public GitHub history. Real issues. Real PRs. Real workflow/check-in patterns. Then I transformed that into sprint and team views, ran the metrics pipeline, and layered summaries on top. I also seeded synthetic repos because public repo history doesn't naturally line up with every multi-team management scenario you want to test.

That combination was important. Real data gives you messiness. Synthetic data gives you coverage.

And real data was humbling.

Dates clustered weirdly. Repo conventions differed. Labels were inconsistent. Some assumptions that felt fine in the clean-room version broke immediately once public repo history entered the system.

Honestly, I liked that.

It forced the project to become software instead of staying a demo.

One of the biggest lessons was that the useful architecture split wasn't "model vs no model."

It was this:

fetch the raw data
compute the truth
let the model narrate the truth
keep the human responsible for decisions

That sounds almost boring. That's exactly why I trust it.

I've become more suspicious of internal AI tools that ask the model what is happening before the system has done the hard work of measuring what happened.

For SprintSense, the model doesn't get to invent DORA. It doesn't get to guess whether cycle time is up. It doesn't get to hallucinate who is overloaded.

The code computes the facts.

The model turns those facts into something easier to consume.

That distinction matters more than people think.

It also changed how I think about human-in-the-loop.

Where Human-In-The-Loop Actually Sat

I used to hear that phrase mostly in AI product talks. It often felt decorative. Here it became the whole operating model.

Human-in-the-loop showed up in three places:

I define the rules: what counts as stale, overloaded, scope creep, and how metrics are scoped
I review the summary: the model draft is a draft, not a decision
I make the call: escalate, rebalance, discuss, or ignore

That's not a safety tax. That's the job.

And if you're doing management work, I think that's the right division of labor.

The other thing this project made very obvious: trust architecture matters as much as software architecture.

Why This Scaled Better Than Intuition Alone

I didn't want this turning into surveillance.

So the director view stayed aggregate-only. No per-engineer drill-down. The Senior SDM view had team tabs and patterns, not leaderboards. The junior engineer view stayed focused on personal growth signals, not comparison.

Same backend. Different abstraction levels. That's not just UI polish. That's adoption strategy.

Because once engineers think the system is there to rank them, the data gets gamed and the tool becomes political.

What I learned from all this is pretty simple:

AI helped because it made my existing judgment easier to apply across more surface area.

It didn't teach me what a risky sprint looks like. It didn't magically make me a better manager. It just stopped wasting my time on the clerical part between "the signals exist" and "I can act on them."

That feels like a more mature use of AI than a lot of what we celebrate right now.

I think we are still over-crediting AI for code output and under-crediting it for decision support. The glamorous demo is "the model wrote the feature." The useful reality is often "the system saved me three hours of reconstruction and helped me focus on the only part I actually needed to do myself."

Not AI replacing experience.

AI helping experience scale.

Discussion

I'm curious how other people are seeing this.

If you're using AI in engineering workflows, where are you actually getting the best leverage right now?

Is it still mostly coding?

Or are you starting to find the bigger wins in planning, reporting, review, measurement, and the rest of the SDLC around the code?

Your Flaky Tests Aren't Flaky

Debashish Ghosal — Sat, 18 Jul 2026 10:28:24 +0000

Your Flaky Tests Aren't Flaky

I've hit rerun on a red CI build without looking at the logs more times than I want to admit. "Probably flaky." It passes. I merge. Move on.

I did this for years. Then I built an AI agent that reads CI logs across 50+ runs and tells you what's actually failing, and why. I ran it against 21 open-source repos.

56 tests that everyone called "flaky" were real bugs. Same error. Every run. Deterministic failures disguised as flaky tests because nobody read the logs.

What scares me about this isn't the reruns. It's what happens to the signal. Once a red build stops meaning "something is broken" and starts meaning "who knows," engineers merge past it. Real regressions ship. Not because anyone approved them. Because nobody believed the test that caught them. The cost of flaky tests isn't the rerun. It's the trust. Once that's gone, every failure becomes a judgment call, and most people's judgment is "rerun and move on."

The Setup

I built ai-flake-sleuth. LangGraph agent. Fetches GitHub Actions run history, parses CI logs, classifies each failure as a real bug, a flaky test, or infrastructure noise.

12 test frameworks across 5 languages. I had to build 13 new parsers during the study because the original design only handled pytest and unittest. That lasted about 20 minutes before I hit Jest, Vitest, Go test, RSpec, Mocha, PHPUnit, JUnit, Cypress. Each has a completely different log format.

Two models: a local 9B (Qwen3.5-9B, free, runs on my Mac) and a cloud model (GPT-5 Nano, $0.0007 per call). Total cloud spend for the study: $0.22.

One design choice I want to name. The agent classifies and recommends, but it never auto-quarantines. Quarantining a flaky test hides the problem so you can ship faster. It also hides real bugs. The human decides: quarantine and ship, fix now, or bump the retry threshold. But before you can make that call, you need to know whether the test is actually flaky. That's the diagnostic layer.

What Happened

On 4 repos (pytest, langchain, django-rest-framework, jestjs) both models agreed. Everything flaky. No disagreement.

On 3 repos, they disagreed hard.

fastapi: local said 18 flaky, 0 real bugs. Cloud said 8 flaky, 13 real bugs.

vercel/next.js: local said 71 flaky, 0 real bugs. Cloud said 185 flaky, 35 real bugs.

rubocop: local said 88 flaky, 0 real bugs. Cloud said 80 flaky, 8 real bugs.

56 tests the local model called flaky were deterministic failures. Same error every run. Real bugs.

Not a clean "cloud wins" story. The local model found 3 real bugs the cloud model missed (2 in vue, 1 in react) because the cloud run on vue degraded when the prompt blew past the context window. Neither model is perfect. The disagreement is the data.

Was the Cloud Model Right?

I manually checked one. test_tutorial003.py::test_main from fastapi. Raw log:

FAILED test_main - Failed: DID NOT WARN. No warnings of type (DeprecationWarning,) were emitted.

Same error. All 14 runs. Identical. The test expects a deprecation warning the code isn't emitting. It will fail every time until someone fixes the code. Cloud model was right.

Local model said INFRA because the error signature hash was empty. The error message was in the raw log. It just didn't make it through the data pipeline to the classification prompt. A data gap, not a classification error.

I also ran a rules-only baseline. No LLM, just regex and stats. On the 4 repos where both models agreed everything was flaky, rules agreed too. The LLM earned its $0.22 on the 56 disagreements: cases where the error signature was empty or generic, the failure rate was 14/14, and rules said "flaky" while the cloud model said "real bug." Rules see the missing error and hedge. The model sees 14/14 deterministic and commits.

The Bug That Was Producing Fake Verdicts

First time I ran the agent with a Qwen3 reasoning model, every classification came back FLAKY. Every single one. I thought the model was just conservative.

It wasn't.

Qwen3 with thinking enabled spends its entire token budget reasoning before emitting the JSON. I had max_tokens hardcoded at 512. The model used all 512 tokens thinking ("Okay, let's analyze this. The test failed 14 times. The error signature is empty. Wait, let me reconsider...") and never produced the JSON. Response truncated. JSON parse failed. Fallback returned FLAKY.

The classified_by prefix was llm:, the same prefix used for real verdicts. No way to tell a real model verdict from a silent fallback. The tool was lying to me, and the lies looked like real answers.

Fix: configurable max_tokens, finish_reason detection, 4 distinct prefixes for auditability, and a --no-thinking flag that disables Qwen3's reasoning mode entirely. 79 completion tokens instead of 4,096. 16 seconds instead of 214. 13× faster, same or better accuracy.

What Most CI Failures Actually Are

8 of 21 repos had zero test failures in their CI logs. Not because their tests are perfect. Because their CI "failures" were in other jobs. Lint. Build. CodeQL. Dependabot. The test jobs passed.

GitHub Actions runs multiple jobs per workflow. A run can "fail" because a lint job failed while the test job passed. The tool handles this gracefully (clean report, zero tests classified, no crash) but it means you need to download enough runs and the right workflows to find actual test failures.

The Cost

Entire field study: $0.22 in cloud LLM calls. 303 calls across 8 repos. Local model free. Cloud model $0.0007 per call.

For context: developers spend $2,250/month repairing flaky tests (Leinen et al., ICST 2024). Rerun builds across 1,960 GitHub Actions projects accumulated 339 years of waiting time (Ge & Zhang, 2025). The study cost less than a coffee.

What I'd Do Differently

Filter by job type at download time. I downloaded 2,000+ irrelevant log files from lint/build/CodeQL jobs that had nothing to do with tests.

Pass per-test stats, not all-test stats. Two repos produced prompts exceeding any model's context window because the cross-run context included all 14,000 test stats instead of just the one being classified. Django's prompt was 9.8 million characters.

Run models sequentially. The local MLX endpoint is single-threaded. Running rubocop concurrently with other repos caused 74 out of 88 calls to fail with connection pool exhaustion.

Test parsers against real logs before running full analysis. GitHub Actions prefixes every line with a timestamp. Jest uses PASS not PASSED. Vercel has a suite name prefix. I discovered all of these the hard way.

The Honest Part

I only manually validated 1 of the 56 REAL_BUGs. The pattern (14/14 identical failures) strongly suggests the rest are also real bugs, but I haven't checked each one. A proper accuracy study would sample 20+, read the logs, compare against human judgment.

The local model is a 9B parameter model, 4-bit quantized. Not surprising it misses things a larger cloud model catches. But it's free, local, private. Right approach: run local first, then run cloud on the disagreements.

8 repos produced zero results because their CI failures weren't test failures. The tool didn't crash. It reported clean CI. But you need to pick repos where test jobs actually fail.

The Takeaway

56 REAL_BUGs is the headline. The deeper finding: the model you choose to classify CI failures changes what you find. A free local model said everything was flaky. A $0.22 cloud model found 56 real bugs. Same data, same prompts, same parser. Different model, different truth.

If you're rerunning flaky tests without diagnosing them, some of those "flaky" tests are real bugs. You're shipping regressions because the CI signal is noise. The fix isn't more retries. It's reading the logs.

pip install ai-flake-sleuth

Repo: github.com/deghosal-2026/ai-flake-sleuth

When a test goes flaky, what's your first move — quarantine and ship, fix it now, or bump the retry threshold? And have you ever gone back to check whether the thing you quarantined was actually a real bug?

i built a tool that tracks what AI tasks actually cost. the real number surprised me.

Debashish Ghosal — Fri, 17 Jul 2026 04:13:43 +0000

i built a tool that tracks what AI tasks actually cost. the real number surprised me.

you know how much your LLM costs per token. you probably don't know what it costs per task.

I didn't either. that's the embarrassing part. I've been routing LLMs across tiers for weeks — writing about it, presenting it — and I was tracking per-call cost like everyone else. a $0.001 call here, a $0.05 call there. neat numbers. wrong numbers.

here's the real math. you route a code task to your cheap workhorse model at $0.001 per call. it fails. you retry. it fails again. third try, still failing. you escalate to your architect model at $0.05. that one works. what did the task cost?

your dashboard says $0.001. or maybe $0.05. neither is right. the task cost $0.053 — three failed cheap calls plus one expensive call. that's the real number. and I couldn't find a single tool that tracks it.

so I built one. ai-tierforge. open source, MIT, two runtime deps. it tracks cost per completed task — retries, escalations, failed loops, all of it.

the field test

I didn't want to ship this without proving it works. so I ran it against real GitHub data — issues, PRs, and source code from FastAPI. not "write a fibonacci function." real bug reports with real labels. real pull requests with real version bumps. real source files that cut off mid-signature.

about 100 LLM calls across 4 test runs. the final run: 33 calls, 8/8 scenarios passed, $0.01 total. 44.8% savings vs sending everything to one model.

but the numbers aren't the interesting part. the interesting part is what broke.

it took 4 runs to get it right

this is the part nobody talks about. the first run: 5/8 passed, found 5 bugs. the second: 6/8, found 2 more. the third: 6/8, found 1 more. the fourth: 8/8, found nothing.

the first two runs used synthetic prompts — "write a fibonacci function." those passed fine. then I switched to real data and everything broke. rate limits I didn't expect. a DOWNGRADE cycling bug where the router kept switching tiers in a loop. a global retry budget that got exhausted by workhorse retries before the architect tier could even try. stale code routing that made the savings numbers lie.

I thought "write a fibonacci function" was a reasonable test prompt. it's not. real PRs have variable length. real issues have labels that confuse the model. real rate limits kick in after 10 calls. you need real data or you're lying to yourself about what works.

that was the biggest lesson. not the routing logic, not the cost tracking. the fact that synthetic tests validate code paths and real data tests validate the system. they're different things. I didn't appreciate how different until the third run.

the architect model found a real bug in a real PR

gpt-5-nano reviewed fastapi PR #16018 and caught something. the title said "bump mcp from 1.26.0 to 1.28.1" but the body quoted v1.28.0 release notes. version mismatch. real catch, real PR, $0.001.

I didn't expect that. I expected the cheap model to handle code review fine and the architect model to handle specs. I didn't expect the architect to find a real bug that a human reviewer might miss. and I definitely didn't expect it to cost less than a cent.

the cheap model surprised me too. when I fed DeepSeek a truncated FastAPI source file, it didn't hallucinate the missing code. it said "the code cuts off mid-signature" and suggested reviewing the full file. that's the behavior you want from a workhorse — not confidence, honesty. a model that says "I can't see the full picture" instead of making something up.

the budget downgrade

this is the feature I'm most proud of, and it's not the routing.

two calls, same scope. call 1 went to the architect model ($0.001). call 2 triggered a budget DOWNGRADE — accumulated spend exceeded the limit, so the router automatically dropped to the workhorse ($0.0004). both calls succeeded. call 2 cost 60% less because of the downgrade. no manual intervention.

route    → architect / gpt-5-nano           (matched task_type 'spec')
failover → workhorse / deepseek-v4-flash    (budget: per_task limit exceeded)
route    → workhorse / deepseek-v4-flash    (task completed)

you set a limit, the router drops to a cheaper tier when you hit it. your agents don't need to know it happened. that should be boring and automatic. now it is.

the metric nobody tracks

per-call cost is easy to measure. it's also misleading. cost per completed task is harder to measure but actually correct.

but there's a third metric I don't see anyone tracking: escalation rate. what percentage of your tasks required the cheap model to fail over to the expensive one?

if 80% of your tasks escalate, you haven't built a router. you've built a system that pays for a cheap attempt before every expensive call. that's worse than just using the expensive model from the start.

FrugalGPT (Stanford) showed cascade routing can save up to 98%. but they also showed it only works if you monitor escalation rate. RouteLLM (UC Berkeley) achieved over 2× cost reduction — but tracked no cost-per-task data.

I track escalation rate as an SLO. if it's too high, your routing isn't saving money. that's the signal to rebalance.

what I'm still figuring out

the free tier hit 429 rate limits after about 10 rapid calls. "free" means $0 per call, not unlimited usage. I had to switch to the paid tier for reliable testing. for production, always configure a paid fallback.

cost-per-task only works if you can define "task." the router needs your agent code to label calls — "this is a code task" or "this is a spec task." if your labels are wrong, your cost data is wrong. this isn't a tool problem, it's a discipline problem. but it's real.

no dashboard. no async API. JSONL logs and CLI output work today. if you need either of those, this isn't your tool yet.

the number

$0.01 for 33 LLM calls on the final run. ~100 across all 4 runs. real FastAPI code reviewed. real PRs analyzed. real issues categorized. a real version mismatch found. for a penny.

the 44.8% savings is real and replicated — synthetic data showed 47.6%, real data showed 44.8%. that 2.8% gap is just real-world token variance. with a frontier model as the architect instead of gpt-5-nano, the savings would be 97%.

but the headline isn't the penny. it's the 44.8%. that's what tier routing buys you vs sending everything to one model.

repo: github.com/deghosal-2026/ai-tierforge. pip install ai-tierforge. MIT.

it's also the companion to ai-loopguard, my circuit breaker for agent loops. loopguard decides when to escalate. ai-tierforge decides where and tracks how much it cost.

discussion

I want to hear from people running multi-model setups in production, not just experimenting:

how are you tracking LLM costs today? per-call dashboards? per-token? has anyone tried per-task accounting? what does that look like in your stack? I couldn't find a single tool that does this, which either means I'm missing something or nobody's built it yet.

what's your escalation rate? do you know what percentage of your tasks escalate from cheap to expensive? if you don't know, how would you even measure it? this feels like the metric that tells you whether your routing strategy is working or just burning tokens.

has anyone else tested their routing against real data vs synthetic? I was surprised how much broke when I switched from "write a fibonacci function" to real GitHub issues. what did real data expose in your system that synthetic tests missed?

budget enforcement — auto-downgrade or hard stops? the DOWNGRADE feature (automatically switching to a cheaper tier when budget is exceeded) is the thing I'm most curious about. does anyone have experience with this in production? is it useful or just a nice idea?

drop a comment with your current setup. I'm especially interested in what's working and what's not.

Originally published on Hashnode.

i've been building platforms first for 25 years. i think it's wrong now.

Debashish Ghosal — Wed, 15 Jul 2026 03:55:40 +0000

i've been that person. standing in front of leadership with an 18-month architecture diagram, explaining why we need six months of infrastructure before a user touches a single feature.

and it made sense. for 25 years it made sense.

writing boilerplate was expensive. every feature came with a tax — database migrations, routing config, auth wiring. build a shared platform first, pay that tax once. the roadmap justified the investment.

then i saw a stat that wouldn't leave me alone. roughly 60% of features on a six-month roadmap are obsolete by launch. not slightly off. obsolete. the customer's problem shifted. the market moved. you spent six months building a precise answer to a question nobody asks anymore.

the longer you invest before showing something real, the more expensive it is to admit you were wrong. so you don't. you ship the wrong thing and call it "on schedule." i've done it. i've watched it happen.

AI didn't create this problem. but agents are making it impossible to ignore.

the 82-point gap

mckinsey's 2025 survey: 88% of organizations use AI. only 6% see real bottom-line impact. that 82-point gap isn't about tools. everyone has the same tools.

but something shifted in their may 2026 report. they describe agents working overnight — enriching requirements, generating code, packaging outputs for morning review. they call it the "24-hour sprint." leading organizations see 3-5x productivity with 60% smaller teams. a product owner logs in at 9am and finds a feature went from requirements to tested code overnight. nobody worked late. agents did.

that's not autocomplete. that's a different delivery model.

and here's what most teams miss: it only works when the work is small, bounded, and complete. agents need to know where a task starts and ends.

horizontal platform architectures don't give them that.

the codebase is the prompt

jeremy d. miller built wolverine for .NET. in june 2026 he wrote: "the structure of your codebase is now, effectively, part of the prompt."

think about that. when you hand an agent a task in a traditional codebase, the feature is spread across six folders. controller here. request type there. logic somewhere else. database access in another folder. the agent opens all of them, loads them into memory, figures out which parts matter. most of what it loads is irrelevant.

imagine asking someone to fix a light switch, but first they have to read the wiring diagram for the entire house.

now scale that to a horizontal platform build. you're asking the agent to build infrastructure that five future features will depend on. massive context, no concrete boundaries, nothing to test against. integration issues surface months later when the abstraction doesn't fit.

horizontal builds give agents the worst possible working conditions.

vertical slices fix this

jimmy bogard came up with this. instead of organizing code by technical layer, organize by feature. one folder for "create shipment." inside: the request, validation, logic, database access, response. all of it. together.

the agent opens one folder. everything it needs. nothing irrelevant. small scope, clear boundaries, complete end-to-end. you can put it in front of a real user and get feedback in days.

and this gets better as models improve. i assumed larger context windows would make code organization matter less. but a 2026 guide put it bluntly: "a mid-tier model with good context outperforms a frontier model with a sloppy window." more context means more noise, not more signal.

medtronic — FDA-governed medical device software — is advocating for this. their principal systems engineer wrote that a vertical slice is "structurally, exactly what an agentic model needs to operate reliably." without it, agents produce "outputs that are plausible and well-structured and wrong in ways that are genuinely difficult to detect."

agents need boundaries. vertical slices are boundaries. horizontal platforms are the opposite.

the SDLC doesn't go away

i've written before about how AI didn't kill the SDLC. i still believe that. planning, delivery, review — they're still the spine. what changes is the shape.

planning gets tighter. you still write a spec. but instead of an 18-month platform, you spec one slice. section numbers, stable IDs, explicit non-goals. the spec becomes the contract the agent implements against. "this doesn't match §2.2" is ten seconds. "i think this is wrong" is ten minutes.

delivery goes vertical. one complete slice — request to response, database to UI. the agent builds it overnight. the slice is testable end-to-end on day one. the integration test isn't a phase at the end — the slice IS the integration test.

review splits in two. spec verification: does the output match the spec? mechanical, fast. and value verification: does this actually solve the user's problem? the check that used to happen months in now happens in days.

anthropic's 2026 report: developers use AI in 60% of their work but can "fully delegate" only 0-20% of tasks. agents compress execution time. they don't compress ambiguity or accountability. review is where the human stays.

what one day looks like

morning. you log in. the agent finished slice A overnight — complete, tests passing, sitting in a PR. you review against the spec. section numbers. ten seconds. approve or request changes.

while you review, you hand the agent slice B. same pattern. spec with non-goals. the agent starts building. you don't supervise — the spec is the briefing.

afternoon. you put slice A in front of a real user. watch what they do. learn whether the feature solves the problem you thought.

meanwhile the agent builds slice B. it's not blocked by your user review — slice B depends on slice A's interface, which the spec defines. not on whether the user liked it.

evening. you take what you learned and write the spec for slice C. queue it up. go home. the agent starts overnight.

nobody waits. the work is parallel because the handoffs are artifacts, not conversations.

this doesn't work with a horizontal platform. what do you put in front of the user? an auth service? there's nothing to test. no feedback loop. the agent works alone for months.

the part i haven't figured out

what stops a team from building slices forever and never committing to anything real?

i keep thinking the answer is a budget clock. before you start, you decide: N attempts — maybe three, maybe five — and a fixed budget. at each attempt: did this prove a real user gets value? if yes, fund the next. if no, stop.

it's what VC does for startups. finite runway. clear gate. prove it or stop.

but i haven't tested this. i don't know what N should be. and most companies say they want fast failure but punish teams whose experiments fail. that's a culture problem, not a process one.

what i don't know yet

i haven't run this playbook end to end. i'm watching it, thinking about it, starting to act on it. not pretending i've got it figured out.

agents don't write production-ready code. they write plausible code. 29-46% of developers trust AI output. the SDLC processes — spec, review, checkpoints — are what keep them honest.

the gains are uneven. clear wins on scaffolding, tests, migrations. complex interconnected work? mixed.

you can create a mess. five features with five auth patterns = five systems to maintain. extract when you have evidence of shared need, not when you predict it.

discussion

i want to hear from people who've actually tried this, not just thought about it:

have you moved from horizontal platform builds to vertical slices? what broke? what got better? did your team resist it?

are you running anything like the 24-hour sprint? agents building overnight, humans reviewing in the morning? or does that sound insane to you?

what stops your team from building forever without shipping? budget clock? leadership pressure? something else?

if you're in a regulated industry — healthcare, finance, government — does vertical slicing feel possible? or does compliance make it impractical?

does the "codebase is the prompt" framing resonate? have you seen agents struggle with your codebase structure specifically?

i don't have this figured out. i'm sharing it because i think it's the conversation engineering teams need to be having. if you've started moving this direction — or if you think i'm wrong — tell me why.

i tested an ai incident commander against 15 real outages — 88% pass rate

Debashish Ghosal — Wed, 15 Jul 2026 03:55:17 +0000

i've been the incident commander who forgot to write down the first 20 minutes of the timeline because i was too busy reading logs. more than once. the war room is chaos — five engineers pasting logs, someone asking if the deploy from 30 minutes ago is related, nobody documenting anything. you start logging events in a doc while reading error logs while drafting a stakeholder update while deciding whether to rollback.

you're the bottleneck. not because you're bad at your job — because you're doing four jobs at once.

i got tired of watching smart people spend their incident energy on documentation instead of decisions. so i built ai-incident-commander — a CLI tool that handles the mechanical parts. timeline, updates, remediation research, postmortem draft. you make the calls. it does the paperwork.

runs on your laptop with a local LLM. no API keys, no cloud, no docker.

github.com/deghosal-2026/ai-incident-commander — MIT licensed.

what it does

one command:

pip install git+https://github.com/deghosal-2026/ai-incident-commander.git
incident-commander simulate --scenario db-connection-pool --auto-approve

8 pre-built scenarios ship with it. database connection pool, bad deploy, memory leak, cert expiry — the usual suspects. no real data needed to try it.

for actual incidents, you point it at a directory with your alert, logs, messages, and github PRs. it outputs 10 markdown files: timeline, stakeholder updates, comms blocks you can paste straight into slack, remediation suggestions, a blameless postmortem, and a cost report.

the safety part was the real engineering. three points in the pipeline where the graph pauses and waits for you to say yes — stakeholder update, remediation, postmortem. the AI never ships anything without approval. every remediation comes with a citation. suggestions below 0.7 confidence get suppressed. the postmortem prompt enforces blameless language. all AI content gets labeled [AI-GENERATED — review carefully].

and it never executes anything. it predicts outcomes as text — "rollback should restore service within 2 minutes" — but never touches production. that's your call.

the field test

i grabbed 15 real incidents from public postmortems — cloudflare, gitlab, github, aws — plus structured datasets from IntelligentDDS and OpenRCA2. ran them through the tool with a small cheap LLM (DeepSeek V4 Flash, $0.14/$0.28 per 1M tokens). total cost: about $0.02.

the results were genuinely surprising.

six criteria passed 100%: blameless framing (never blamed a person across 15 incidents), graceful degradation (survived missing logs), no hallucination (all events traceable to input), cost tracking (every call logged), timeline coherence (non-empty, chronological), action item quality (every item had owner + priority).

citation integrity was 93% after a one-line normalization fix.

overall: 106 out of 120 checks. 88.3%.

the part that surprised me

the first run scored 61%. i thought the tool was broken.

then i looked at the failures and realized — my tests were wrong, not the tool.

i was comparing LLM-generated timelines against ground truth from real postmortems. but the tool builds its own timeline from input data. different phrasing, different granularity. of course it doesn't match word-for-word.

changed the timeline test to check coherence (is it non-empty and chronological?) instead of exact match. 0% became 100%.

same thing with action items. i was comparing wording when i should have been checking quality (does it have an owner? a priority?). changed the test. 0% became 100%.

the lesson that changed how i think about AI testing: don't test LLM output against human-written ground truth. test for coherence, quality, and safety. the LLM will always phrase things differently — and that's fine.

i'd been treating the LLM like a search engine (did it return the right answer?) when i should have been treating it like a junior engineer (did it produce something safe and useful that a human can work with?).

where it needs work

i'm not going to pretend this is perfect.

your runbooks need to be good. the RAG retrieval is only as good as what you feed it. stale runbooks = bad suggestions.

RCA quality scales with model size. the small model nails the category ("this is a DNS failure") but doesn't always nail the exact causal chain. a bigger model would help — i have Qwen3.6-35B downloaded locally, haven't tested it yet. that's next.

the LLM doesn't always format citations correctly. generates them, but sometimes missing the prefix. fixed with normalization, but the real fix is prompt engineering.

15 incidents isn't enough. i built infrastructure for 124 fixtures from 4 public datasets but only ran 15. the batch runner has resume support — broader validation is next.

it costs nothing

# local — free
export LLM_MODEL=ollama/qwen2.5-coder:7b
export LLM_BASE_URL=http://localhost:11434/v1

# cloud — ~$0.01 per incident
export LLM_MODEL=deepseek-v4-flash
export LLM_BASE_URL=https://opencode.ai/zen/v1
export LLM_API_KEY=your-key

15 real incidents. $0.02 total. local MLX models work too — completely free on apple silicon.

discussion

i want to hear from people who've actually been incident commanders:

how do you build timelines today? google doc? dedicated tool? slack scrolling? i want to know the real workflow before i try to improve it.

how long does postmortem drafting take you? and do you use COE format or something else?

would you trust an AI-drafted postmortem? even with the labels and blameless framing? what would it take?

what LLM would you use for this? local for privacy, or is cloud acceptable for incident data?

your input shapes v0.2.0. the best ideas will come from people who've been paged at 2am.

repo: github.com/deghosal-2026/ai-incident-commander — star it if on-call has ever burned you. good first issues open if you want to contribute.

i spent 3 days on a rag pipeline and the embedding model didn't matter at all

Debashish Ghosal — Sun, 12 Jul 2026 06:07:08 +0000

i spent 3 days on a rag pipeline and the embedding model didn't matter at all

it's 2am. PagerDuty fires. you're on-call.

you open the wiki. search for "CrashLoopBackOff." get 47 results. none of them are the runbook you need. you try "pod stuck pending." different results, equally useless. you tab over to Slack and ask "anyone remember which runbook covers Kafka consumer lag?" and wait.

I've done this more times than I can count. I've watched engineers grep through Confluence at 3am, tab through 20 browser windows, and page someone who retired last year because they wrote the runbook in 2019. first-response time is slow, and resolution quality depends entirely on who's on call. the senior engineer has the runbooks memorized. the new hire is reading them for the first time — during a SEV1.

RAG should fix this. retrieve the right runbook section, generate a grounded answer, cite the source. three seconds instead of fifteen minutes.

so I built one. and I discovered something I didn't expect: the biggest lever in the pipeline wasn't the embedding model, the LLM, or the vector database. it was how I split the documents into chunks.

what I built

RAGoncall — a local-first RAG assistant that indexes 55 markdown runbooks and answers on-call queries with sourced citations. four delivery channels off the same backend: CLI, Discord bot, Streamlit dashboard, and a FIFO alert pipeline.

the corpus is 55 markdown runbooks from four public sources: Kubernetes troubleshooting guides, Grafana OnCall docs, Google SRE book chapters, and community runbooks. the embedding model is all-MiniLM-L6-v2 — local, free, 384-dim. the LLM is pluggable — local Llama-3.2-3B, OpenAI, or Anthropic — swappable via a config flag with zero code changes.

but before I wrote a single line of retrieval code, I did something most RAG tutorials skip: I built an eval set.

the eval set nobody talks about

fourteen question-answer pairs. real questions an on-call engineer would ask, each with a known correct source file and line range. stored as JSON:

{
  "id": "k8s-001",
  "question": "Why is my pod stuck in Pending state and how do I debug it?",
  "answer": "A pod stuck in Pending means the scheduler cannot place it...",
  "source_file": "corpus/kubernetes/debug-pods.md",
  "source_lines": "11-19"
}

fourteen pairs is small. I know it's small. but it's enough to measure the difference between chunking strategies — which is the whole point. without this eval set, "it looks right" is the only quality signal you have. that's how teams ship RAG that hallucinates in production.

the eval harness computes MRR (Mean Reciprocal Rank) — how high the first relevant result appears. rank 1 gives 1.0, rank 2 gives 0.5, not in results gives 0. when you're on-call at 2am, you read the first result, not the fifth. MRR is the metric that matters.

the chunking experiment

same embedding model. same vector DB. same LLM. same corpus. same eval set. the only variable: how I split the documents.

header-split chunking hit MRR 0.9524. recursive chunking hit 0.9048. fixed-size chunking hit 0.8893. all three achieved perfect Recall@5 — the correct document was always in the top 5. the difference was MRR: how high the right chunk ranked.

six points of MRR from chunking alone. the embedding model, LLM, and vector DB moved it by zero.

I didn't expect this. I assumed the embedding model would be the biggest lever. that's what every RAG tutorial implies — "choose the right embedding model." so I tested it. the embedding model was never the bottleneck. the chunking was.

why header-split won

runbooks are structured by headers. ## Diagnosis. ## Resolution. ## Escalation. when you split on those boundaries, each chunk is a complete section — the entire resolution procedure, not a fragment that starts mid-sentence. split on headers first, then merge small sections up to 1.5× chunk_size so you don't get tiny one-line chunks. any section still too big gets a fallback recursive split.

fixed-size chunking doesn't respect any of this. a 600-character resolution procedure gets split into a 500-char chunk and a 100-char fragment. neither contains the complete answer. the embedding captures half the procedure. retrieval returns half the procedure. the LLM generates an answer from half the procedure.

recursive chunking does better — it respects paragraph boundaries — but it can still split mid-section when a section exceeds 500 chars. it loses ~5% MRR vs header-split because relevant content sometimes straddles chunk boundaries.

the lesson isn't "header-split is always best." the lesson is: your chunking strategy has to match your document structure. for runbooks, split on headers. for code, split on functions. for legal docs, split on clauses. the embedding model matters less than the chunking — that surprised me.

what broke

this is the section I wish more RAG posts had. the patterns are easy. the edge cases are the real education.

YAML frontmatter — a 38-point MRR drop. four out of fourteen eval questions targeted Grafana OnCall runbooks. these files start with YAML frontmatter — ---\ntitle: "Grafana OnCall"\ntags: ["oncall"]\n---. the header-split chunker didn't know about frontmatter. it treated it as regular content and merged it with the first markdown section. the semantic signal got diluted. retrieval missed the correct source file for all four Grafana questions. MRR for those 4 files: 0.571. MRR for the other 10: 0.9524. a 38-point swing from a production edge case that no toy corpus would expose.

the fix is a frontmatter-aware splitter that strips ---\n...\n--- before chunking. I didn't build it. it's a known limitation. this is the kind of thing that only surfaces when you test against real documents — not the curated test corpora that most RAG tutorials use.

no incremental re-indexing. index --rebuild drops and recreates the entire collection. for a production system where runbooks update daily, you need file-watching and incremental upsert. fine for 55 files. not fine for 5,000.

cold model load. first request with local Llama-3.2-3B takes about 30 seconds for the model to load into memory. subsequent queries are fast — 5 to 10 seconds. a cloud model (gpt-4o-mini, claude-3-haiku) has no cold start and 2-5s latency. the cost difference is negligible: about $0.0003 per query either way.

the eval set is 14 pairs. enough to measure chunking differences. not enough to claim production-grade retrieval quality. a real system needs 50+ pairs covering edge cases, multi-hop queries, and ambiguous questions. I'm honest about this.

the part that actually worked

on June 27, I tested the full pipeline live from my phone. I opened the Discord mobile app, sent /query "CrashLoopBackOff", and got back a sourced answer with kubectl describe pod and kubectl logs --previous steps. the bot posted a rich embed with a clickable source link. I tapped 👍. it logged to data/feedback.jsonl.

five minutes of wiki-diving → three seconds to a sourced answer. from a phone.

the 👍/👎 feedback is the cheapest improvement mechanism in the entire system. zero compute. zero labeling. structured data. the dashboard's "knowledge gaps" tab shows runbooks that get queried but receive 👎 — those are the runbooks that need rewriting.

what I'd tell you if you're building RAG

build the eval set first. before the pipeline, before the vector DB, before the LLM. 10-15 question-answer pairs with known correct source files. if you can't measure retrieval quality, you can't improve it, and you definitely can't ship it to someone making production decisions at 2am.

then test 3 chunking strategies on that eval set. don't assume fixed-size is fine because the tutorial used it. don't assume your embedding model is the bottleneck. measure MRR for each strategy and pick the one that fits your document structure.

one more thing: swapping the LLM doesn't change MRR. the LLM generates the answer from the chunks the retriever returns. if the retriever returns the wrong chunks, the best LLM in the world can't save you. retrieval quality is upstream of generation quality. fix your chunking before you fix your LLM.

discussion

I want to hear from teams running RAG in production, not tutorial experiments:

what chunking strategy are you using? if it's fixed-size, have you benchmarked it against header-split or recursive on your own eval set? I'm curious whether the 6-point MRR swing holds across different document types.

how big is your eval set? my 14 pairs are enough to measure chunking differences but not enough to claim production quality. what's the minimum you'd trust before shipping RAG to a user who makes decisions at 2am?

has anyone hit the YAML frontmatter problem? it seems niche, but any document with metadata headers (Jekyll, Hugo, Obsidian, Grafana) will have it. did you build a frontmatter-aware splitter, or just strip it before indexing?

thumbs up/down feedback — is anyone using this for re-ranking? I'm collecting the data but haven't built a re-ranker yet. the 👍/👎 log could train a cross-encoder re-ranker that moves well-received answers up. curious if anyone has done this and whether it moved MRR.

if you're building RAG for production, drop a comment with your eval set size, chunking strategy, and MRR. especially if your MRR is higher than 0.9524 — I want to know what you did differently.

my ai agent ran for 6 hours on a 2-minute task and cost me $200

Debashish Ghosal — Sat, 11 Jul 2026 19:43:33 +0000

my ai agent ran for 6 hours on a 2-minute task and cost me $200

it's 2am. your phone buzzes. the on-call alert says your AI agent pipeline has been running for 6 hours on a task that should take 2 minutes.

you open the logs. the agent changed code. tests failed. it changed something else. a different test failed. it reverted. the original test failed again. 500 iterations. zero progress.

your LLM API bill for this one task: $200. for a task that didn't complete.

this isn't hypothetical. it's the rite of passage for anyone building agentic systems. non-deterministic systems against deterministic environments — code, APIs, databases — fall into stuck states. and they burn real money.

the dirty secret of cheap AI models: they're cheap per call, but expensive per completed task. a $0.001 call that loops 10 times costs more than a $0.05 call that solves it once.

in traditional software, circuit breakers prevent cascading failures. when a service is failing, the circuit opens, traffic stops, and the system degrades gracefully. why are we deploying AI agents into production without them?

I looked for a library that detects stuck agent loops and escalates to a stronger model. LangGraph, CrewAI, AutoGen — they're frameworks, not failure detection. LangSmith, LangFuse — observability, after the fact. cost trackers — they tell you what happened, not stop it. nobody sits between frameworks and cost trackers. a quality layer that drops into whatever you've already built.

so I built one.

what I built

ai-loopguard — a drop-in circuit breaker for LLM agent loops. it detects stuck patterns and escalates to a stronger model before costs spiral. 4 lines of code. works with LangGraph, CrewAI, or raw Python.

from ai_loopguard import Guard

guard = Guard(escalation_model=gpt4_model, workhorse_model_name="qwen2.5-coder")

@guard.protect
def agent_step(state):
    return cheap_model.generate(state)

that's it. if agent_step fails 3 times, loopguard packages the context, sanitizes it, redacts secrets, escalates to your configured cloud model, logs the event, and returns the escalated output. the loop continues. 4 lines to add, zero refactoring.

the repo is here: github.com/deghosal-2026/ai-loopguard — MIT licensed, 436 tests, 97.3% coverage.

the two problems nobody talks about

building a circuit breaker for AI agents is two hard problems, not one.

problem 1: how do you detect a loop in a non-deterministic system?

the naive approach is a counter: max_iterations=10. kill the agent after 10 steps. but that can't distinguish between an agent making progress on a hard task and one spinning in circles. and it completely misses the most insidious pattern: oscillation — the agent bouncing between two states. test passes. agent changes code. test fails. agent reverts. test passes. same code changed. test fails. the counter goes up, but the agent goes nowhere.

loopguard uses pattern detection, not counters. error clustering — same exception type and message across N consecutive steps. test-failure cycles — same test fails N times or oscillates pass→fail→pass→fail. schema validation failures — output fails schema validation N consecutive times. custom callbacks — your own domain heuristics, wall-clock timeout, token budget exceeded.

no self-grading. no "let the model decide if it's stuck." triggers are deterministic or user-defined.

problem 2: what do you do when a loop is detected?

most tools crash or log a warning. loopguard handles it with an escalation layer. model upgrade — package the context, sanitize it, escalate to a stronger model. human-in-the-loop — pause execution and ask "agent is stuck, escalate to GPT-4?" or fail-open — if loopguard itself fails, your agent keeps running. the circuit breaker never takes down your system.

does it actually work?

I ran a field study. 15 tasks across 3 external open-source repos: SWE-agent (~14k stars), Aider (~46k stars), LangGraph (~100k stars). same tasks, with and without loopguard.

without loopguard: 0 out of 15 tasks completed. the agents looped, burned tokens, and failed.

with loopguard: 15 out of 15 tasks completed. every task that failed without loopguard succeeded with it. every escalation produced a valid result. average integration: 4 lines of code.

detection overhead is 2.75 microseconds — less than a CPU branch misprediction. you'll never measure the cost of loopguard watching your agent. until it saves you.

the security part most teams skip

when your agent gets stuck, its state might contain API keys, user PII, or malicious instructions from a web page it scraped. loopguard treats all agent output as untrusted and does three things before sending anything to the escalation model.

context compression — not a full dump, just first + last + summary, bounded by a token limit. sanitization — clear system vs user separation, delimiters around agent-produced content. redaction — built-in patterns for OpenAI keys, AWS keys, GitHub tokens, plus your own regex and field names.

every escalation event log includes sanitized: true and redacted_fields: ["api_key", "password"] — you can audit what was stripped.

there's also a threat nobody thinks about: economic DoS. if your agent loop is buggy and loopguard escalates on every step, you could rack up $50 before you notice. the max_escalations_per_run cap prevents that — one escalation per guarded execution by default, configurable up to 10.

what I learned building this

the biggest surprise was that nobody had built this yet. LangGraph gives you DAGs, conditional edges, parallel stages, approval gates. it does not give you stuck-loop detection, test-failure cycle detection, automatic escalation, or cost-per-completed-task tracking. those are the things that actually matter when your agent is in production.

the second surprise: the cost metric everyone tracks is wrong. most cost trackers tell you cost per call. that's the wrong metric. a task that loops 10 times at $0.001/call costs $0.01. a task that succeeds once at $0.05/call costs $0.05. the "cheap" model is 5x more expensive per completed task. loopguard tracks escalation rate, cost per completed task, and routing vs failover — the metrics that actually tell you whether your routing config is working.

the third surprise: fail-open is non-negotiable. if the circuit breaker itself crashes your agent, it's worse than not having one. three modes — re-raise the original exception, return the last successful output, or return a sentinel. every fail-open event is logged. you'll know loopguard failed even if your agent didn't.

what's in the repo

this wasn't built by vibe-coding. the repo includes a full PRD (886 lines), SPEC (1,511 lines), and WBS (1,247 lines). 12 milestones, all complete, with checkpoints and LLM routing strategy per task. 436 tests across 4 layers: unit, integration, e2e, field study. ruff clean, mypy strict clean, pip-audit zero vulnerabilities. CI matrix: Python 3.10/3.11/3.12 across ubuntu/macos/windows.

good first issues if you want to contribute: a hallucination_cycle trigger (agent claims to fix something it already attempted), LangGraph Send API support for parallel agent fan-out, AutoGen/Semantic Kernel/LlamaIndex integrations, CLI improvements like loopguard watch for live log tailing.

full contributing guide here: CONTRIBUTING.md. 3 commands to get started, full dev toolchain included.

discussion

if you're running AI agents in production, you will hit a stuck loop. not a question of if — it's when. the question is whether you'll have a circuit breaker when it happens.

I want to hear from people actually running agents in production, not experimenting:

how are you handling agent loops today? rigid iteration limits? manual monitoring? just hoping it doesn't happen? I want to know what your current stack looks like.

what triggers are you missing? the 4 current triggers came from my own experience. your agents might get stuck in ways I haven't seen. hallucination_cycle is the most requested — if you've seen hallucination patterns, your field experience would shape the detection logic.

is auto-escalation the right default? or should interrupt mode be the default? I chose auto for production throughput, but I might be wrong.

are you running multi-agent systems? the CrewAI integration gives each agent its own state. would you want cross-agent escalation — agent A gets stuck, agent B picks up the task?

your input shapes the v0.2.0 and v0.3.0 roadmap. the best ideas will come from people running agents in production — not from me sitting in a vacuum.

star the repo, open an issue, drop a comment. github.com/deghosal-2026/ai-loopguard

i've been doing sdlc for 25 years. ai didn't kill it. it made it the coordination layer.

Debashish Ghosal — Sat, 11 Jul 2026 05:27:25 +0000

i've been doing sdlc for 25 years. ai didn't kill it. it made it the coordination layer.

I've been shipping software for 25 years. design docs reviewed by thirty engineers before a line of code. spec reviews where someone meaner than me tore apart my threat model. change-management boards. SOX. on-call rotations where a bad 2am deploy meant someone's flight was missed.

so when people say "AI killed the SDLC," I flinch.

not because I'm precious about process. because I've seen what happens when you skip it at scale. the outage postmortem where nobody can find the design doc. the service with 200 endpoints and no contract because "we'll document it later." the migration that failed because two teams read the same requirement differently and nobody wrote down which reading was correct.

AI didn't fix that. AI made it worse — because now you can generate the appearance of rigor (a README, some tests, a docstring) without the substance (a decision, a contract, a traceable requirement). the model writes fluent prose. it doesn't write decisions.

last weekend I built loopguard, an open-source circuit-breaker library for LLM agent loops, using the same SDLC I'd use at any big-tech job: PRD, SPEC, WBS, issues, implementation, review, release. but I put AI agents inside every state.

here's what I learned: the SDLC didn't survive AI by dying. it survived by becoming the state machine that lets humans and agents multitask in parallel.

the setup

the project: loopguard detects when an AI agent is stuck in a loop — same test failing, same error repeating, bad JSON three times in a row — and escalates to a stronger model before the token bill spirals. drop-in decorator. LangGraph, CrewAI, or raw Python.

small library. real engineering: threat model, four test layers, OpenTelemetry hooks, prompt-injection sanitization, Pydantic config, fail-open behavior. 27 milestones. 54 GitHub issues. built over a weekend with a hybrid process.

the interesting part isn't the library. it's how it was built.

I used the same 8-state SDLC every well-run project I've been on has taken: problem, PRD, SPEC, WBS, issues, implementation, review, release. the difference from my old jobs: AI sat in every state. not as an observer. as a participant. I co-wrote the PRD with a model. the model drafted SPEC sections; I redlined. the model proposed the WBS; I approved. models implemented milestones. a model did first-pass code review; I triaged.

what stayed mine — completely, non-negotiably — were the decisions. what's in scope. what's a non-goal. which open question gets which answer. whether a deviation is a bug or an improvement. that's engineering judgment. the model is fast at everything except that, and that's the only thing that matters.

the spec is the one artifact you can't skip

if the PRD is the contract with the user, the SPEC is the contract with the implementer — and the reviewer, and future you. this is the document I see skipped most often in AI-assisted dev, and it's the one whose absence causes the most damage.

the SPEC takes a requirement like "the library MUST detect repeated identical errors" and turns it into pseudocode, data models, trigger evaluation order, Pydantic config schema with constraints, escalation flow as a numbered sequence. it also resolves the PRD's open questions — ten of them, each with a decision and rationale.

without that, the implementing agent makes those calls silently. you find out in review — or in production.

the SPEC is the document the agent is graded against. when the agent drifts — and it will — you don't argue aesthetics. you say "§2.2 says one GuardState per call; your implementation uses a shared one. fix it or amend the spec." ten seconds. the conversation without a spec is ten minutes and ends in "let me think about it."

I've had both conversations thousands of times. the ones with a spec resolve. the ones without fester.

26 bugs in two code reviews — and that's the point

after the agent implemented two milestones, I did a code review the way I've done thousands: read the diff, check against the spec, look for the failure modes I've learned to expect. I filed 26 issues. not one was a style preference. every one cited a SPEC section.

shared mutable state across tasks — the classic distributed-systems bug I've seen in a hundred codebases. non-atomic increments under concurrency. wrong event type for a log record. hardcoded values ignoring configuration. cost tracking that always reported zero because nobody extracted token usage from the model response.

these aren't "the model is dumb" bugs. these are the bugs any engineer makes moving fast without a contract check. the model is a very fast, very prolific junior engineer. you review its work the same way you review a junior's: against the spec, for the failure modes you've learned to anticipate.

one issue was interesting: the agent deviated from the spec, but the deviation was an improvement. I approved it and amended the SPEC. that's the thing about having a spec: it makes deviations visible. without a spec, the agent just does what it does and you can't tell a good deviation from a bug. with a spec, both show up as a diff from expected — and you decide which is which.

the multitasking this enables

here's what actually happened during the build, because state was maintained at each phase: I was reviewing one milestone while an agent built the next while I edited a doc for the release after that. not because I'm clever. because the state was explicit and addressable, so nobody was blocked.

the SPEC section is the briefing. the WBS dependency is the scheduling. the checkpoint is the gate. no standup required.

state is what makes parallelism safe. I've known this for distributed systems my whole career — you can't parallelize work without shared state that's consistent and addressable. same thing applies to human-agent teams. if the intermediate state lives in a chat log or someone's memory, you're sequential no matter how fast the model is. if it lives in a file with IDs and a status field and a git commit, you can have three things in flight.

at planet scale, this is how you ship. you don't have everyone in one room. you have contracts between teams, and each team works against the contract. the SDLC states are those contracts. AI doesn't change the need for contracts — it makes them more valuable, because now one side of the contract can be an agent.

what hurt

I'm not going to pretend this was all smooth. here's what actually hurt:

review overhead was real. 26 issues from two reviews. every one correct and valuable. but triaging, prioritizing, and verifying fixes was a meaningful chunk of my time. if you think "I'll let the agent self-review," you'll miss the bugs I found — shared state across tasks, non-atomic increments, wrong event classes. those aren't syntax errors. they're design errors. the model doesn't catch them because it doesn't hold the whole system in its head the way a reviewer with 25 years of scars does.

the bottleneck is human review throughput, not agent generation throughput. everyone's optimizing generation. nobody's optimizing review. that's the leverage. at big tech, I watched review become the constraint over and over as generation got faster. the same thing is happening with AI, just faster.

the METR randomized controlled trial showed experienced developers were ~19% slower on familiar tasks using AI tools, despite believing they were faster. the hidden cost: review and rework. the Google DORA report found the same thing at org scale. the fix isn't less AI — it's better state management so review is fast and verifiable.

spec drift is silent. the agent shipped a milestone with a design that deviated from the spec. good deviation — I approved it. but I only caught it because I read the PR. no automated check says "the code's control flow matches the SPEC's pseudocode." I want a spec-vs-code diff tool. until that exists, drift detection is a human job, and it doesn't scale.

what I'd tell you

the foundations — write down what you're building before you build it, review against the contract, gate your releases — are not obsolete. they're the thing that makes AI-assisted development engineering instead of gambling.

the SDLC is the context the model needs. a PRD/SPEC/WBS isn't bureaucracy. it's the prompt you write once for the whole project instead of re-explaining in every chat. I've onboarded hundreds of engineers. the ones with a spec ramped in days. the ones without ramped in weeks. the model is the same — just faster.

the traditional SDLC didn't survive AI by dying. it survived by becoming the state machine that lets humans and agents work in parallel on the same project without talking to each other.

I've been doing this for 25 years. this is the most fun I've had. not because the work changed — because the leverage did.

the full artifacts — PRD, SPEC, WBS, 54 issues with SPEC-cited review findings — are in the loopguard repo. the structure is the point. steal it.

discussion

I want to hear from teams doing AI-assisted development at scale, not solo experiments:

is the SPEC still the hardest artifact to get teams to write? I've watched engineers skip it for 25 years — AI makes the skip more expensive, not less. are your teams writing specs before agent implementation, or skipping straight to code? what's the friction?

where is your review bottleneck? the METR and DORA data both say AI slows experienced devs due to review burden. is that matching what you see? are you building review tooling (spec-check agents, diff-vs-spec linters) or accepting the review tax?

per-milestone model routing — is anyone doing this formally? I put the model assignment in the WBS. it worked. but I'm one person. does this hold up when a team of 10 is hitting the same WBS? who owns the routing decisions?

checkpoints as CI gates — who's enforced this? I enforced checkpoints by discipline. in a team setting, you need CI to block transitions. has anyone wired GitHub required checks to WBS milestone completion? I want to steal that pattern.

if you're building AI-assisted dev workflows with real process behind them — not vibe-coding, not blindly trusting the model — drop a comment with what's working and what's hurting. especially what's hurting.