DEV Community: Don Johnson

389 Tests Passed. NIST Still Caught the Bug.

Don Johnson — Sat, 25 Jul 2026 06:17:51 +0000

I gave an AI agent a calculator because I wanted one hard, inspectable point
inside a probabilistic workflow.

The model could interpret the request and explain the result. The calculator
would perform the computation. It seemed like a clean division of labor.

Then I changed one multiplication sign into addition.

The calculator still passed 389 of the 390 tests in its Rust library harness.
The sole failure compared its answer with NIST's certified results for the
Longley regression dataset.

That bothered me more than a completely broken build would have. I had treated
deterministic computation as safer than asking a language model to improvise
arithmetic. But deterministic does not mean trustworthy. A program can return
the same wrong answer forever.

“Source of truth” suddenly felt too comfortable. Before an AI agent delegates
authority to a tool, that authority should be challenged—and remain revocable
by evidence.

The calculator is only the specimen. The larger idea is a way to place
inspectable, replayable instruments inside probabilistic systems.

The useful boundary is generation versus execution

The interesting distinction is not model weights versus a “real CPU.” Model
inference also runs on processors, and language models can learn genuine
arithmetic procedures. The useful boundary is between generating an answer
and executing a defined operation under a tested contract.

Research on Program-Aided Language Models
(PAL) makes a related split:
the language model reads and decomposes a natural-language problem, while a
runtime such as a Python interpreter executes the generated program. The model
contributes flexible interpretation; the runtime contributes executable
semantics.

That is the division I want in an agent:

At the semantic edge, the model interprets the request, chooses a procedure, identifies relevant quantities, and explains the result.
At the computational edge, a narrow tool validates inputs, applies specified operations, enforces limits, and returns structured output.

This does not make the whole agent deterministic, and it does not make the
model unnecessary. The agent can still choose the wrong tool, supply the wrong
arguments, misunderstand units, or misread the result.

The promise is smaller: one claim becomes inspectable, replayable, and
independently testable.

The opposite of probabilistic is not trustworthy. It is repeatable.

A CPU can be precisely wrong.

Why NIST became my external witness

Calibration cannot be entirely self-referential. The implementation should not
be the sole author of its own expected answers.

That is why I chose NIST—not because government authority turns a result into
mathematical truth, but because NIST has a long institutional practice of
building shared, independently evaluated references.

Congress established the National Bureau of Standards in
1901 to
strengthen the United States' measurement infrastructure. The Standard
Reference Data Act of
1968 authorized a federal program to
collect, critically evaluate, publish, and distribute standardized scientific
and technical reference data. NBS became NIST in
1988. In 1999, that lineage reached statistical
software through the Statistical Reference Datasets
project, usually shortened to StRD.

StRD pairs datasets with certified expected values for specific statistical
procedures. Its collection includes generated and real-world cases of varying
difficulty. For linear procedures, NIST carried 500 digits through its
calculations
so ordinary floating-point representation error would not become the
benchmark.

Longley is a small but numerically challenging linear-regression dataset in
that collection. Its certified results gave my tests something the
implementation could not manufacture for itself: an expected answer produced
outside the code under test.

That qualification matters. StRD does not certify Oddly Exact, prove the
statistics engine correct, or make the software traceable to NIST. NIST
explicitly says the datasets are an aid for evaluating software and that no
mechanism establishes software
traceability.

The reference data was not an oracle for the entire program. It was an
independent witness for the calculations it covered.

Test the tests

Example-based tests ask whether familiar inputs still produce familiar
outputs. Mutation testing asks a more uncomfortable question:

If I introduce a small, plausible defect, does the suite notice?

I used cargo-mutants to replace multiplication with addition in the
multiple-regression standard-error calculation:

- residual_std_dev * sum_sq.sqrt()
+ residual_std_dev + sum_sq.sqrt()

The mutation preserved valid Rust, valid types, and a plausible-looking numeric
result. In the library harness, 389 of 390 tests still passed. The assertion
against NIST's Longley values was the sole failure.

The public mutation
record
preserves the exact command, environment, result, and the limit of what that
experiment establishes:

I cannot claim Longley was the only test in the entire repository capable of
catching the mutation; the run stopped after the failed library harness. I can
claim something narrower and more useful: hundreds of tests tolerated a
semantically broken formula, while an assertion anchored to independently
produced values rejected it.

A green test count is evidence only to the extent that those tests would turn
red when the implementation meaningfully changes. Mutation testing measures
that sensitivity instead of admiring the count.

Challenge the contract around the answer

The formula mutation challenged numerical meaning. My next audit challenged
the promises around the calculation: what the tool accepts, what it refuses,
and how much work it will perform.

The first crack was a contradiction between the advertised contract and the
executable one. The generated JSON Schema forbade additional properties, but
the Rust deserializer silently accepted an unknown field at the request root
and another inside an expression node.

The answer was still numerically correct. That did not make the behavior
harmless. A caller validating against the schema saw a stricter instrument than
a caller speaking directly to the binary. A misspelled or misunderstood field
could disappear without warning.

For an agent-facing tool, silently interpreting a different contract is itself
a correctness defect.

The second crack was a resource boundary. The expression engine limited things
such as integer size, expression depth, and precision. The optimization
interface, however, accepted grid resolution and iteration counts without upper
ceilings.

I requested a grid search with one billion sample points. The pre-repair binary
produced no JSON before an external one-second watchdog terminated it. The tool
had input validation, but it had not earned the bounded-work guarantee I
thought I had built.

Those failures became permanent regression requests. The deserializer now
rejects unknown fields at both levels. The optimizer now publishes and enforces
ceilings of 100,000 iterations and 1,000,000 grid points. The same billion-point
request returns a typed resource_limit response before entering the search.

The contract challenge
Gist
is a runnable replay of all three repaired cases:

That script is more valuable than a screenshot of a green run. It lets another
observer cross-examine the boundary directly.

Repair is the next claim to attack

A new guard is only another claim until the tests prove they care about it.

After repairing the optimizer limits, I selected every mutation
cargo-mutants generated for the two new validators. Eleven mutations tried to
remove the checks, replace them with unconditional success, or alter their
boundary comparisons. The tests caught all eleven.

The focused optimizer mutation
record
captures the selection and outcome:

That completed the loop:

claim → attack → fail → repair → mutate → replay

If I published only the final green suite, you would see confidence. By
preserving the changed operator, the request that hung, the contract
contradiction, the repair, and the mutations that tried to undo it, I can show
a reason for confidence.

The scar is part of the calibration record.

A five-layer challenge stack for agent tools

Oddly Exact is one calculator, but the method travels. Before giving a narrow
tool authority inside an agent workflow, I now ask five questions:

Is the contract explicit and executable? The schema, parser, runtime,
limits, and failure modes must agree. Documentation the executable does not
enforce is only a suggestion.
Is there an independent reference? In this statistical case, NIST StRD
moved selected expected answers outside my implementation. Another domain
might use a standards specification, a reference implementation, a verified
corpus, or a separately derived test oracle.
Which properties should survive across many inputs? Property-based tests
generate cases and check laws rather than memorizing individual examples.
One Oddly Exact property checks that translating every value in a sample by
a large exact offset leaves sample variance unchanged within a tight
floating-point tolerance.
Can the tests detect plausible corruption? Mutation testing changes
operators, comparisons, return values, and guards. A surviving mutation does
not prove the implementation is wrong; it exposes a behavior change the
suite cannot currently distinguish.
Are failures bounded, typed, and replayable? An agent must distinguish a
rejected computation from an unavailable tool. Division by zero, resource
exhaustion, malformed JSON, and a crashed process should not collapse into
the same ambiguous failure.

These layers do different jobs. External references challenge specific
answers. Properties challenge behavior across an input space. Mutations
challenge the tests themselves. Contract and resource attacks challenge the
boundary around all of it.

None provides universal correctness. Together they create a narrower and more
useful result: evidence that a particular tool deserves provisional authority
for validated requests inside a declared contract.

A source under challenge

For a validated request inside that contract, the tool's structured result can
serve as the agent's operational source of truth.

That is a runtime role, not a claim of infallibility. If the agent encounters a
disagreement, it can inspect the arguments, surface the conflict, or consult
another independently trusted tool. It should not quietly replace a structured
result with fresh prose arithmetic.

The trust relationship runs in opposite directions at different times:

While building the tool, the developer should distrust it aggressively.
Inside a contract the tool has earned, the agent should respect its answer or its typed refusal.

New evidence can always return the tool to the first phase. That is a source
under challenge: trusted in operation, open to appeal, and always one
counterexample away from revision.

The beautiful thing is not that source code is truth.

The beautiful thing is that source code can be cross-examined.

I did not end this experiment with a calculator an AI agent can trust forever.
I ended it with something more useful: a calculator whose authority can be
revoked by evidence.

AI tools assisted with editorial research and revision. I verified the
technical claims and stand behind the final text.

Give Your Coding Agent a Deterministic Vulnerability Oracle

Don Johnson — Tue, 21 Jul 2026 15:04:00 +0000

AI agents can write code, run tests, inspect dependencies, and open pull requests. But when they encounter a vulnerable package, their security reasoning often collapses into a web search, an opaque API score, or whatever the model remembers from training.

The problem is not simply missing vulnerability data. It is the contract between the agent and its harness—the automation layer that supplies tools, executes commands, and interprets results. If “no match” silently becomes “safe,” stale intelligence looks current, or a network failure resembles an empty result, the agent can produce a confident answer without trustworthy evidence.

I designed VulnGraph around a different premise: turn continuously changing vulnerability intelligence into a deterministic local primitive. vulngraph-data compiles upstream security sources into verified, content-addressed snapshots. vulngraph-cli installs those snapshots and checks CVEs, package versions, or entire lockfiles offline.

The result is a security tool shaped for both humans and machines: an accepted snapshot and target produce the same verdict; every verdict carries typed evidence; stale data fails explicitly; and unknown is never misrepresented as clean.

This article explains the design decisions behind that contract and shows how to embed VulnGraph into agent and harness workflows.

The dangerous gap between “not found” and “safe”

Suppose an agent is reviewing a dependency update. It needs to answer a seemingly simple question: Is this version safe to ship? A useful answer depends on distinctions that are easy for a harness to erase.

What happened	A careless harness concludes	What the contract must preserve
The package exists and the version is outside every affected range	No vulnerabilities found	`not-affected`
The package is absent from the dataset	No vulnerabilities found	`unknown`
The local snapshot is too old	The last result is probably fine	A freshness failure
The tool cannot open or verify its data	Empty result	An operational or integrity error

These are not cosmetic labels. They lead to different agent actions: proceed, investigate, refresh the data, or stop the workflow. Once a harness flattens them into a Boolean, the model cannot recover the missing meaning through better prompting.

Here is the distinction in practice. One command checks an exploited CVE, a real package version, and a package the snapshot has never observed:

$ vulngraph check CVE-2024-4577 npm:lodash@4.17.15 npm:no-such-package@1.0.0

  CVE-2024-4577
  verdict: ACTIVELY EXPLOITED  (confidence 0.99)
  action:  patch now
  reasons: CRITICAL_SEVERITY, HIGH_EXPLOIT_PROBABILITY,
           KNOWN_EXPLOITED, PUBLIC_EXPLOIT, ...
  cvss:    9.8
  epss:    100.0%
  kev:     listed

  npm:lodash@4.17.15
  verdict: PROOF OF CONCEPT  (confidence 0.96)
  action:  prioritize
  package: npm:lodash — 6 CVE(s) affect this version

  npm:no-such-package@1.0.0
  verdict: UNKNOWN  (confidence 0.00)
  action:  investigate
  reasons: NOT_OBSERVED

  snapshot: sha256:41ee5fbbf58f0a46b99234af89c5388e5f27e0dcd3dbd52298bcd2c2d87ca90e

The output does not ask the agent to infer policy from prose. It supplies a disposition, an action, stable reason codes, confidence, and the exact snapshot that produced the result. The human-readable rendering is useful in a terminal; --json exposes the same distinctions through a versioned machine envelope.

Split changing intelligence from deterministic execution

Vulnerability intelligence changes every day. Agent execution still needs to be reproducible. I separated those concerns instead of allowing every check to fetch and reconcile live data.

CVE List V5 · EPSS · CISA KEV · exploit sources · ATT&CK · OSV · …
                              │
                              ▼
                    vulngraph-data
              fetch → normalize → build → verify
                              │
                              ▼
            immutable data-YYYYMMDD release
       manifest + checksums + content snapshot_id
                              │
                  vulngraph update
                              ▼
             verified local snapshot
                              │
          vulngraph check --offline --json
                              ▼
                  agent or harness

The vulngraph-data repository owns the nondeterministic edge: retrieving bulk publications from upstream sources and reconciling them into one graph. Its output is deterministic. Identical inputs produce byte-identical semantic files, and their hashes produce a snapshot_id that names the graph by content. If a daily build has no semantic change, there is no new release.

The release artifact is the distribution boundary. A consumer does not need the raw source collection or the machinery that built it. It receives a fixed database archive and a manifest recording the snapshot identity, file hashes, source freshness, graph format, and engine revision.

The vulngraph-cli owns the deterministic side. vulngraph update downloads a release into a staging area, verifies the archive checksum and every file hash, recomputes the snapshot identity, checks format compatibility, sanity-opens the graph, and only then activates it. Failed updates cannot replace the last verified snapshot.

After activation, checks do not touch the network. The snapshot becomes an explicit input to the decision:

verdict = policy(snapshot_id, target)

There is no wall clock, random value, model call, or remote response inside that function. Once a snapshot passes the freshness and integrity preflight, the same snapshot and target give a developer laptop, a CI job, and an agent sandbox the same verdict.

A harness workflow that does not guess

A reliable integration has three phases. Only the update phase needs network access; discovery and checks are stable interfaces that the harness can validate.

1. Discover the contract

Do not make the agent reverse-engineer help text. Let the harness inspect the machine-readable capability document and the bundled JSON Schemas:

vulngraph --json capabilities
vulngraph schema command
vulngraph schema observation
vulngraph schema status

capabilities declares the commands, output schema versions, exit codes, and two behavioral guarantees: checks are offline and deterministic. A harness can validate support during startup instead of discovering an incompatibility halfway through a run.

2. Refresh and preflight outside the restricted step

Install or refresh the snapshot in a setup job that is allowed to use the network:

vulngraph update
vulngraph --json status

Then mount or cache the resulting VulnGraph home directory inside the agent environment. This makes network access a controlled data-provisioning concern instead of an ambient capability available during every security decision.

status reports whether a snapshot is installed, its identity, its integrity, and its freshness. A dataset older than fourteen days is not merely accompanied by a warning: check refuses to answer and exits with code 4.

3. Check the artifact the agent is actually changing

The check command accepts individual CVEs, package coordinates, and dependency files. Pointing it at a lockfile expands every dependency and applies the same version-range policy:

vulngraph --offline --json check package-lock.json > vulngraph.json

The CLI recognizes lockfiles from npm, Yarn, pnpm, Cargo, Bundler, Poetry, pip, Go, Composer, Maven, Gradle, and Pipenv. The abridged JSON response below shows the stable envelope:

{
  "schema": "vulngraph.command.v1",
  "command": "check",
  "ok": true,
  "partial": false,
  "snapshot_id": "sha256:41ee5fbb…",
  "data": [
    {
      "schema": "vulngraph.observation.v1",
      "target": { "type": "package", "value": "npm:lodash@4.17.15" },
      "verdict": {
        "disposition": "proof-of-concept",
        "confidence": 0.96,
        "action": "prioritize",
        "reason_codes": [
          "ATTACK_MAPPED",
          "HIGH_SEVERITY",
          "MULTISOURCE_CORROBORATION",
          "PUBLIC_EXPLOIT",
          "SEVERITY_SCORED",
          "VERSION_IN_AFFECTED_RANGE",
          "WEAKNESS_CLASSIFIED"
        ]
      }
    }
  ],
  "warnings": [],
  "errors": [],
  "metrics": { "elapsed_us": 781250 }
}

The important separation is that VulnGraph observes; it does not gate. A successful check exits with 0 even when it finds an actively exploited vulnerability. Exit codes describe whether the tool completed reliably; the harness applies organizational policy to the returned dispositions.

Disposition	Evidence	Sensible default harness action
`actively-exploited`	Listed in CISA KEV	Block or require an explicit emergency exception
`weaponized`	Public exploit and very high EPSS	Block or require security approval
`proof-of-concept`	At least one public exploit	Prioritize remediation and require review
`scored`	CVSS or EPSS evidence, no public exploit	Apply the repository's severity policy
`recorded`	Known record without stronger risk signals	Monitor and retain the evidence
`not-affected`	Known package outside all affected ranges	Proceed for this snapshot
`unknown`	Target absent from the snapshot	Investigate; never translate to clean

This split keeps probabilistic reasoning in the right place. The harness deterministically decides which states require a stop, review, or escalation. The agent then uses the evidence to explain the finding, inspect reachability, propose the smallest safe upgrade, and communicate the trade-off to a human.

A minimal Node.js harness

The following wrapper turns the versioned VulnGraph envelope into one of five local workflow decisions. It does not ask the model whether an operational failure is acceptable, and it never treats an unrecognized disposition as approval.

import { spawnSync } from "node:child_process";

const binary = process.env.VULNGRAPH_BIN ?? "vulngraph";
const target = process.argv[2] ?? "package-lock.json";
const run = spawnSync(
  binary,
  ["--offline", "--json", "check", target],
  { encoding: "utf8" }
);

let envelope;
try {
  envelope = JSON.parse(run.stdout);
} catch {
  throw new Error(`vulngraph returned invalid JSON: ${run.stderr}`);
}

// Exit codes describe tool reliability, not vulnerability severity.
if (run.status === 4) {
  throw new Error("VulnGraph snapshot is stale; refresh before continuing");
}
if (run.status !== 0 || !envelope.ok) {
  throw new Error(`VulnGraph failed: ${JSON.stringify(envelope.errors)}`);
}
if (envelope.schema !== "vulngraph.command.v1") {
  throw new Error(`Unsupported schema: ${envelope.schema}`);
}

const policy = new Map([
  ["actively-exploited", "block"],
  ["weaponized", "block"],
  ["proof-of-concept", "review"],
  ["scored", "review"],
  ["recorded", "monitor"],
  ["not-affected", "proceed"],
  ["unknown", "investigate"]
]);

const decisions = envelope.data.map(({ target, verdict }) => ({
  target: target.value,
  disposition: verdict.disposition,
  decision: policy.get(verdict.disposition) ?? "investigate",
  action: verdict.action,
  confidence: verdict.confidence,
  reasons: verdict.reason_codes
}));

const evidence = {
  snapshotId: envelope.snapshot_id,
  decisions
};

console.log(JSON.stringify(evidence, null, 2));

if (decisions.some(({ decision }) => decision === "block")) {
  process.exitCode = 10; // This is our harness policy, not a VulnGraph exit code.
}

The evidence object is what I would place in the agent's context. The prompt can remain narrow:

Review the attached vulnerability evidence for this dependency change.

- Never describe `unknown` as safe or clean.
- Preserve the snapshot ID in your report.
- Explain the reason codes in plain language.
- Check whether each affected dependency is reachable from the changed code.
- Propose the smallest compatible upgrade, but do not edit files until approved.

The deterministic layer establishes what was observed and which policy branch applies. The agent contributes the work it is good at: repository-specific investigation, explanation, remediation planning, and communication.

Design lessons for agent-facing security tools

Building the graph was only half the problem. Making it safe for automation required treating the interface as part of the security model.

Make ignorance a first-class result. unknown must survive every layer from data lookup to terminal output to agent context.
Return evidence, not just a score. A verdict should carry stable reasons and source observations that a human can audit.
Separate update time from decision time. Network retrieval belongs in a controlled provisioning phase; checks should be local and reproducible.
Identify the data behind every answer. A content-derived snapshot ID turns “what did the scanner know?” into an answerable question.
Let tools describe their contract. Capabilities, schemas, freshness, and exit semantics should be machine-readable.
Keep gating policy outside the observation tool. Different repositories have different risk tolerances. The harness should own that policy explicitly.

An agent does not become trustworthy because it receives more tokens or a longer system prompt. It becomes more trustworthy when its environment preserves distinctions, constrains side effects, and supplies evidence through contracts that cannot silently change meaning.

That is the role I designed VulnGraph to play: not an autonomous security authority, but a fast, local, auditable source of vulnerability evidence that gives agents and their harnesses something solid to reason from.

Build and release pipeline: copyleftdev/vulngraph-data
Offline CLI: copyleftdev/vulngraph-cli

Disclosure: I designed VulnGraph. This article and its cover were created with AI assistance. I reviewed the technical claims against the linked repositories and live CLI output.

If this open-source work is useful to you, you can support its continued development:

[Boost]

Don Johnson — Sat, 18 Jul 2026 17:47:12 +0000

Don Johnson

Jul 17

The Last Honest Abstraction: Why AI Coding Isn't the End of Engineering

#discuss #ai #programming #softwaredevelopment

5 min read

The Last Honest Abstraction: Why AI Coding Isn't the End of Engineering

Don Johnson — Fri, 17 Jul 2026 16:47:19 +0000

Why every generation thinks the next programmer understands less—and why AI changes the burden of proof, not the need for engineering.

"Developers can't even explain their own code anymore."

If you've spent any time around software lately, you've probably seen some version of that argument.

The culprit, we're told, is AI.

Developers are prompting language models, accepting generated code, and shipping software they couldn't possibly explain line by line. Somewhere between autocomplete and autonomous agents, we supposedly crossed a professional line. We traded craftsmanship for convenience.

It sounds like a new argument.

It's one of the oldest in computing.

Before We Go Any Further...

Before I make my case, I want to pay homage to a rare breed of engineer.

You know who you are.

You're the people who can look at compiler output and immediately tell when an optimization didn't quite land. You read assembly like most people read JavaScript. You understand calling conventions, cache locality, virtual memory, branch prediction, linker behavior, and processor architecture—and you can still build elegant software without turning every project into a dissertation.

You're rare.

And you've quietly carried this industry for decades.

Many of the abstractions the rest of us rely on exist because someone like you spent years wrestling with complexity until everyone else could work one layer higher.

Every generation has its handful of engineers who descend into the deepest layers of the machine, then return carrying tools that allow the rest of us to build something higher.

This article isn't arguing against mastery.

It's a thank-you to the people who made abstraction possible.

Every Layer Was Once Considered Cheating

Again and again, computing has moved upward by adding layers.

Machine code gave way to assembly for most programming. Someone inevitably complained that assembly programmers weren't "real programmers" because they no longer had to manipulate raw binary.

Higher-level languages followed. Compilers hid more of what the processor was actually doing. Managed memory, virtual machines, dynamic languages, frameworks, cloud platforms, containers, and orchestration systems each took another category of complexity and moved it behind an interface.

And now...

AI.

The technologies aren't a single, tidy lineage. They abstract different problems, and old layers rarely disappear. But the cultural pattern repeats:

"Do they really understand what's happening underneath?"

The nouns change.

The complaint doesn't.

The Myth of the Complete Programmer

Here's an uncomfortable truth.

Most software has never been written by people who fully understood every layer beneath it.

That isn't criticism.

It's reality.

Ask an application developer to explain exactly how their operating system schedules threads.

Ask them why their compiler emitted a particular optimization.

Ask them to walk through every page-table translation.

Ask them to trace how cache behavior affected one production slowdown.

Ask them to explain every TCP retransmission or every filesystem recovery after an unexpected shutdown.

Some can.

Most can't.

And yet...

The software still ships.

Not because understanding stopped mattering, but because software engineering evolved around abstractions with bounded responsibilities and increasingly testable guarantees.

The Entire Purpose of Engineering

Engineering has never been about refusing abstraction.

It's been about earning it.

Every abstraction represents an enormous investment of human understanding. Someone had to master the ugly details so the next engineer could focus on solving a different problem.

A civil engineer need not manufacture every bolt before designing a bridge, but must know which loads it can bear. Software abstractions work the same way: we delegate construction, not accountability.

Requiring every software engineer to hand-write machine code before building a web application wouldn't protect craftsmanship.

Progress wouldn't slow down.

It would stop.

Abstractions aren't shortcuts.

They're accumulated knowledge.

They're understanding, compressed into reusable form.

That's not laziness.

That's civilization.

What AI Actually Changed

AI did change something.

Just not what many people think.

A conventional compiler is expected to produce reproducible output from fixed inputs. A language model samples from probabilities. A compiler hides implementation behind a defined system; a model proposes implementation without guaranteeing that the proposal is correct.

That makes AI a less trustworthy abstraction by default—not an invalid one. It means the engineer must supply the trust boundary through review, tests, observability, constraints, and a willingness to reject plausible-looking output.

The real shift is that authorship is becoming an even weaker proxy for understanding—and an even less useful signal of correctness.

For decades, many engineers subconsciously equated "I wrote it" with "I understand it."

Those were never the same thing.

We've all inherited libraries. We've all trusted frameworks. We've all depended on operating systems we didn't write. We've all deployed software built on millions of lines of code authored by strangers.

AI simply made that reality impossible to ignore.

The responsibility hasn't disappeared.

It's moved.

Instead of taking familiarity as evidence because you typed every character yourself, you establish confidence by inspecting the design, challenging assumptions, measuring behavior, and validating outcomes.

This doesn't absolve the engineer of understanding. It changes the required depth. You may not need to explain every generated line from memory, but you do need to understand the system's boundaries, invariants, dependencies, and failure modes well enough to be accountable for what it does.

That's engineering.

Judgment Is Becoming the Scarce Resource

The best engineers I've met have never impressed me with how quickly they type.

They impress me with how quickly they notice something feels wrong.

They can review a thousand lines and, through experience, stop at the seven that deserve attention. They know where abstractions leak. They know when benchmarks are lying. They know when a beautiful architecture is solving yesterday's problem.

They ask better questions than everyone else.

AI doesn't replace that.

If anything, it magnifies its importance.

Syntax gets cheaper.

Judgment becomes priceless.

Standing on Invisible Giants

There's a beautiful irony here.

The engineers most qualified to criticize abstraction are often the same people who built it: compiler engineers, kernel developers, database architects, networking pioneers, language designers, and chip architects.

They spent years making impossibly complicated systems disappear behind clean interfaces—not because they wanted everyone staring at the gears, but because they wanted the rest of us to keep building.

Every abstraction they created became another floor in a building nobody could have constructed alone.

The Last Honest Abstraction

Every generation believes the abstraction immediately beneath them was the last honest one.

Assembly programmers point to machine code.

C programmers point to assembly.

Framework skeptics point to handwritten C.

Now AI skeptics point to handwritten code.

Ten years from now, someone will publish an article insisting developers don't even understand what their autonomous software factories are doing.

Someone else will nod and say:

"Back in my day, we actually wrote the prompts ourselves."

And the cycle will begin again.

Because the history of software has never been the history of losing understanding.

It's been the history of redistributing it.

A relatively small number of extraordinary engineers venture into the deepest layers, wrestle complexity into submission, and transform hard-earned understanding into abstractions that millions of others can build upon without mastering every detail beneath them.

Those abstractions aren't evidence that engineering is dying.

They're evidence that engineering succeeded.

Progress has never required everyone to understand everything.

It has always required enough people to understand each layer deeply enough to build the next one—and everyone who uses that layer to understand its promises and failure modes well enough to take responsibility for the result.

The rest of us stand on their shoulders.

And someday, if we do our jobs well enough, someone else will stand on ours.

Editorial note: This article was refined with AI-assisted editorial review, and its cover illustration was generated with AI. The argument, factual review, and final wording remain the author's responsibility.

A Vibe Is Not a Verdict: I Built a Tool That's Allowed to Say 'I Don't Know'

Don Johnson — Mon, 13 Jul 2026 19:37:38 +0000

A tool should do one thing, do it well, and — this is the part everyone forgets — know exactly where its knowledge ends.

I built kilo for a morning like this one. I just didn't know the morning would come this soon.

The message that was waiting for me

I woke up, poured the coffee, and did the thing you're not supposed to do before you're fully awake: I opened my messages. And there it was, sitting at the top of the inbox like it had been placed there on purpose — a cold recruiter pitch.

An "exciting opportunity" for a Quality Engineer role at an aerospace firm. Warm, professional, faintly urgent. Please click the link below to view the job description and apply. The link was a long tracking URL, the kind with an opaque token and source and method parameters stapled to the end.

And then the signature. One "recruiter" who claimed, in a single line, to be a physical therapist, a call-centre manager, an e-commerce power seller, a freight dispatcher — half a dozen unrelated trades piled into one job title, now, apparently, placing quality engineers at an aerospace company. That's not a résumé. That's a slot machine.

There it was: the exact situation I wrote the tool for, delivered to my inbox before I'd finished the coffee.

Because here's what would normally happen next. My gut says scam. My gut is usually right. But "usually right" is how you eventually click the one link that isn't. A gut feeling is a vibe, and a vibe is not a verdict. That frustration is precisely why, months ago, I sat down and wrote kilo — a way to stop feeling about an address and start knowing about it. So this morning wasn't an annoyance. It was the experiment finally running itself.

I put the coffee down and opened a terminal.

What I actually built, and why

Let me back up and tell you what kilo is, because the design is the whole argument.

KiloCheck is an offline IP-reputation engine. One binary. ~12MB. Written in Rust. Its entire worldview is a four-stage pipeline: ingest a signed, versioned threat release → normalize typed observations → compile an immutable local index → check IPs against it. Cryptographically verified going in. Zero network calls coming out.

I made it offline on purpose. An ordinary check never reaches an intelligence API. When I'm triaging something sketchy, I don't want my queries leaking, I don't want to tip anyone off, and I want the same answer at 3AM on a plane as I get at my desk. The threat knowledge is baked into a signed local snapshot. The check is just a lookup against evidence I already trust.

Install is a checksum and a copy — the way a tool should be:

curl -fsSL https://raw.githubusercontent.com/copyleftdev/kilocheck/v0.2.0/scripts/install.sh | sh
kilo update      # pulls the signed data release → prints "Claims  3160"
kilo status --json
# → "integrity": "verified", "freshness": "fresh"

That was the tool. This was the morning. Time to point one at the other.

The experiment, step one: turn the lure into a number

kilo answers exactly one question — is this IP a known bad actor? — so first I had to turn that tracking link into an address. That's dig's whole job: resolve the hostname behind the URL down to the IP actually serving it.

$ dig +short <the-redirect-host>
203.0.113.90

Then I asked my tool the question I built it to answer. This is the moment the experiment either validates the design or embarrasses me:

$ kilo check 203.0.113.90 --json | jq '.data[0].verdict'

{
  "disposition": "unknown",
  "confidence": 0.0,
  "recommended_action": "monitor",
  "reason_codes": ["NOT_OBSERVED"]
}

NOT_OBSERVED. Confidence 0.0.

And I grinned, because this is exactly the behavior I designed for — and the exact behavior a worse tool would have gotten wrong.

Think about the pressure in that moment. I clearly think it's a scam. The vibe is overwhelming. A tool built to please its operator would have felt that pressure and handed me a scary red verdict to confirm what I already believed. kilo refuses. It says: I have no evidence about this address, and I am not going to invent some to make you feel clever.

That refusal is the single most important thing I built into it. A tool has to know the difference between "safe," "bad," and "I don't know" — and it has to refuse to smear them together. Missing, unknown, stale, incomplete, unsafe: five distinct states in my model, not three shades of the same shrug. This morning it chose the honest one under maximum temptation to lie.

Step two: prove it can still bite

An honest "I don't know" is only worth something if the same tool can say a hard "yes." If kilo returned unknown for everything, it wouldn't be humble — it'd be broken. So I ran the control, right there in my pajamas: the suspicious host, a known command-and-control node, and a boring public DNS resolver, all in one shot. Distilled from the JSON:

203.0.113.90      → unknown   monitor   NOT_OBSERVED        (0 sources)
162.243.103.246   → CRITICAL  block     COMMAND_AND_CONTROL (1 source, conf 0.99)
8.8.8.8           → unknown   monitor   NOT_OBSERVED        (0 sources)

There. A known C2 node gets critical / block / 0.99 with provenance attached. The public resolver and the suspicious host both get a calm NOT_OBSERVED. No false alarms on clean infrastructure, no missed alarm on a real one. The experiment held: the tool's silence means something precisely because its alarm means something.

Step three: the twist my own tool pointed me toward

So the infra came back clean. Was my gut wrong? No — it was pointed at the wrong target. And that's the part I didn't see coming.

kilo's honest negative didn't close the case. It moved it. "The danger is not in this infrastructure" is a coordinate — it told me to go look somewhere else. So I followed the rest of the trail with the neighbors on my toolbelt.

curl followed the redirect — headers only, I never touched the body:

HTTP/2 302 → /<captcha-gate>?...&source=...&method=...

openssl read the cert straight off the wire: a real, correctly issued certificate for a legitimate, well-known job-traffic aggregator. And a quick lookup closed the loop: the aerospace company in the pitch is real, with genuine engineering openings, and the tracking link was an affiliate click-tracker redirecting through the aggregator's own anti-click-fraud gate. No payload. No credential harvest. No bad IP anywhere in the chain.

So what was the actual threat? It was never in the packets. It was the sender — a freelance lead-gen spammer blasting affiliate commission links across social platforms, wearing a recruiter costume stitched together from half a dozen unrelated careers.

And here's why I'm writing this at all: my own tool solved the case by refusing to pretend. Its 0.0 sent me away from a malware rabbit hole and toward the social-engineering signal where the truth actually lived. If kilo had cried wolf on clean infrastructure to match my mood, I'd have wasted the morning dissecting a job board. Because it stayed honest, it handed me the thread on the first pull.

That is the exact experiment I wrote the tool to run. And it passed.

What the morning actually taught me

The recruiter was never the point. The shape of the morning was.

I didn't solve this with one brilliant tool. I solved it with a chain of small, honest ones: dig turned a name into an address so kilo had something to check; kilo's honest negative reframed the question so curl and openssl knew where to look. And every one of them spoke stable JSON — so I was consuming contracts, not parsing prose. Exit codes matter. Schemas matter. Failure stays legible.

That's the manifesto I keep coming back to as I build these things: contracts, not vibes. When I pipe kilo check --json into jq, I'm having a typed conversation with reality. I can't hallucinate a verdict, because the verdict arrives as a signed struct with its provenance stapled on.

It's the difference between a tool that sounds right and one that is right:

A gut feeling is an opinion. A sharp, composable, honest toolbelt is a witness.

And witnesses — unlike hunches — can say the three most valuable words in this whole trade: "I don't know." Then go find out.

Steal the pattern if it's useful:

Small, single-purpose tools that speak stable JSON compose into something no chatty do-everything API can match.
Honesty beats confidence. A tool that returns unknown under pressure is worth ten that guess to please you.
Let the negatives navigate. The most useful result of the entire morning was a 0.0.

So the next time a link makes your gut twitch, don't argue with the vibe. Resolve it, and ask a tool that's allowed to say nothing.

kilo didn't catch a scam — there wasn't one, just spam in a lab coat. It did something I value more: it told me the truth about what it knew, drew a hard line around what it didn't, and pointed me at where the real answer was hiding.

I built a 12MB binary to say "not observed" without flinching, on the exact morning I most wanted it to lie. It didn't. That's the whole experiment. That's why I built it.

kilo check <your-suspicions-here> --json

Tooling in this piece: KiloCheck · the eternal dig / openssl / curl. The story is real; the identifying details have been filed off.

The AI Cost-Modeling Handbook: I let Claude do the modeling, but never the arithmetic

Don Johnson — Wed, 01 Jul 2026 02:51:48 +0000

Every "what's the cheapest model?" thread online is people trading vibes. I got tired of it, so I built a pipeline that pulls live, cited prices and runs the numbers through an exact-rational math kernel — no floating-point drift, no LLM hallucinating a multiplication. Then I pointed it at eight of the cost questions every agent builder actually faces. Here's everything it found, and the repo so you can re-run all of it.

There are two kinds of LLM cost advice. The first is a benchmark leaderboard with a price column, which tells you nothing about your workload. The second is a confident tweet that's quietly wrong because someone multiplied a per-million price by the wrong token count in their head.

I wanted a third kind: a model you can audit. Re-run it and you get bit-identical numbers, with the source of every input price one file away. This article is the result — a tour through eight cost decisions, each answered with real money math, and each one teaching something that intuition gets wrong.

Let's start with the question that kicked it off.

Cold open: what's the most cost-effective model to run an agent on?

The setup: I run Hermes Agent, a model-agnostic agent framework, and I wanted the cheapest token provider to drive it. Not cheapest per token — cheapest per unit of quality, because a model that flubs tool calls and retries can cost more than a pricier reliable one.

So the metric is blended cost ÷ agentic-quality-score. Blended cost uses a production-agentic token mix (more on that below); quality is a normalized agentic score (mean of BFCL / τ²-bench / SWE-bench Verified). Filters: open-weights, prompt caching, no-train.

Model @ Provider	Blended $/1M	Quality	$/quality (×1000)
DeepSeek V3.2 @ OpenRouter	0.1145	77	1.49
DeepSeek V3.2 @ DeepInfra	0.1951	77	2.53
MiniMax M2 @ MiniMax	0.2629	73	3.60
GLM-4.6 @ z.ai	0.5276	71	7.43
Kimi K2 @ DeepInfra	0.6613	66	10.02
DeepSeek R1 @ DeepInfra	0.6519	54	12.07

DeepSeek V3.2 wins decisively — it's both the highest-quality open model in the set and near the cheapest, because its output token price is absurdly low for its tier. The emerging best deal (pending a quality test) is DeepSeek V4 Flash on Fireworks at $0.0896/1M blended — cheapest in the field, with ZDR-by-default and a 50% batch discount.

That's a useful answer. But the interesting part is everything that question opens up. If you can split traffic across providers, what's the optimal mix? Should you self-host instead? Is the "smart expensive model" actually cheaper once you count its thinking tokens? Each of those is a chapter.

First, how this is built.

The toolchain: research agents + an exact-math kernel

The pipeline has two halves, and the whole point is that the LLM never touches the arithmetic.

1. Research agents gather live, cited inputs. Token prices in 2026 move monthly — half the models in my training data are already legacy. So every price and benchmark in this guide came from a research agent doing live web search/fetch, returning a structured table with source URLs and observation dates. Those land in data/ (pricing.md, quality.md, gpu.md, frontier.md, decline.md), each one auditable.

2. An exact-rational kernel does the math. I used agent-calc, an AI-native computation kernel that works in exact rationals — 3/5 × 1,000,000 returns 600000, not 599999.9999998. It exposes typed domains: rational, linear (LP), finance (NPV/amortization), solve (algebra), stats (distributions), polynomial, interval (uncertainty arithmetic). Every chapter below drives a different one.

Why bother? Because cost models are exactly where floating-point lies and LLMs fumble: tiny per-token prices, huge token counts, compounding rates. When an answer is "self-hosting breaks even at 87.2% utilization," you want that to be computed, not vibed. And the secret structure of this guide is that each chapter answers a money question and exercises one kernel domain — it's a cost-modeling handbook that doubles as a tour of exact computation.

The shared workload model

Every chapter uses one production-agentic token profile, because agent tool-loops are wildly input-heavy:

Component	Share of tokens
Fresh input	21.25%
Cached input	63.75%
Output	15%

That's an 85/15 input/output split with 75% of input being a cache hit (the system prompt, tools, and skills get re-sent every tool-loop step). Blended price = 0.2125·in + 0.6375·cached + 0.15·out. Chapter 7 explains why this profile is forced on you; Chapter 6 explains why the cache hit rate is so high. For now, take it as the workload.

Chapter 1 — Optimal multi-provider routing (a linear program)

You don't have to pick one provider. Run a cheap one for the bulk, escalate elsewhere when constraints demand. The optimal split — minimize spend subject to a quality floor, per-endpoint capacity caps, and a US-jurisdiction floor θ — is a literal linear program. agent-calc solves it (solve_lp), then I re-verify the winning allocation's cost in exact rationals (the LP solver is f64; the headline number shouldn't be).

Sweeping the privacy knob θ traces a cost-of-privacy frontier:

US floor θ	min $/1M	$/mo @ 1B	allocation
0.00–0.40	0.1468	$146.77	A=0.60, B=0.40
0.60	0.1629	$162.89	A=0.40, B=0.60
0.80	0.2716	$271.61	A=0.20, B=0.60, F=0.13, H=0.07
0.872	0.3116	$311.59	A=0.13, B=0.60, F=0.27
≥ 0.873	—	INFEASIBLE	the privacy wall

Three things intuition misses:

The first 40% of US-jurisdiction traffic is free — the cost-optimal blend already lands there, because the cheapest provider is capped at 60% and the overflow falls on a US endpoint anyway.
The marginal cost of privacy is convex — 40→60% is cheap; past 60% the quality floor forces in weaker fillers and the bill nearly doubles by 87%.
There's a hard wall at 87.2%. Only one high-quality US endpoint exists in this menu; beyond its capacity the math is infeasible. To break the wall you need a second high-quality US provider, or you accept single-point-of-failure risk.

Chapter 2 — Self-host vs. serverless: it's a utilization question

100% utilization (impossible); only owned 8×B200 wins at 72%" width="800" height="459">

Everyone's instinct: "at some volume, renting GPUs beats per-token APIs." The reframe that changes everything: a self-hosted node costs the same at 5% load or 95% load, while the API scales linearly with tokens. So it's not a dollar break-even — it's a break-even utilization: how busy must the box stay to win?

u* = node_$/mo × 1e6 / (capacity_tokens/mo × API_$/1M)

If u* > 100%, self-hosting loses even at full tilt. Modeling DeepSeek V3.2 (8-GPU node) against DeepInfra's $0.195/1M, with agent-calc handling the owned-hardware amortization (finance) and the exact u* (eval):

Hardware / procurement	break-even u*	verdict
8×H200, rent on-demand	531%	never
8×H200, own (amortized)	220%	never
8×B200, rent reserved	174%	never
8×B200, own (amortized)	72%	the only winner

For a commodity model this cheap on the API, renting GPUs loses in every configuration — the serverless provider batches across thousands of tenants to hit a utilization you can't. The only config that wins is owned Blackwell silicon kept above ~72% utilization 24/7 (~56B tokens/month of steady load). And against the falling API floor ($0.0896/1M), even that roughly doubles its break-even. The takeaway: self-host for privacy, control, or latency — not to cut the token bill.

Chapter 3 — The reasoning-token tax

Reasoning models emit hidden "thinking" tokens, billed at the output rate. A 600-token answer can bill 7,800. Does the higher success rate justify the burn? The honest unit isn't $/token — it's cost per solved task = attempt cost ÷ success rate.

With agent-calc computing exact costs (eval) and the break-even success rate (solve):

Model	k (think mult.)	$/solved	×champion
DeepSeek V3.2	4	3.17m	1.00×
MiniMax M2	2	4.03m	1.27×
Kimi K2	1	8.79m	2.77×
GLM-4.6	5	13.77m	4.35×
DeepSeek R1	12	36.80m	11.61×

(m = milli-dollars = $0.001/task.)

The killer result: DeepSeek R1 would need a 627% success rate to match V3.2's cost-per-solved — mathematically impossible. Even a perfect R1 stays far more expensive, because its per-attempt token burn alone exceeds V3.2's entire cost-per-solved. The sweep confirms it: even at k=0 (zero thinking), R1 is still 2.6× the champion.

The rule: the reasoning tax only pays when output tokens are cheap. Thinking is a multiplier on the output price. Pay $2.15/M and ×12 and you've built the most expensive possible way to solve a task; pay $0.38/M and the same multiplier barely registers.

Chapter 4 — The retry cascade

You don't need one model for every task. Run a cheap model on everything, escalate to a reliable premium model only on the residual failures. agent-calc computes the escalation probability (stats/binomial_pmf) and the exact expected cost (eval).

Strategy	cost/solved	coverage	% → premium
Cheap-only (DeepSeek V3.2)	3.17m	77.0%	0%
Premium-only (Claude Opus 4.8)	82.96m	88.0%	100%
Cascade V3.2 → Opus	19.78m	97.2%	23%
Cascade V3.2 → Gemini → Opus	13.52m	99.5%	4.4%

The cascade beats premium-only on both axes: higher coverage (97% vs 88%) at a quarter the cost, because only 23% of tasks reach the expensive model. Adding a mid-tier pushes coverage to 99.5% while cutting cost further — the cheap+mid tiers absorb 95.6% of traffic, so the $73m Opus call fires on 4.4% of tasks.

And the trap to avoid: retrying the same model never changes cost-per-solved (it stays C/p), and for deterministic failures it doesn't even raise coverage. Only escalating to a different, stronger model moves the needle. "Just add retries" is a no-op; "add a tier" is the win. (The cost advantage survives even strong failure correlation — 3.6× cheaper than premium-only even when the premium rarely rescues what the cheap model botched.)

Chapter 5 — The NPV of waiting

A provider offers a discounted reserved rate if you commit for a year. Locking it in feels prudent — but token prices fall fast, and you might be locking yourself above where pay-go will be in six months. This is a present-value problem: NPV the committed cash-flow stream against the declining pay-go stream (agent-calc finance/net_present_value), and find the break-even decline rate.

The decisive input is that price decline depends entirely on what you hold fixed:

Fixed-quality (same capability getting cheaper): ~80%/yr (a16z/Epoch).
Frontier (always run the best): ~30%/yr, sometimes flat or rising.

At a 30%-off, 12-month commit:

Workload	central decline	winner
Fixed-quality	80%/yr	pay-go (−23%)
Frontier	30%/yr	commit (+22%)

And the break-even decline grid shows committing never wins for commodity models (80%/yr exceeds every reservable discount's break-even), while longer terms always make it worse: a 30% discount breaks even at 57.8%/yr over 12 months but only 24.5%/yr over 36 months. The rule: reserve where prices are sticky (frontier); ride the curve where they fall fast (commodity). Never sign a 3-year inference commit in a market falling >25%/yr.

Chapter 6 — Prompt-cache ROI

Caching looks free, but a cache write often costs more than a normal input token (Anthropic charges up to 2×), while reads are cheap. So it's a bet: pay the write premium to save on reads, and it only pays if you reuse enough before the cache expires. The break-even reuse count is N* = (write − read)/(in − read).

Provider	break-even N*	savings ceiling
DeepInfra / DeepSeek / OpenAI / Fireworks (no write fee)	1.00	50–90%
Anthropic Opus 5-min TTL (1.25× write)	1.28	90%
Anthropic Opus 1-hr TTL (2× write)	2.11	90%

Caching pays almost immediately — most providers from the second call, even the harshest write premium by the third. The read multiplier sets the long-run ceiling (0.1×-read → 90% savings; 0.5×-read → 50%), so for cache-heavy loops that's the number that compounds. Using agent-calc's interval domain, an 8k-token agent prefix reused an uncertain 3–20 times costs $0.088–0.156 cached vs $0.12–0.80 uncached — 80% off at the high end, even with a 2× write premium. This is why the guide's 75% cache-hit assumption holds: agent loops reuse a large prefix 10–20× per task.

Chapter 7 — Agent-loop compounding (the O(K²) tax)

Why is that reused prefix so large? Because a multi-turn agent re-sends its growing context every step — at step k, the input is the base prompt plus every prior tool call and result. Per-step input grows linearly, so cumulative input over a K-step task grows quadratically:

cumulative over K = (δ/2)·K² + (B − δ/2)·K

agent-calc evaluates that polynomial exactly (polynomial/evaluate):

K steps	cum tokens	naive $	cached $	compacted $
10	147,500	0.0384	0.0220	0.0384
50	2,237,500	0.5817	0.3015	0.3267
100	8,225,000	2.1385	1.0896	0.6907

A 100-step task isn't 10× a 10-step task — it's 56×. Your 50th tool call's input costs 10× your first. Caching bends the curve (lowers the constant); only compaction breaks it — capping the context window turns the quadratic linear, and the gap widens without bound as loops get longer. They compose: cached and compacted is read-rate pricing on a linear token count. For any loop past ~20 steps, do both.

This closes the loop on the whole guide: the quadratic re-send is exactly why agent workloads are so input-heavy and so cache-dependent — the entire shared workload profile falls out of this one chapter.

The cheat sheet

Everything above, compressed to decisions:

Decision	Verdict
Which model?	Cheap efficient hybrid (DeepSeek V3.2-class). Cheap output + high success beats everything on cost-per-solved.
One provider or many?	Blend via LP. First ~40% of any jurisdiction/diversity constraint is usually free.
Self-host?	No — for cost. Renting always loses for commodity models; only owned next-gen hardware at >72% utilization wins. Self-host for privacy/control.
Reasoning model?	Only if its output tokens are cheap. A pricey heavy reasoner can be mathematically uncatchable on cost-per-solved.
Retries?	Cascade to a different, stronger model — never retry the same one. 3 tiers ≈ 99.5% coverage at ~6× lower cost than going straight to frontier.
Commit to reserved capacity?	Only for sticky frontier prices, short terms. Never for commodity models in a market falling 50–90%/yr.
Caching?	Always, for any reused prefix. Pays from call 2–3. Prefer 0.1×-read providers.
Long agent loops?	Cap the context window (compaction) past ~20 steps, or pay quadratically.

Reproduce it

The whole thing is a repo. Each chapter is a self-contained script that shells out to agent-calc; each input price lives in data/ with a source URL and observation date.

git clone https://github.com/copyleftdev/rational
make all       # re-run every chapter
make ch04      # or just one

Same inputs → bit-identical outputs, every time. The LLM did the modeling and pulled the data. The math kernel did the math. That division of labor is the only reason I trust any number in this article — and it's the part I'd encourage you to steal.

AI Didn't Invent Slop. It Only Made It Infinite.

Don Johnson — Sat, 27 Jun 2026 00:01:32 +0000

Every generation believes it is witnessing the death of taste.

The printing press was going to flood the world with trash. The cheap camera was going to destroy photography. The internet was going to drown serious thought in blogs, forums, spam, and noise. Smartphones were going to turn every sacred moment into a blurry rectangle.

Now artificial intelligence, we are told, has brought us to the final collapse: endless images, endless songs, endless essays, endless pull requests that compile and ship and mean nothing.

We call it AI slop.

And to be fair, much of it deserves the name.

You can feel it when you see it. The over-polished fantasy portrait. The too-perfect lighting. The fake article that says everything and means nothing. And if you write software: the 600-line PR that passes CI and no human can explain. The README generated by a machine that has never run the code. The test suite with 100% coverage that asserts only that the mock returned the mock.

It is work nobody will remember making.

It has texture but no conviction. Output, but no memory of why.

But AI did not invent slop.

It only made it infinite.

We Have Been Here Before

There was a time when a photograph meant something simply because it was difficult to make.

A camera was not just an object. It was a ritual. Film had to be bought. Shots had to be chosen. Light had to be respected. Mistakes cost money. The photographer carried the burden of scarcity.

Then cameras became cheap, then digital, then they entered every pocket on Earth. Suddenly, humanity became a species of photographers. We photographed lunch, and mirrors, and sunsets we barely watched. We created billions of images, most of them forgotten before the day ended.

Photography did not die.

But the average photograph became meaningless.

That distinction matters. The medium survived. The scarcity did not.

No one calls it "camera slop," but perhaps we should have.

Slop Is What Happens When Creation Becomes Cheap

This is the uncomfortable truth underneath the AI panic:

Whenever the cost of creation collapses, the volume of mediocre work explodes.

That is not a flaw unique to artificial intelligence. It is one of the oldest patterns in media history — and it has names. Sturgeon's Law already told us that ninety percent of everything is forgettable. What changes, across eras, is only how cheaply we can manufacture the ninety percent.

When printing became cheaper, the world got pamphlets, propaganda, gossip sheets, and disposable novels. When desktop publishing arrived, everyone became a designer. When blogging platforms arrived, everyone became a columnist. When GitHub made it trivial to publish code, every weekend produced ten thousand repos that would never see a second commit.

Every tool that democratizes creation also democratizes mediocrity.

That sounds cruel, but it is not. It is simply math. When the gates open, more people enter. Most are beginners. Many are opportunists. Some are brilliant. The room gets louder before it gets better.

AI is not breaking that pattern. AI is accelerating it.

Slop Is a Taste Crisis, Not a Technology Crisis

The real danger of AI slop is not that machines can make things.

The danger is that humans may stop caring whether the things are worth making.

That is the disease. Not automation. Indifference.

Slop is not defined by the fact that AI was used. It is defined by the absence of intention. A human can make slop with a camera. A human can make slop with a paintbrush. A human can make slop with vim and forty years of experience and a million-dollar budget.

Slop is not a medium. Slop is a relationship to creation.

It is what happens when the goal is output instead of meaning. Velocity instead of correctness. Shipped instead of understood.

The machine did not create that hunger. The machine simply feeds it faster.

The System Loves Slop

Slop thrives because the systems around us reward frequency.

Merge faster. Chase the metric. Bump the contribution graph. Worship the green build. Close the ticket. Remove the silence where thinking used to live.

AI fits perfectly into this economy because AI is tireless. It does not sleep, or get bored, or stare at a failing test for three days before finding the one assumption that was wrong from the start.

The pipeline wants endless output. AI can provide it.

That marriage — between machine generation and a culture that measures motion instead of meaning — is where slop becomes industrial. Not because AI is evil. Because infinite production met infinite distribution, and nobody in the middle asked whether any of it mattered.

Copilot Did Not Make You an Engineer

A cheap camera allowed anyone to take a picture. It did not make everyone a photographer.

A laptop allowed anyone to make music. It did not make everyone a musician.

A blog allowed anyone to publish. It did not make everyone a thinker.

Copilot lets anyone generate working code. It does not make everyone an engineer.

This is where the confusion begins. We keep mistaking access for mastery.

The tool can produce the surface of a thing. It can mimic the genre, satisfy the linter, pass the tests it also wrote. It can imitate the grammar of competence.

But engineering is not merely code that runs.

It is selection, restraint, and the refusal to write the clever abstraction that will rot in eighteen months — even though the model offered it, ready to paste, today. It is a human being deciding that one design matters more than another and being able to say why.

AI can generate a thousand implementations in an hour.

The engineer is the one who knows which one should not exist.

The Scarce Resource Was Never the Tool

For decades we confused engineering value with the difficulty of execution.

The person who could write tight assembly possessed rare power. The person who could wrangle pointers, or hold a distributed system in their head, or make the build green, possessed rare power.

But technology has a way of eating technique. Again and again, the machine absorbs what once required years of specialized execution. And every time, people panic and say the craft is dead.

But the craft is not dead. It moves.

When execution becomes easier, taste becomes more important. When production becomes abundant, judgment becomes more important.

When everyone can generate a pull request, the real question becomes: who can read it? Who can see the load-bearing line? Who can look at infinite plausible code and say, "Not that. This."

But Abundance Is Not the Enemy of Craft

Here is the part the optimists get right and the part they get wrong.

The right part: every explosion of mediocrity also expands the possibility of genius. Cheap cameras gave us billions of forgettable images — and handed visual language to people who would never have entered a photography school. Home studios created oceans of bad music — and gave us bedroom producers who reshaped global sound. The early web buried us in spam — and also gave us Linux, Wikipedia, and Stack Overflow, each built from the same flood of amateur contribution that produced the junk.

Democratization tends to move in waves. The first wave is noise. The second is imitation. The third discovers what the medium is actually for.

The wrong part — the part the optimists wave away — is that the third wave is not guaranteed. Plenty of ecosystems hit the noise wave and simply stayed there, or died: local newspapers did not ascend to some higher form, they collapsed and did not come back. "It worked out before" is not a law of nature. It is a bet. The honest claim is narrower: abundance makes a better equilibrium possible. It does not deliver one. Someone has to do the choosing.

But Won't the Machine Just Do the Choosing Too?

This is the strongest objection to everything above, so let's not dodge it.

Every earlier democratization left one thing untouched: a human layer that decided what was good. The press multiplied books, but humans still chose what to read. The camera multiplied images, but humans still chose what to hang on the wall. The scarce resource could shift to judgment precisely because judgment stayed ours.

AI attacks both sides of that deal at once. It floods faster than any reviewer can read — and it is being trained, right now, to do the reviewing: to rank, to critique, to pick the better of two diffs. If the machine can write the thousand implementations, why can't it choose the one that should exist?

Partly, it can. Taste is not magic. Much of it is pattern, and patterns are trainable. An honest version of this argument has to concede that "humans simply judge better" is a moat that will not hold — the same way "humans simply write assembly better" did not hold.

But there is a remainder, and it is not a skill. It is accountability.

Someone has to be the one who says ship it and owns what comes next — the outage, the breach, the regression that reaches a million users, the quiet harm nobody modeled. A system can rank the options. It cannot be answerable for the choice. The engineer who clicks merge is not just exercising sharper judgment than the generator; they are the human a consequence can attach to.

So the moat is not "we judge better." That one erodes. The moat is "we are responsible." Discernment is the craft you sharpen. Accountability is the part the machine structurally cannot take, because responsibility requires someone who can actually be held to it.

The Engineer Becomes an Editor of Infinity

The next great technical skill may not be generation. It may be discernment exercised under that responsibility.

We already have a name for the work: review. And it is moving from the chore at the edge of the job to the center of it.

Watch what AI does to a team. It lifts a junior's output to senior-looking quality almost overnight — clean, idiomatic, plausible. It does not lift their judgment at the same rate. So the senior becomes the bottleneck, reviewing an infinite stream of confident, well-formatted diffs from people (and models) who cannot yet see what is wrong with them. "LGTM" stops being a rubber stamp and becomes the most dangerous sentence in the codebase. The reviewer becomes the rate limiter on quality for the entire organization.

That is the job now. Not typing faster. Reading better.

The engineer of the next decade is part architect, part editor, part skeptic — someone who can run the machine without serving it, who generates without limit, then cuts without mercy. Someone with memory. Someone with scars from the last system that paged them at 3 a.m.

Because when any code can be written instantly, the most valuable thing you can do is refuse to merge most of it.

Slop Is the Shadow of Democratization

AI slop is real. It is everywhere — feeds, marketplaces, search results, issue trackers, and the diff you have to review this afternoon.

But it is not an alien substance. It is the shadow that appears whenever creation becomes easier than judgment. Scarcity used to do some of our filtering for us: difficulty kept the repo smaller, cost slowed people down, the sheer friction of writing it yourself meant you mostly wrote what you meant. Now that friction is gone, and the burden shifts back to us. We have to decide what matters.

The cheap camera did not destroy photography. It destroyed the illusion that every photograph was precious. AI will not destroy engineering. It will destroy the illusion that every act of generation is creative.

And maybe that is necessary. Maybe we are being forced to admit what was always true: the magic was never in the tool. The magic was in the seeing and the choosing — in taking the chaos of a problem and shaping it into something that says:

I was here. I understood this. This mattered.

In the age of infinite slop, that may become the rarest thing of all.

Notes on the Argument

This is a historical argument, not a technological one, and it is a synthesis rather than a discovered law — so here is the lineage it rests on, made explicit so you can check it:

Sturgeon's Law — "ninety percent of everything is crap." The baseline claim that most output in any field is forgettable. (ref)
Clay Shirky's "mass amateurization" — when you remove the barrier to publishing, you don't get less junk, you get "publish, then filter": the filtering moves downstream, onto the reader. This essay's "engineer as editor" is that filter, relocated to code. (ref)
The Jevons paradox — making something cheaper to produce raises total consumption rather than lowering it. Cheaper creation means more creation, not less — now widely invoked for AI. (ref)

The concrete parallels are well documented: the printing press produced a flood of cheap Reformation pamphlets alongside its enduring works (ref); consumer and smartphone photography pushed the world past a trillion-plus photos a year (ref); desktop publishing democratized design (and gave us the "ransom note" era), while content farms like Demand Media industrialized algorithmic filler until Google's Panda update was built to bury it (ref).

The argument deliberately holds two ideas at once: democratization of creation (good — more people gain the tools) and a decline in average quality (the expected cost when participation grows from thousands to millions). They are complementary, not contradictory. Reasonable people can disagree with where it lands. The point is to offer a frame for the debate, not to claim it is settled.

A Note on Collaboration

This essay was written through a collaboration between a human author and AI. The ideas, the thesis, the structure, and every editorial decision came from a human; AI was used as a thinking partner and drafting tool, and the historical claims above were checked against the sources cited.

Writers have always worked with tools — editors, dictionaries, research assistants, cameras, compilers, spell checkers, search engines. AI is another one, albeit a remarkably capable one. The quality of a work has never been determined solely by the sophistication of the tool that helped make it. It has always been determined by the quality of the thinking behind it.

Judge this essay by its reasoning. Not by the instrument used to write it.

(Disclosed per DEV's guidelines for AI-assisted articles.)

Cogs and Sparks: Who Wins When Machines Learn to Speak Human

Don Johnson — Sat, 13 Jun 2026 22:57:15 +0000

Disclosure: this article was written from my own thesis and lived experience, then shaped and edited with AI assistance. I reviewed the argument, revised the structure, and stand behind the ideas here.

For most of the digital age, humans had to learn how to think like machines.

We memorized syntax. We learned rigid menus, brittle interfaces, command lines, templates, ticket flows, and rules that punished ambiguity. We trained ourselves to break imagination into tiny mechanical instructions because computers could not meet us halfway.

To make technology useful, we first had to translate ourselves into its language.

That translation did not just shape software. It shaped people.

The modern workplace rewarded the person who could become a little machine-like: precise, procedural, reliable, consistent, patient with repetition, able to memorize commands and tolerate the tedious work required to keep the system moving.

Those people mattered. They still matter.

They were the cogs.

And for a long time, being a good cog was a survival strategy.

But AI changes the direction of adaptation.

For the first time, machines are learning to move toward us. They are learning language, context, style, pattern, intent, image, tone, code, conversation, and approximation. They are still flawed. They still hallucinate. They still need judgment, taste, and correction.

But the direction is unmistakable.

The machine is no longer asking only:

Can you speak machine?

It is beginning to ask:

Can you show me what you mean?

That shift changes who gets leverage.

The old world favored the cog.

The new world favors the spark.

The Cog Was Never the Problem

A cog is not an insult.

Cogs are dependable. Cogs execute. Cogs maintain order. Cogs remember how the system works when everyone else is chasing novelty. Cogs show up. They keep the lights on. They learn the process and repeat it with precision.

Every serious system needs cogs.

The problem is not the cog.

The problem is a world that taught people the safest way to survive was to become only a cog.

For decades, technical leverage lived behind gates. If you could not code, you could not build software. If you could not design, you could not ship a product experience. If you could not navigate the tools, you needed someone else to translate your idea into execution.

That gatekeeping was not always malicious. Much of it was just mechanical reality. Computers were powerful, but they were not generous. You had to meet them on their terms.

So the system rewarded people who could memorize the terms.

But many people never fit that world cleanly.

They were visual thinkers in text-only systems. Writers in spreadsheet cultures. Artists trapped behind tool complexity. Teachers with product instincts but no engineering team. Generalists who could see connections but lacked the credentialed path to execute them.

They were sparks in a cog-shaped world.

What a Spark Does

A spark sees possibility before there is a process.

A spark experiments before there is permission.

A spark can look at two unrelated things and feel the bridge between them before they can fully explain it.

Sparks ask strange questions. They remix. They test. They wander. They notice tone, friction, story, behavior, emotion, and timing.

Sparks are artists, writers, designers, teachers, builders, founders, comedians, strategists, researchers, tinkerers, and curious people who were often told they were impractical because their value did not fit the spreadsheet.

In the old world, sparks often needed cogs to translate them.

In the AI world, sparks get a new kind of leverage.

AI lowers the cost of execution. It can draft, summarize, code, calculate, organize, translate, simulate, critique, generate, and iterate. It can turn a vague sketch into a prototype, a paragraph into a campaign, a napkin idea into a working demo, a conversation into a plan.

That does not make the work automatic.

It moves the bottleneck.

The question is no longer only:

Who can perform the task?

The question becomes:

Who can imagine the right thing worth doing?

The Bottleneck Moves Upward

When execution gets cheaper, taste gets more expensive.

When syntax gets easier, judgment matters more.

When tools can produce ten versions in seconds, the valuable person is not the one impressed by abundance. It is the one who can tell which version is alive.

That is the real AI transition.

Not from human to machine.

From memorization to judgment.

From task completion to problem framing.

From obedience to direction.

From labor as identity to creation as leverage.

The person who can only follow instructions will increasingly compete with machines that follow instructions faster.

The person who can create better instructions, better constraints, better questions, better taste, and better visions becomes more powerful.

This is why the spark matters.

A spark does not merely ask an AI to make something. A spark senses what is missing. A spark knows when the output is technically correct but emotionally dead. A spark can say:

No, not that. Make it quieter. Make it sharper. Make it feel like a system designed by someone who has actually lived this problem.

That is not button-pushing.

That is direction.

What This Means for Developers

Developers are in a strange position because we are both protected and exposed by this shift.

We understand the machinery better than most people. That matters. AI-generated code still needs review, architecture, debugging, security judgment, and production discipline. The machine can create plausible code faster than it can understand consequences.

But we should not mistake that for safety.

If our value is only that we remember syntax, our value is shrinking.

If our value is that we can reason about systems, frame problems, notice edge cases, understand users, evaluate tradeoffs, and build things that survive contact with reality, our value compounds.

The best developer in the AI era is not the one who refuses the tool.

It is also not the one who blindly trusts it.

It is the one who can use the tool without surrendering judgment.

The Cog Has to Remap

If you were trained by the old world, this shift can feel insulting.

You spent years learning the hard way. You memorized commands. You built discipline. You earned your scars. Now a beginner can prompt a machine and get something that looks, at first glance, like the thing you worked years to produce.

That reaction is understandable.

But resentment is a bad strategy.

The cog does not need to disappear. The cog needs to remap.

Here are the shifts I think matter most:

From certainty to experimentation. In the old world, being right early was rewarded. In the AI world, discovery often comes from fast comparison.
From memorization to judgment. Facts are easier to retrieve. Syntax is easier to generate. But judgment remains scarce.
From task completion to problem framing. A weak frame creates polished garbage. A strong frame creates leverage.
From obedience to imagination. When machines can produce correct-enough work quickly, direction becomes practical.
From effort to impact. If something becomes easier, it does not become meaningless. It means the value moves upward.

The question is not whether you can still do the hard thing manually.

The question is whether you can aim the system toward something worth building.

The New Creative Class Is Broader Than We Think

The painter who could not code can now prototype an app.

The writer who could not design can now shape a visual world.

The teacher who could not afford a product team can now build a learning experience.

The mechanic can become an automation builder.

The founder without funding can test the first version before asking anyone for permission.

The future may not belong only to the people who were best at computers.

It may belong to the people who were best at being human, now armed with computers that can finally keep up.

This does not mean technical skill stops mattering.

Deep expertise still matters. Engineering still matters. Precision still matters. In fact, expertise may matter more because AI can produce confident nonsense at scale. Someone has to know what good looks like.

But the ceiling rises for people who were previously blocked by technical gates.

The next great product designer might be a poet.

The next great software founder might be a teacher.

The next great systems thinker might be someone who never called themselves technical because the old tools made them feel stupid.

They were not stupid.

They were early to a world that had not built the right interface yet.

The Hybrid Wins

The spark alone is not enough.

Unfocused creativity becomes noise. Ideas without discipline become vapor. Taste without execution becomes performance.

The cog alone is not enough either.

Perfect execution of the wrong thing is still waste. Process without imagination becomes maintenance of a dying machine.

The people who win are hybrids.

They have the discipline of the cog and the imagination of the spark. They can execute, but they can also question the premise. They can use the tool, but they are not hypnotized by it. They can move fast without confusing motion for meaning.

The cog must learn to spark.

The spark must learn to aim.

And the best builders will become both: precise enough to finish, imaginative enough to invent, and adaptable enough to keep changing as the tools change.

The Machine Is Bending Toward Us

The great mistake is to treat AI as merely a faster machine for old work.

That is the shallow reading.

The deeper reading is that AI changes who gets to participate. It gives language, imagination, taste, and judgment a new kind of leverage. It lets people who were once trapped outside the machinery step closer to the center.

For years, humans had to bend toward the machine.

Now the machine is bending toward humans.

That is not the end of human value.

It is the beginning of a different test.

Not:

Can you remember the command?

Not:

Can you tolerate the process?

Not:

Can you make yourself small enough to fit the machine?

But:

Can you see what is possible?

Can you tell what is good?

Can you aim the spark before it burns out?

That is the work now.

AI-assistance note: I used AI to help structure and edit this essay, but the central argument, final review, and publication responsibility are mine.

Topics: #AI #Programming #DeveloperCareer #Creativity

Stop Feeding Agents Raw Data

Don Johnson — Tue, 09 Jun 2026 06:05:05 +0000

I used to think the problem was the agent.

I would hand it a large JSON export and ask a reasonable question: what changed, what looks risky, what should we investigate before release?

It would find something. It always found something.

But it missed fields. It over-indexed on irrelevant values. It hallucinated patterns in JSON that weren't there. It noticed one dramatic-looking record and ignored the boring distribution that made the record meaningful. So I tried the usual fixes: stricter prompts, longer instructions, bigger context windows, more examples.

The real problem was simpler.

I was handing the agent noisy context.

Not bad data. Bad context.

Raw structured data is full of repetition, incidental shape, schema noise, nulls, IDs, timestamps, nested boilerplate, and fields that only mean something in aggregate. Humans struggle with that. Agents do too.

If you paste a huge API response, GitHub export, log dump, CSV, or source tree into a model and ask it to "find what matters," you are asking the model to do two jobs at once:

Measure the data.
Reason about the measurements.

Those are different jobs.

Vajra is my attempt to split them apart.

Before the model reasons, Vajra measures. Before the agent plans, Vajra reduces the raw input into evidence.

The Agent Context Problem

The default agent workflow is still too close to copy and paste.

Give the agent a repository. Give it a JSON dump. Give it a pile of logs. Give it a directory of semi-structured files. Then ask it to infer structure, identify anomalies, detect drift, preserve evidence, and decide what to do next.

That sounds powerful, but it is a bad division of labor.

Agents are useful for interpretation, synthesis, planning, and tradeoff analysis. They are not the best tool for counting every path in a nested document, computing entropy across values, comparing two distributions, detecting type instability, or producing a repeatable fingerprint of a shape.

Those jobs should be done by boring software.

Reliable software.

Software that returns the same answer every time.

That is the layer I wanted in front of the agent.

What Vajra Does

Vajra is a Rust CLI and library for analyzing structured data. It started from a simple frustration with large JSON, then grew into a signal layer for things agents routinely ingest:

structured data like JSON, NDJSON, YAML, CSV, and TSV
operational artifacts like Markdown, PDF text, CPU profiles, and strace logs
repositories, GitHub exports, and source code parsed through tree-sitter

The goal is not fuzzy summarization. The goal is stable measurement.

For example:

vajra inspect payload.json
vajra stats events.ndjson
vajra anomalies batch.json
vajra drift baseline.json current.json
vajra essence payload.json --profile ai --format compact-ai --budget 500 --redact
vajra inspect src/main.rs --input-format source --lang rust --semantic-paths
vajra governance commits.json

That gives you paths, types, fingerprints, entropy, cardinality, outliers, rarity, schema drift, source-code structure, contributor concentration, and compact essences that are designed to be read by humans or agents.

The important word is stable.

Same input. Same config. Same version. Same result.

That matters more than it sounds.

If an agent is going to call a tool, update a plan, make a recommendation, and then call the tool again after a change, the tool cannot behave like a mood ring. It has to be an instrument.

Raw Data Is Not Context

Here is the mistake I kept making.

I treated "available to the model" as equivalent to "usable by the model."

They are not the same.

A 5 MB JSON export may fit in a context window. That does not mean it is good context. Most of those tokens may be repeated object structure, duplicated IDs, common timestamps, low-signal values, or fields whose meaning only appears after aggregate analysis.

Vajra turns that raw input into a smaller set of measured claims.

Instead of giving an agent every record and hoping it notices that one path is unstable, give it a measured signal shaped like this:

Path: $.claims[*].status
Dominant type: string
Cardinality: 7
Entropy: 1.82
Rare values: reversed_manual, pending_external_review
Null rate: 0.00

Instead of asking it to eyeball schema drift between two payloads, give it a drift report shaped like this:

Removed path: $.member.coverage.plan_id
Added path: $.member.coverage.policy_ref
Type changed: $.claim.amount string -> number
Distribution drift: $.claim.status high
Severity: critical

Instead of making it inspect a full repository export, give it project-health signals such as:

Bus factor: 2
Contributor concentration: high
One-commit contributor rate: 0.61
Most active author share: 0.44
Velocity trend: declining

These examples are representative shapes, not a promise that every command emits exactly these lines. The point is the workflow: let the tool compute the measurements, then let the agent reason from them.

Now the agent can do the work it is actually useful for.

It can reason about consequences. It can ask whether the drift is intentional. It can propose a migration plan. It can decide what evidence belongs in a report. It can open an issue with the relevant fields instead of a blob of raw data.

Measure, Then Reason

The agentic workflow I want looks like this:

raw data / repo / logs / API export
    -> deterministic analysis
    -> compact evidence bundle
    -> agent reasoning
    -> action plan or code change
    -> re-analysis
    -> verification

That loop is different from the usual paste-and-hope loop.

The agent has fewer reasons to infer measurements from raw records. It gets to reason from measured facts.

Vajra can sit before the model and decide what deserves attention. It compresses without pretending to be creative. It surfaces anomalies without pretending every anomaly is automatically important. It preserves paths and evidence so the agent can point back to the data.

That is the difference between a summary and an instrument.

A summary says, "Something changed."

An instrument says, "This path changed, this value distribution moved, this type is unstable, and here is the evidence."

A Concrete Agent Workflow

Imagine an agent responsible for reviewing a GitHub project before a release.

The naive version gets a large export of issues, pull requests, commits, and releases. It tries to infer project health from raw records. It may notice recent failures, but miss contributor concentration. It may summarize issue titles, but miss stale ownership patterns. It may produce a confident report with weak evidence.

The Vajra version starts by measuring first:

vajra ingest-github owner/repo --output .vajra/repo
vajra governance .vajra/repo/commits.json --format json > .vajra/governance.json
vajra core-team .vajra/repo/commits.json --format json > .vajra/core-team.json
vajra anomalies .vajra/repo/issues.json --format json > .vajra/issue-anomalies.json
vajra essence .vajra/repo/issues.json --profile ai --format compact-ai --redact > .vajra/issues.essence.json

Now the agent does not need to start by reading everything.

It reads the measurements.

Instead of producing a vague release memo, it can flag specific evidence-backed risks:

ownership concentrated in two maintainers
unusual issue clusters before release
repeated labels or states that deserve triage
fields that changed shape across exports
anomalies that need business context before they are treated as problems

If the agent recommends action, the action can be checked.

Did the generated report cite the right paths? Did the relevant anomaly disappear or become explainable? Did a migration remove the type instability? Did the same input produce the same output after a refactor?

That is the kind of feedback loop agents need.

Why Determinism Matters

Agentic systems are already probabilistic at the reasoning layer. The tools around them should not add unnecessary randomness.

If a tool summarizes differently every time, the agent's plan changes for reasons that are hard to debug. If a score cannot be decomposed, the agent cannot explain it. If a drift report is not stable, CI cannot trust it. If a fingerprint changes because object keys were ordered differently, the tool is measuring formatting noise instead of structure.

Vajra is built around boring constraints:

deterministic ordering
stable fingerprints
explicit paths
explainable scores
profile-driven reduction
no silent mutation of source data

It is not glamorous. It is what makes the output usable as agent context.

Agents do not just need more context. They need context they can trust.

What This Does Not Solve

This layer does not make judgment disappear.

Vajra can measure structure and distributions. It does not decide whether an anomaly is business-important. It can flag contributor concentration. It cannot know whether that concentration is a temporary release push, an organizational risk, or just how a small project works.

Compact essences are evidence bundles. They are not replacements for source data when auditability, legal review, or incident response requires the original records.

And if sensitive data is involved, it should be redacted or minimized before it is sent to any model or external service. That is why the CLI supports --redact, but the operational policy still belongs to you.

The point is not to remove human or agent judgment.

The point is to stop wasting judgment on work that deterministic tools can do better.

Where This Goes

Serious agent systems will not be built from prompts alone.

They need toolchains that produce reproducible intermediate artifacts. They need ways to shrink context without losing evidence. They need stable measurements before interesting reasoning.

That is the bet behind Vajra.

Not that agents should know everything.

That agents should be handed better instruments.

Because when you stop feeding agents raw data, you give them a better job:

reasoning from evidence.

The Container Runtime Nobody Told You About (And Four Others)

Don Johnson — Tue, 26 May 2026 20:19:00 +0000

Here's something the container ecosystem doesn't say loudly enough: runc is not the only option, and for a growing number of production workloads, it's the wrong one.

AWS Lambda doesn't run your function in a Docker container. It runs it in a Firecracker microVM. Fly.io's Machines? Firecracker fork. Google's multi-tenant GKE nodes? gVisor. Cloudflare Workers? WASM. These companies didn't reach for exotic runtimes because they were bored — they reached for them because the default isolation model was insufficient for their threat model, their latency requirements, or both.

This article takes one tiny Go HTTP server and runs it through all five of them: runc/distroless, gVisor, Kata + QEMU, Kata + Firecracker, and WASM/WASI. You'll see exactly what changes (almost nothing), what the real numbers look like, and — most importantly — which runtime belongs in which situation.

TL;DR: gVisor, Kata, and Firecracker all run the exact same 3 MB OCI image — only --runtime=X changes. WASM is a different compilation target entirely. Cold-start ranges from ~20 ms (runc) to ~500 ms (Kata/QEMU), with Firecracker splitting the difference at ~125 ms. Request latency overhead at steady state is shockingly small across all of them. The real cost is memory and compatibility, not throughput.

The App

Before the runtimes, the subject. A Go HTTP server with one meaningful endpoint:

RUNTIME_NAME is injected at docker run time. Everything else — Go version, arch, PID, uptime — is live from inside whatever sandbox is holding it. When the runtime changes, the response field tells the story.

Runtime 1: Distroless + runc

What it is

The default Docker runtime (runc) but with a distroless base image. No shell, no package manager, no apt, no curl. Just the Go binary and CA certificates.

The image comes out at 3.0 MB. Alpine would be ~18 MB. Ubuntu ~80 MB.

The honest security story

Distroless does not change your isolation model. The container still shares the host kernel. What it does is remove every tool an attacker would use after a successful exploit — no shell to drop into, no package manager to pull more tools from, no /tmp scripts to run. You're not preventing the breach; you're making the post-breach environment hostile.

Ultimate use cases

Internal microservices in a trusted, single-tenant cluster
GitOps pipelines where you control every image in the registry
Replacing fat Alpine images — the size drop alone is worth it
The security baseline every team should hit before adding runtime overhead

Runtime 2: gVisor (`runsc`)

What it is

gVisor ships a user-space Linux kernel called the Sentry — written in Go — that runs alongside your container. Every syscall your container makes goes to the Sentry. The host kernel never sees your container's syscalls.

# Same 3 MB image. One flag.
docker run --rm --runtime=runsc \
  -p 8080:8080 -e RUNTIME_NAME=gvisor \
  micro-containers

The Sentry re-implements the Linux ABI. In ptrace mode it intercepts via ptrace; the newer Systrap mode (shipped 2023, ~2× faster) uses seccomp to intercept. Either way, a kernel exploit in your container cannot reach the host kernel — there is no direct path.

The honest security story

gVisor's threat model is syscall isolation. A container escape via a kernel CVE (your dirty_pipe, your runc breakout) is stopped at the Sentry. But gVisor is not a VM — the container still shares memory, CPU, and the host's network stack at some layers. It's a strong sandbox, not a hard boundary.

2025 state of the world

GKE Sandbox is gVisor, enabled with a single node pool annotation
Systrap mode is now the default — nearly removes the performance cliff that made early gVisor a tough sell
GPU support is production-ready for A100/H100 via vGPU passthrough — relevant if you're sandboxing AI inference workloads

Ultimate use cases

CI/CD runners — the #1 production use case. GitHub Actions self-hosted, GitLab runners, Buildkite agents that execute arbitrary user pipelines. You don't control the code; gVisor limits the blast radius.
ML inference APIs where users submit model weights or custom code — you can't trust what's in those pickles
SaaS plugin execution — any platform that lets users run custom logic (Zapier-style automations, Retool actions, webhook processors)
Cloud IDE backends — Codespace-style environments where each user gets a container that feels like root

Runtime 3: Kata Containers (QEMU VMM)

What it is

Kata Containers boots a lightweight QEMU MicroVM per container. Your app runs inside a VM with its own kernel. containerd sees an OCI runtime; your process sees a dedicated Linux instance.

docker run --rm --runtime=kata-runtime \
  -p 8080:8080 -e RUNTIME_NAME=kata-qemu \
  micro-containers

The host sees a qemu-system-x86_64 process — nothing inside leaks out. The container image is mounted via virtiofs. The kernel boundary is real.

The honest security story

Kata/QEMU is the only option here that provides a true hardware-enforced boundary between container and host. gVisor is software isolation. Kata is a VM. If your threat model requires that a kernel exploit inside the container cannot affect the host, Kata is the answer.

2025 state of the world

Kata 3.x ships with confidential container support: Intel TDX and AMD SEV-SNP give you hardware-attested memory encryption. The host operator can't inspect container memory — relevant for regulated data.
Cloud Hypervisor is now a supported VMM alternative to QEMU, lighter and faster to boot
Confidential Containers (CoCo) as a CNCF project wraps Kata + hardware attestation into a first-class primitive — watch this space

Ultimate use cases

PCI-DSS, HIPAA, FedRAMP — when the compliance checklist literally says "VM-level isolation," Kata is the only container runtime that checks that box without running actual VMs
Financial services — trade processing, settlement systems, anything touching payment card data
Healthcare data pipelines — PHI processing where you need a kernel boundary in the audit trail
Multi-tenant databases — giving each tenant a database that physically cannot escape its VM
Government/defense workloads — environments where the security control plane doesn't trust the container runtime

Runtime 4: Kata + Firecracker VMM

What it is

Firecracker was built by AWS in 2018 specifically for Lambda and Fargate. It replaces QEMU as Kata's VMM. The device model is stripped to the minimum a serverless function needs: one network interface, one block device, one serial port. No BIOS. No PCI bus. No USB enumeration. No legacy device emulation of any kind.

# Kata reads configuration-fc.toml and invokes Firecracker instead of QEMU
docker run --rm --runtime=kata-fc \
  -p 8080:8080 -e RUNTIME_NAME=kata-firecracker \
  micro-containers

Cold start drops from ~500 ms (QEMU) to ~125 ms. Memory overhead drops by nearly half.

The honest security story

Same VM isolation guarantee as Kata/QEMU — a dedicated kernel per container. The tradeoff for the speed gain is device compatibility: no GPU passthrough, no USB, fewer PCIe options. For stateless functions, you don't need any of that.

2025 state of the world

Firecracker 1.7+ — production-stable, used in billions of Lambda invocations per day. AWS open-sourced it and it ships new major versions regularly.
Fly.io Machines use a Firecracker fork as the core primitive — every fly machine run is a microVM
AWS Serverless Aurora uses Firecracker to isolate query execution environments
Confidential Firecracker is in active development — combining Firecracker's boot speed with AMD SEV memory encryption

Ultimate use cases

Serverless function platforms — this is what Firecracker was made for. If you're building the next Lambda, Railway, or Render, Firecracker is the substrate.
AI/ML inference bursts — LLM inference is bursty; Firecracker's 125 ms cold start makes scale-to-zero viable. A GPU instance spun up with Firecracker can take traffic in under a second.
Short-lived test runners — each test run gets a clean VM, boots in 125 ms, exits, gets GC'd. No shared state, no contamination between runs.
Multi-tenant job queues — background jobs that process user-submitted data. Firecracker gives you VM isolation at a price point runc used to own.
Preview environments — spin up a full-stack environment for each PR, destroy it on merge. The economics work at ~125 ms boot + minimal memory overhead.

Runtime 5: WASM / WASI preview1

What it is

The binary is compiled to WebAssembly with Go's WASI target — an entirely different binary, an entirely different image:

The resulting image: 3.1 MB scratch base + one .wasm binary. The sandbox is enforced at the language-runtime level — no syscalls, capabilities explicitly granted by the host.

The honest HTTP story

net/http doesn't work in WASI preview1. The spec has no socket API. This demo outputs JSON to stdout. That's not a cop-out — it's the current state of the standard. The wasi-http proposal shipped as part of WASI 0.2, which is ratified. Fermyon Spin 2.x implements it today. Go's WASI 0.2 support is in progress.

2025 state of the world

WASI 0.2 (Component Model) is ratified and shipping in wasmtime, WasmEdge, and Fastly Compute. wasi-http is a real, stable interface.
Docker+Wasm is GA in Docker Desktop 4.27+ — run a WASM container with --platform=wasi/wasm and a containerd shim
Fermyon Spin 2.x compiles Go to WASM with a full HTTP server abstraction — the framework paper over the WASI/HTTP gap today
WasmPlugin in Kubernetes — Envoy and Istio support WASM plugins for custom policy, auth, and observability logic
Extism — a cross-language WASM plugin framework that lets you embed sandboxed user code in any Go/Rust/Python host

Ultimate use cases

Edge functions — Cloudflare Workers, Fastly Compute, Deno Deploy, and Vercel Edge Functions are all WASM at the bottom. The same binary runs in London, Singapore, and São Paulo with no containers to spin up.
Cross-platform CLI tools — compile once, run on Linux/macOS/Windows/browser with no CGO, no cross-compilation matrix
Sandboxed plugin systems — give users scriptable extensions with a real capability boundary. Zellij (terminal multiplexer) uses WASM plugins; VS Code extensions are moving this direction.
Business logic in the browser + server — tax calculation, pricing rules, validation logic that needs to run identically client-side and server-side
AI prompt/response filters — fast, sandboxed, hot-reloadable logic at the edge before a request hits your inference endpoint

The Numbers

All OCI runtimes run the same 3 MB distroless image. The distroless/runc row is measured hardware; microVM rows are reference numbers from project documentation — run make bench-md for your own numbers.

Runtime	Image	Cold Start	p50	p95	Memory
distroless / runc	3.0 MB	~20 ms¹	0.28 ms	0.41 ms	6.9 MB
gVisor (runsc)	3.0 MB	~50 ms	~0.5 ms	~1.0 ms	~18 MB
Kata / QEMU	3.0 MB	~500 ms	~0.8 ms	~1.5 ms	~52 MB
Kata / Firecracker	3.0 MB	~125 ms	~0.7 ms	~1.3 ms	~28 MB
WASM (wasmtime)	3.1 MB	N/A²	—	—	—

¹ First run ~174 ms (overlay FS init); subsequent ~20 ms on warm cache.
² WASM has no HTTP server in wasip1; exec time ~8 ms for the stdout variant.

Three things the numbers tell you that prose doesn't:

Image size is not the story. All five runtimes land at 3–3.1 MB. Switching from runc to Firecracker doesn't touch your image pipeline.

Latency overhead at steady state is negligible. Even inside a Kata VM, p50 latency is under 1 ms. The isolation boundary costs you cold-start and memory, not throughput. If you're worried about runtime overhead on a running service, stop — that's not where the overhead lives.

Firecracker hits the practical sweet spot. 125 ms is the number AWS decided was fast enough for Lambda. 500 ms (QEMU) is where users start feeling it. Firecracker lands right where microVM isolation becomes viable for interactive-latency workloads.

The Decision Framework

Stop asking "which is more secure?" Start asking "what's my threat model?"

Your tenants are you. You control every image, every workload, every user. → runc + distroless. Fast, simple, no overhead.

Your tenants are your users, but you control the runtime environment. CI runners, SaaS execution engines. → gVisor. Drop-in, no KVM, syscall isolation stops the most common container escapes.

You have compliance paperwork that says "VM-level isolation." → Kata + QEMU. The only option that satisfies an auditor asking for a kernel boundary.

You're building a platform. Functions, jobs, preview environments, AI inference. Cold-start matters. → Kata + Firecracker. This is the production-proven answer for platforms.

Your code runs everywhere, or users supply the code. Edge compute, plugins, sandboxed scripts. → WASM/WASI. The sandbox is portable; the isolation model is capability-based, not kernel-based.

Run It Yourself

git clone https://github.com/copyleftdev/micro-containers
cd micro-containers

make check        # see what's installed
make bench-fast   # quick smoke-test with 20 samples
make bench-md     # full benchmark → Markdown table

Each runtime has an install.sh in runtimes/<name>/. The benchmark driver skips unavailable runtimes and tells you exactly what to install.

Source, Dockerfiles, benchmark driver, and install scripts: copyleftdev/micro-containers

The Linux Commands You Forgot Exist (And Why AI Workflows Make Them Relevant Again)

Don Johnson — Mon, 25 May 2026 21:28:22 +0000

These weren't in your bootcamp. They're not in most tutorials. They've been quietly available on every Linux box since before "AI workflow" was a phrase — and they're more useful now than they've ever been.

Try it yourself: clone linux-archaeology-lab, run bash setup.sh, and every command in this article has a working exercise waiting for you.

`watch` — monitor anything without a single line of code

watch runs a command on a repeating interval and fills your terminal with the refreshing output. That's it. No loop, no sleep, no script.

Why it's back: AI inference runs take time. watch -n1 nvidia-smi is the fastest way to see GPU memory climb and fall without touching the model process at all. watch -n2 'ls outputs/ | wc -l' tells you how far a batch job has gotten. One flag, zero instrumentation.

`tee` — two destinations, one stream

tee reads stdin and writes it to both stdout and a file simultaneously. Not sequentially — simultaneously, as data flows.

The pattern that comes up constantly in AI work:

agent-command 2>&1 | ts '[%H:%M:%S]' | tee run-$(date +%Y%m%d-%H%M%S).log

You see it live. It's in a timestamped log. Stderr is captured. One pipeline, three things handled.

`pv` — a progress bar for any pipeline

pv is a transparent pipe segment. Data passes through it unchanged; it prints throughput, elapsed time, and a progress bar to stderr.

You don't modify the commands on either side. You just insert pv into the middle:

cat data.jsonl | pv | python3 process.py

A blinking cursor becomes a progress bar with an ETA. For long inference batches — thousands of rows, slow API calls, large embeddings — pv turns a black box into something you can actually reason about.

`ts` — timestamp every line of output

ts prepends a timestamp to every line it receives on stdin. Nothing else.

The power is in the relative mode:

agent-command | ts -s '%.s'

Each line is prefixed with the time since the previous line — so you can see exactly where an agent spent 4 seconds between steps. No profiler. No code changes.

ts is from moreutils. Install once: sudo apt install moreutils.

`sponge` — safe in-place pipeline transforms

This command exists to solve one specific problem, and it solves it perfectly.

The shell opens output files for writing before the pipeline starts — which truncates the file before it's been read. sponge soaks up all of stdin into memory first, then writes when it gets EOF. The file is safe.

sort file.txt | sponge file.txt        # safe
python3 -m json.tool cfg.json | sponge cfg.json   # safe
grep -v DEBUG app.log | sponge app.log            # safe

Also from moreutils.

`column` — readable tables without Python

column formats delimited input into aligned columns. One flag for the delimiter, one flag for table mode.

Before:

model   provider    params  context_k
llama-3.1-8b    Meta    8B  128
mistral-7b  Mistral AI  7B  32

After column -t -s $'\t':

model          provider      params  context_k
llama-3.1-8b   Meta          8B      128
mistral-7b     Mistral AI    7B      32

For any command that emits structured text — tool call logs, benchmark results, model comparisons — column makes it scannable in one pipeline stage. No pandas. No formatting code.

`comm` — surgical set operations on text files

comm compares two sorted files and gives you three columns: lines only in file A, lines only in file B, lines in both. Suppress any column you don't need.

The comm -12 (intersection) and comm -23 (A minus B) patterns are the correct answer to "what's consistent across these two model runs?" and "what did run B drop that run A had?" — in one command, no Python, no diff | grep.

Process substitution makes it flexible:

comm -23 <(sort run-a.txt) <(sort run-b.txt)

`tac` — read any file from the bottom

tac is cat spelled backwards. It reverses line order.

The killer use case:

tac agent.log | grep -m1 'ERROR'

Find the most recent error in a log without reading the whole file. -m1 stops at the first match — which, in a reversed file, is the last occurrence. No tail, no awk, no Python.

Pair with head for newest-N-lines: tac logfile | head -20.

`vidir` — batch rename in your text editor

vidir opens a directory listing in $EDITOR. You rename files by editing text. You delete files by deleting lines.

1   outputs/output-1.txt
2   outputs/output-2.txt
3   outputs/output-3.txt

Run :%s/output-/summary-/g, save, quit. All three files renamed. Your editor's full power — regex, macros, multicursor — applied to filesystem operations.

Replaces rename 's/pattern/replacement/' * (Perl regex you have to look up) and for f in *; do mv ...; done (quoting hell).

Also from moreutils.

`parallel` — concurrent tasks without threading code

GNU parallel is xargs -P with readable syntax, job control, retries, and output you can actually parse.

The batched inference pattern:

cat prompts.jsonl | parallel -j4 --pipe --block 10k inference-tool

Four workers, each receiving a 10K block of JSONL. No threading code. No async boilerplate. Output is ordered and labeled with --tag. Failed jobs retry with --retries 3.

For AI workloads — running the same prompt against multiple models, calling an embedding API for each document in a dataset, processing output files — parallel turns a sequential loop into concurrent execution in one command.

Load the reasoning skill into Claude Code

Knowing the commands is one thing. Knowing which one to reach for is another.

The lab repo ships .claude/skills/linux-archaeology.md — a Claude Code skill that maps natural-language descriptions to the right command. Describe your problem and it reasons through the answer:

"I need a progress bar for this pipeline" → pv
"How do I timestamp my agent logs?" → ts

"I want to rename a batch of files without writing a script" → vidir

Install in any project:

mkdir -p .claude/skills
curl -sL https://raw.githubusercontent.com/copyleftdev/linux-archaeology-lab/main/.claude/skills/linux-archaeology.md \
  > .claude/skills/linux-archaeology.md

The thread

watch, tee, pv, ts, sponge, column, comm, tac, vidir, parallel — none of these are new. They were built for the terminal long before AI workflows existed. But AI workflows surfaced the exact problems they solve: long-running processes with no visibility, streams that need to go two places, logs that need timestamps, files that need in-place transforms, tasks that need to run in parallel.

The tools were there. The problems caught up.

Run every command in this article against real data:
→ linux-archaeology-lab — clone it, bash setup.sh, open exercises/.

Which one did you not know about? Drop it in the comments.

Tags: linux productivity devtools ai bash

Sister article: The git Commands You Forgot Exist

The git Commands You Forgot Exist (And Why AI Workflows Make Them Relevant Again)

Don Johnson — Mon, 25 May 2026 20:34:38 +0000

Most devs know git commit, git push, git stash. Then there's a whole floor below that nobody visits.

Try it yourself: clone git-archaeology-lab, run bash setup.sh, and every command in this article has a working exercise waiting for you.

`git worktree` — multiple checkouts, one repo

This one is criminally underused. By default, git lets you have exactly one working directory per clone. git worktree breaks that constraint.

You now have two fully independent working directories — same repo, different branches — with no stashing, no switching, no context loss.

Why it's back: AI coding agents. When you're running Claude Code or Cursor on one branch and need to review a hotfix on another, switching branches mid-session breaks everything. git worktree lets both live simultaneously. Each agent gets its own tree. No collisions.

`git bisect` — binary search your blame

You have a bug. You know it didn't exist three weeks ago. You have 200 commits in between. git bisect turns that into about 8 tries.

The real power is git bisect run — pass any command that exits 0 (good) or non-zero (bad). Your whole test suite, a curl health check, a grep — anything that detects the regression works as the oracle. git drives itself to the culpable commit with zero manual steps.

`git rerere` — never resolve the same conflict twice

rerere = Reuse Recorded Resolution.

Enable it once globally and forget it's there — until you notice conflicts silently resolving themselves. The payoff is most obvious during long interactive rebases where the same conflict appears across a dozen commits.

`git log -S` — the pickaxe

You want to know when a specific string was added or removed. Not which commit touched the file — which commit changed this exact text.

-S searches diff content, not commit messages. It finds commits where the string's count in a file changed — added or removed. Even after a secret is deleted from HEAD, git log -S finds the commit that introduced it. Deletion isn't enough. Rotate the credential.

`git notes` — annotate commits without touching them

Commits are immutable. But sometimes you want to attach information to one — a JIRA ticket, a test result, a deployment timestamp — after the fact, without rewriting history.

Notes live in a separate ref (refs/notes/commits) and don't alter the commit hash. Great for CI/CD pipelines that want to annotate commits with build metadata without touching history.

`git range-diff` — diff of diffs

You rebased a branch. You want to verify the rebase didn't silently mangle any patches. git range-diff compares two sequences of commits patch-by-patch.

= means the patches are equivalent. ! means something drifted — and git shows you the diff-of-diffs inline. Code review tools don't show you this. Only range-diff does.

`git sparse-checkout` — check out only what you need

Mono-repo with 40 packages and you only work in two? Sparse checkout lets you tell git to only materialize specific paths.

Everything else exists in git history but won't appear on disk. Your editor is faster. Your find commands are sane. In an AI workflow, sparse checkout reduces the surface area your agent sees — fewer files means faster greps, leaner context windows, and no accidental edits to packages you don't own.

`git commit --fixup` + `git rebase --autosquash`

You committed, reviewed your own diff, spotted a typo in the third commit back. There's a clean path that doesn't require a painful interactive rebase.

--fixup is the honest alternative to git commit --amend. Amend rewrites HEAD; fixup targets any prior commit and leaves an auditable trail until the rebase squashes it.

`git blame -C` — follow moved code

Standard git blame breaks when code moves between files. -C tells git to detect copied or moved content and attribute it correctly.

Any time you move functions between files, copy-detection blame gives you the true lineage — who decided this logic should work this way, not just who moved it.

`git bundle` — the git sneakernet

No network. Air-gapped machine. USB drive. git bundle packs your entire repo (or a range of commits) into a single file you can carry anywhere.

The bundle is a valid git remote. You can clone from it, fetch from it, inspect it. It's just a file.

Load the reasoning skill into Claude Code

Knowing the commands is one thing. Knowing which one to reach for in the moment is another.

The lab repo ships a Claude Code skill file at .claude/skills/git-archaeology.md. When you open the repo in Claude Code, the skill is available automatically. Describe your problem in plain English — "I need to find when this bug appeared", "I keep resolving the same conflict", "can I have two branches open at once?" — and it reasons through the right command for your specific situation.

To install it in any of your own projects:

mkdir -p .claude/skills
curl -sL https://gist.githubusercontent.com/copyleftdev/c9c12ea89231680d5ef4a68785ecc125/raw/git-archaeology.md \
  > .claude/skills/git-archaeology.md

The thread

These aren't obscure for obscurity's sake. They were built for problems that are more common now than they were in 2012 — big repos, parallel workstreams, automated agents, compliance trails. The commands existed. The problems caught up.

Want to run every command in this article against real git history?
→ git-archaeology-lab — clone it, run bash setup.sh, open exercises/.

Which one did you not know about? Drop it in the comments.

Tags: git productivity devtools ai linux