DEV Community: Swift

I got Jev to zero mistakes. I'm still using Flash-Lite.

Swift — Thu, 08 Oct 2026 20:00:18 +0000

Jev is a brilliant decision model. Gemini Flash-Lite is the model nobody talks about, and it’s fast, accurate, and nearly free. Here's what we measured picking between them, and why the pair is the cheapest setup of all.

Jev is the most hyped model right now. TypeSafe's decision model doesn't chat and doesn't write. It reads some text and a list of options and returns calibrated probabilities over those options. A few hundred milliseconds, for almost nothing. Every feed I read has a post about it.

We had a real job for exactly that kind of model, already running in production on something much less fashionable. So instead of reading about Jev, we put it on our own data, ran both models a few thousand times, and kept score.

I love Jev. I'm still using Gemini Flash-Lite. Here's why.

This is the long version of a lightning talk I gave at AI Tinkerers NYC on October 7, 2026. The slides are on Speaker Deck.

The setup

A test that passes when the code is broken is worse than no test. With no test, you know you don't know. With a false green, you ship.

At Major League Hacking (MLH), we grade AI coding agents with benchspec, an open-source eval runner we built for the job. You write what an agent should have done as plain-English checks in a markdown file. benchspec runs the agent in a sandbox and grades every check, with and without the skill under test, so the difference is what the skill actually teaches. After a run, the checks look like this:

- ./.meta/index.md exists
- Exactly six files match ./.meta/templates/*.md
- The summary faithfully reflects the three key facts from the source

Each check gets verified one of two ways. Cheap code: does this path exist, how many files match this glob, does this file have a line matching a pattern. Free, instant, exact. Or an LLM judge reads the content for meaning. That costs money every run.

A classifier, the router, reads each check and picks a checker (file_exists, glob_count, regex, a few more) or defers to the judge.

The router's job: one check in, one checker out, or a deferral.

The router's two mistakes aren't equal:

Send something to the judge that code could have handled: you lose a few cents.
Send something to code that code can't actually verify: a broken result reports green and nobody finds out. The person who trusted that green ships on it. We lose a user's trust. Cents come back. Trust doesn't.

So: when in doubt, defer.

The secret

The router I actually run is Gemini Flash-Lite, the smallest of Google's three Gemini tiers. Pro is the frontier model. Flash is the everyday one. Flash-Lite is the one Google itself describes as built for "high-volume, latency-sensitive tasks like translation and classification".

Flash-Lite has the same million-token context window as its bigger siblings, an answer in well under a second, and $0.30 per million input tokens. The current version, 3.5, shipped in July. It got a line in the release notes and no keynote. Nobody hypes it, but they should be.

99% of my monthly LLM calls go to Flash-Lite. That number comes from my own AI Studio usage page, across every project I have. Pro got nothing. Flash, almost nothing. And the calls are long in, short out: the shape of classification work.

Then Jev showed up. Text in, probabilities over a fixed set of options out, nothing written, by design. $0.042 per million input tokens, output free, a few hundred milliseconds. For a router, that's a perfect pitch.

Round 1: the mistake you can't afford

The experiment is simple. Our router is a single call with one job: read a check, pick a checker or defer. We swapped Flash-Lite out of that slot, put Jev in, and ran the same checks through both. Each model got its own prompt and nothing else changed.

Here are eight real ones. ✓ is safe, ~ is over-cautious (deferred when code could have done it), ✗ is dangerous:

Jev made three dangerous (false positive) calls. Two checks need the content read for meaning, and Jev sent them to regex. A regex can find a source: line. It can't tell you whether that line names the right move. The third: "no .DS_Store anywhere" is about the whole tree, and Jev picked a checker that looks at one path. It passes on a repo full of .DS_Store files.

Flash-Lite: zero false positives. Over-cautious twice, which costs cents (not trust!).

Make No Mistakes

Jev reads literally. TypeSafe's own docs say it answers the question you wrote, not the one you meant. So I rewrote the question. I expected worked examples to do the heavy lifting, because that's what works for chat models. They didn't. What worked was one sentence:

Read quickly, it's the equivalent of writing "make no mistakes" when prompt-engineering. Here's why it isn't that, and why it worked on Jev specifically.

It's a test, not a plea. "Be careful" gives the model nothing to do. "Imagine this checker passed. Could the check still be false?" is a procedure it can run once per option. Take "the log entry has a source: line naming the move". Imagine regex passes. A source: line exists. Could it still name the wrong move? Yes. So regex is out. Take "no .DS_Store anywhere". Imagine not_file_exists passes on ./.DS_Store. Could there still be one in a subfolder? Yes. Out. Every checker gets a mechanical answer, and the mechanical answer is the routing decision.

It asks about the checker's blind spot, not the check's shape. The original prompt asked "can this be verified mechanically?" That's a question about the check, and regex looks mechanical. The rewrite asks "what does this checker fail to see?" That's a question about the tool. Same information, opposite direction. Jev answers exactly what you ask, so it gives a different answer.

The criteria changed to match. The old criteria described what a check should look like: regex is for "a named file has a line matching a stated pattern." The new ones describe what the checker mechanically does and cannot see:

Checker	Before	After
`regex`	The assertion says a named file has a line matching a stated pattern or literal prefix.	Tests whether one named file has a line matching a pattern. It cannot tell what the matched text means or refers to.
`not_file_exists`	The assertion says a single path is gone or never existed.	Tests one exact path for non-existence. It looks at that path and nowhere else.
`skill_invoked`	A bare activation line: skill X was invoked.	Tests whether the named skill was invoked. It sees nothing about what the skill did.

Examples made it worse. I folded our worked examples into Jev's original prompt and its dangerous checks doubled. Examples teach pattern-matching: "a line that says X" looks like the regex examples, so regex it is. The test teaches the failure mode instead.

Checks that must go to the judge but were sent to code. Flash-Lite got to zero on Jev's weak prompt too.

With the rewrite, Jev went to zero on the eight, and on every check we have. Zero dangerous, zero wrong checkers, run after run. Tuned on half, zero on the half it never saw.

The one sentence fixed Jev. It didn't fix Flash-Lite. We wrote a fresh batch of whole-tree checks neither model had seen, like "nothing named draft.md exists in any subfolder".

Both models made false positive calls on them. With the rewrite, Jev went from 35 false positive runs to 0. Flash-Lite barely moved. The difference is how each one failed. Jev got the same checks wrong run after run: a blind spot. Flash-Lite got a check wrong about one run in five and right the other four: a flip. Jev obeys a precise criterion literally, so a better criterion fixes it. Flash-Lite forgives a sloppy criterion but doesn't fully obey a precise one. A blind spot is a prompt bug you fix. A flip is sampling noise you measure for.

Does a smarter model mask a weak prompt?

If the prompt is the biggest variable, does a better model make it stop mattering? Same phrasings, Jev's bare prompt, three models.

Yes. Flash-Lite was still wrong about one run in five. Gemini 3.8 Flash and Opus 5.5 were never wrong. A stronger model masks a weaker prompt.

It doesn't make the prompt stop mattering. The same prompt change that fixed Jev pushed both bigger models toward deferring. It cut Opus's right answers by more than half and 3.8 Flash's to none. The prompt is still the single largest variable; the model decides how expensive your mistakes are. And the mask costs an order of magnitude more in both price and latency.

Beyond our data

Our checks come from one project. So we ran the same experiment on a slice of the Berkeley Function Calling Leaderboard. The task: pick the right function and write its arguments, or say none fits. "None fits" is that benchmark's version of defer. Calling a function that doesn't fit is the dangerous direction.

The first thing we learned had nothing to do with either model. Flash-Lite kept calling a function on items the answer key said had none, and every time I marked it wrong.

After the fourth or fifth failure, I read the item. "Who won the World Series in 2020?" with one candidate function, get_champion(event, year). The function answers the question. The key said it didn't. So I checked every "none fits" item in our sample. Seven items matched. Flash-Lite had flagged all seven by disagreeing with the key. We score without them and list them in the repo. A cheap model with a real job found a bug in a public benchmark.

Jev's false-positive rate is at a 0.8 probability bar: call the function only if Jev is at least 80% sure.

Neither model is at zero. With its top label, Jev called a function on about one in thirteen items where nothing fit. Flash-Lite, about one in twenty-five. But Jev has a knob. Raise the bar to 0.8 and its false positives drop to 1%, at the price of deferring twice as often. Flash-Lite has no knob, but it writes the arguments, and gets them fully right over nine times in ten.

The third card, Jev + Flash-Lite, gets Jev's false-positive rate and Flash-Lite's arguments for well under half the one-call price. That combination is where this ends up, so the rest of the post is about it.

The call you don't need

Take the _Last updated check. Jev returns regex with a probability distribution. And stops. Which file? What pattern? That takes a second model. Flash-Lite, one call, returns the label, the arguments, and a reason, because it can write.

Measured, billed through OpenRouter. Time is mean milliseconds; cost is per 1,000 checks.

Now look at the fourth row: Jev → Flash-Lite, $0.10. Cheaper than Flash-Lite on its own, at $0.52. How does adding a model make it five times cheaper?

How Jev + Flash-Lite saves the money

Nothing clever. Two multipliers, both on Flash-Lite's bill.

1. The second call only runs when there's something to write. A check that goes to the judge has no arguments. Jev defers it and Flash-Lite never sees it. On our full set that's most checks: fewer than four in ten have anything for Flash-Lite to do.

2. The second call uses a shorter prompt. The one-call prompt is long, with worked examples, because that's what keeps Flash-Lite safe on the routing decision. And it goes out on every check. In the pair, Jev has already made the routing decision. The second prompt only has to say: here's the check, here's the checker that was chosen, write its arguments. That's about a seventh of the tokens.

An eval assertion has already been routed to a deterministic checker. Write that checker's arguments.

Assertion: {check}
Checker already chosen: {label}

Arguments needed per checker:
- file_exists / not_file_exists: "path"
- glob_count: "glob" and "count"
- regex: "path" and "pattern"
...
Reply with one JSON object holding only the arguments and nothing else.

Jev itself costs about three cents per thousand checks. Rounding error.

Blue is Flash-Lite input, yellow is Flash-Lite output, coral is Jev.

Put the multipliers together: a call on about half the checks, with a prompt a seventh the size, plus a near-free first hop. Measured, $0.52 becomes $0.10 per 1,000. On the production mix it'd be lower still.

The same thing happens on public data. On BFCL the pair costs less than half of one Flash-Lite call. The second hop runs on about half the items and reads one function's schema instead of every candidate's.

What you pay for it: one more round trip, about a quarter of a second on average. And you inherit Jev's routing, not Flash-Lite's. On BFCL that means Jev's false-positive rate and Jev's knob.

The takeaways

Flash-Lite is Google's best-kept secret. For most classification work it's the right balance of fast and cheap. It's smart enough to forgive a sloppy prompt and to generate. It gave us zero dangerous calls on Jev's bare prompt and wrote the label and the arguments in one call.
Prompts are your single biggest variable. One sentence took Jev from 35 dangerous runs to 0, and examples made it worse. Smarter models soften the problem. They don't remove it: the same prompt change still pushed Opus from mostly answering to mostly deferring.
Private evals separate real from hype. We didn't have to take anyone's word for Jev, including TypeSafe's. We had real checks from a real product. So we ran the hyped model against the one we already use and found out. What's true: fast, cheap, zero once you ask the right question. What isn't: examples help, a smarter model makes prompting irrelevant. What nobody was saying: the cheapest setup is both of them.

To go deeper, start with the practical getting-started guide to Jev. Google's Gemini 3.6 Flash & 3.5 Flash-Lite developer guide covers the model.

Happy Hacking!

[Boost]

Swift — Wed, 15 Jul 2026 01:30:18 +0000

The gap between code logic and human intent

Ben Halpern

Jul 13

The Myth of the Post-Documentation Era

#ai #documentation #opensource #codequality

3 min read

[Boost]

Swift — Tue, 07 Jul 2026 21:20:45 +0000

AI Engineer World's Fair Coverage

Ishaan Sehgal for Daily Context

Jun 30

The Log Is the Agent

#aie #agents #ai

5 min read

Is Forward Deployed Engineering Killing DevRel?

Swift — Thu, 02 Jul 2026 13:52:16 +0000

When Palantir invented Forward Deployed Engineering (FDE) in the 2010s, the industry mocked them as "consultants with equity." Nobody's laughing now.

During the opening keynote, @swyx called out the brand new FDE track at the AI Engineer World’s Fair as one of the things he was most excited about. Cursor just hired a VP of Forward Deployed Engineering. Anthropic teaches 101 classes on it. LinkedIn says it's the fastest-growing job AI has created, with postings up 42x since 2023. In May, OpenAI put $4B behind DeployCo, an entire company built out of FDEs. At AIE, nine companies explained how they embed engineers with customers, and half ended with hiring pitches. When that many companies are hiring for the same role, it's time to pay attention.

AI products fail at integration, not awareness. Everyone at this conference knows Cursor exists. The question is whether it works in your janky monorepo. That's a problem DevRel can't solve. Awareness is already won, and another blog post won't get you past the integration wall. Someone has to get into the codebase and make it work. That's the FDE.

What is a Forward Deployed Engineer (FDE)?

An FDE is the vendor's engineer, embedded in the customer's codebase, shipping production code. They sit with the customer's team, learn the weird edge cases the demo glossed over, and build the actual integration. Their code merges into the customer's repo and stays there.

The customer pays, through services line items or contracts priced to include deployment. DevRel is a cost center under marketing, but FDEs sit on the revenue side.

It's easy to confuse with roles you already know:

Not a sales engineer: FDEs stay post-sale and build real systems.
Not a consultant: they only deploy their own product.
Not support: they have commit access.

Nat Meurer, Head of Agent Engineering at Sierra, made the argument at AIE that AI software is moving to outcome-based pricing, and someone has to guarantee the outcome. That someone has commit access, not a content calendar.

So who makes a great FDE? Engineers with customer empathy, ex-consultants who ship, sales engineers who'd rather build, senior product engineers bored of internal roadmaps. Communication is the differentiator, not raw coding. AI narrowed that gap.

If I were hiring FDEs today, I'd look in two places. First, hackathons: people who build under pressure with strangers are the archetype. Second, your own DevRel team: they've been doing the empathy half of this job all along, and if you're doing DevRel right, they're already highly technical.

So is DevRel dead?

No. But the funnel is splitting. A billion new software creators are coming online, and their agents will increasingly be the ones making buying decisions. Winning those software creators and the agents choosing tools on their behalf is still DevRel's responsibility.

DevRel owns the top of the funnel: awareness, winning the customer's attention. FDE owns the bottom: converting that attention into real usage.

What gets squeezed is the middle. Developer marketing aimed at engineers who read docs and deliberate is the most at risk. That audience is shrinking from both ends. The disposable-software crowd delegates the decision to agents and the enterprise crowd wants an FDE in the room.

There's a credibility inversion here: A conference talk earns applause, but a merged PR in the customer's repo earns a renewal.

DevRel isn't dead. Somebody still has to win the attention of a billion new software creators and the agents buying tools on their behalf. But attention alone never made an AI product work in production. That's the piece we've been missing, and it's the piece FDE fills. The funnel finally has both halves. Go ship in someone else's repo.

This Is Software’s iPhone Moment

Swift — Tue, 30 Jun 2026 14:17:40 +0000

In 2007, humans took roughly 85 billion photos a year. Photography was a specialized craft that required expensive equipment, years of practice, and lots of time. Then the iPhone came out. Today in 2026, humans take roughly 2.1 trillion photos per year, a 25x increase. A single company, Instagram, is now worth 5x more than the entire global photography market combined back then.

The vast majority of photos today aren't taken by photographers anymore either. They're taken by people like you and me on the phones in our pockets. For us, photography isn't our craft — it's a reflex. You probably don't think twice about the 10 photos per day you snap on average. It's an everyday skill we've developed as a result of the radical democratization of the technology. This explosion of photography post-iPhone is one of the most concrete examples we have of "Jevons Paradox". When technology makes things cheaper and more abundant, our overall consumption counterintuitively increases.

Are there still photographers in the world in 2026? Absolutely. In fact, despite many eerily familiar doom-and-gloom predictions, the number of professional photographers worldwide has remained remarkably consistent for decades. We still hire them for life's biggest moments. Conferences, weddings, graduations. We probably always will too. When the game is on the line, you want the expert on your team.

The same way that the iPhone transformed photography, AI is transforming software now. The 40 million software engineers in the world today will quickly be dwarfed by the billion "software creators" coming online tomorrow. We have an opportunity to learn from the past in order to predict the future of our industry.

The Future of Software Is Hyper-Personalized and Disposable

The software of the future will look drastically different from the high-stakes production software of today. Most software will be hyper-personalized and entirely disposable. Rather than worrying about scaling to millions of users, this software will often have a user base of one. It will solve an immediate problem and may very well never run again.

We're still in the earliest days of this transformation. In the same way that it would be hard to imagine social media as the future of the web looking at the web portals of the 1990s, it's impossible to imagine the new interaction paradigms that AI-generated code will enable once the technology matures.
It's tempting to see AI as a cheaper way to build the software we already have. It misses the point. The iPhone wasn't a cheaper camera. It gave us the selfie, the story, the photo sent instead of a text. Cheap, abundant software won't just let us do the same things for less. It'll let us do things that never made sense to build before. Whatever it looks like, it'll have exponentially more software in it than today.

So how do you know what to build? In a world where code is effectively free, the three things that matter are your ideas, taste, and distribution. How do you know what problem to solve, what good looks like, and how to get it into users' hands? When you're writing software for yourself, you implicitly have the answers. You're the world's leading expert on your own problems, you know what a useful solution looks like, and you're the only person you need to convince to use it. For these throwaway tools, scalability and technical debt stop mattering. As long as the agent has the spec, you can throw it away and rebuild it tomorrow.

DevRel Is Dead. Long Live DevRel.

For 30 years, DevRel has been about one thing: winning the hearts and minds of the engineers who make technology decisions. You write the docs, give the talks, build the sample apps, answer the Stack Overflow questions – all so that when an engineer reaches for a database or an API, they reach for yours. It works because engineers have opinions about their stack, and they defend them.
That model breaks in a world of disposable, hyper-personalized software built by nonengineers. The person describing the app they want has no idea whether it's running Postgres or MongoDB, and more importantly, they don't want to know. They'll delegate every one of those decisions to an agent and never think about them again. The developer we spent 30 years courting is being replaced by someone who will never read your docs. DevRel is dead.

Except it's not. It just got bigger. The audience isn't shrinking from 40 million engineers — it's exploding to a billion creators and the agents acting for them. When the volume of software written goes up 1,000x, becoming the default tool is one of the most valuable growth functions in tech. The question is no longer "How do I get this engineer to choose me?" It's "How do I become the default an agent reaches for?"

We don't have the playbook for that yet. Docs traffic, conference booths, workshop signups — all of it was built for humans who read and decide. We need new business models and new ways to think about ROI when your most important customer isn't a person at all, but an agent acting for a million of them. Whoever earns that default will own distribution in the next era of software. Long live DevRel.

The Camera in Everyone's Pocket

We've seen this movie before. The iPhone didn't end photography — it just put a camera in everyone's pocket and changed what photography was for. The professionals are still here, still booked for the moments that matter. But the trillions of photos we take every year aren't theirs. They're ours.

Software is having that same moment right now. The 40 million engineers won't disappear any more than the photographers did. We'll still want the experts when the game is on the line, for the systems that have to scale, stay secure, and not go down. But everything else — the quick fix, the personal tool, the app with a user base of one — is about to belong to everyone.

The people in this room get to decide what that world looks like. We're the ones who'll build the tools, set the defaults, and write the playbook nobody has yet. Photography took 20 years to become a reflex. Software is going to be faster. The only question is whether you're documenting the change or driving it.