DEV Community: Hoelee

When a Self-Hosted Model Gateway Tripped Trust & Safety

Hoelee — Fri, 06 Nov 2026 00:17:31 +0000

Originally published on my blog — read it there with the full context, code and updates.

A single self-hosted OpenAI-compatible gateway served every model tier: long-context reading, refactors, cheap background jobs. One key, one model picker, one dashboard where spend was visible in one place. Six independent workflow seats — repo indexing, CI triage, interactive refactors — sat behind it, each authenticating with its own subscription OAuth credential.

Then all six came back disabled inside one short detection window. None of them had been doing anything unusual. What tripped was a scheduled credential check, and what it found was a block already applied upstream.

The interesting part was not the block. It was the ten minutes I spent rotating credentials that were never wrong.

The error that sent me the wrong way

For the first ten minutes, every consumer of the gateway's error reported a credential failure — orchestrator, shell probe, CLI — because that is what the body said: 401 invalid_api_key. The upstream had actually returned a policy decision.

Three probes separated the layers, and they are worth keeping for any "the model stopped answering" ticket: confirm the gateway answers its own health endpoint; confirm a model from an unrelated provider succeeds on the same key; then loop the subscription models and compare the error codes. One model failing means a broken model. Every model on one provider failing identically, while other providers succeed on the same key, means the provider relationship is closed — and no key rotation moves that.

The real defect is the error contract, not the outage:

# Before: policy decisions disguised as auth failures
if upstream.status == 403:
    return jsonify(error="invalid_api_key"), 401

# After: distinct classes, distinct client behaviour
if "Terms of Service" in upstream.text or "disabled in this account" in upstream.text:
    return jsonify(error="upstream_policy_block",
                   upstream_status=403,
                   alert="credential-policy"), 502
return jsonify(error="invalid_api_key"), 401

A 401 that means "your key is wrong" and a 403 that means "the vendor closed this seat" must never collapse into the same response. Flattening them costs the responder the first ten minutes of every incident.

Why a shared token is not a shared key

An API key is a bearer credential: whoever holds it can use it. An OAuth token is bound to a client identity — client id, consent record, scopes. When a different process presents that token, the request keeps the claims and loses the behaviour: headers, TLS fingerprint, request cadence.

Two properties of the initial gateway topology made the pattern obvious to any detection system:

One egress. Every seat left through one container on a single public address.
One fingerprint. The gateway normalised headers and stripped client telemetry, so six workflow seats looked like one client at six times the volume.

Simultaneous failures across independent workflows are not six human reviews. They are one deterministic cluster rule firing. The vendor's terms had said it in advance: scripting the official CLI is acceptable, pooling seats or automating rotation across them is not.

Task dispatch instead of routing

The lesson is not "use fewer seats". It is that a subscription is a seat, not an API. The moment its token leaves the vendor's client, it becomes a shared credential.

So the gateway keeps serving models paid for by API key, and subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the official CLI bound to a dedicated workflow seat. No pooled credentials, no hijacked persistent session, no rotation. The token never leaves the vendor's binary.

orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
                       |
                       +- cwd, permission scope, timeout, tree-kill, no retries

Four dispatcher properties mattered more than any prompt:

Explicit working directory — each task starts inside one repository, never a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell.
Closed stdin — interactive prompts in a headless run become errors instead of silent hangs. One forgotten -y otherwise blocks the parent forever.
Process-group kill on timeout — killing the parent leaves grandchildren running; kill the group and return a timeout code, not a half-finished diff.
No automatic retries — an inner agent loop times an outer retry policy is hundreds of upstream calls in minutes, the exact cadence that got the previous design flagged.

proc = subprocess.Popen(
    argv, cwd=repo, stdin=subprocess.DEVNULL,      # no interactive hangs
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    start_new_session=True,                         # own process group
)
try:
    out, err = proc.communicate(timeout=task_timeout)
except subprocess.TimeoutExpired:
    os.killpg(proc.pid, signal.SIGKILL)             # kill the whole tree
    raise

Context stopped travelling in prompts as well: the delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged status: draft and final archiving stays with the orchestrator.

The economics survived, because the workload splits

Nine turns in ten are routine orchestration — tool loops, retrieval, cron, log triage, short edits. Those run on a curated open-weight plan: $10/month carrying a $100 monthly allowance on the fast model, with per-model pools beside it. At $0.15 per million input tokens and $0.60 per million output, a 4k-in/500-out turn costs about $0.0009 — under a tenth of a cent.

The remaining tenth is heavy engineering: multi-file refactors, legacy archaeology, anything with a real build-and-test contract. That dispatches to an authorised CLI seat, where the vendor's client owns the credential and the allowance belongs to the subscription rather than to a pooled bucket.

Pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.

💬 Let's Compare Notes

If a credential is bound to a client, where should a platform team draw the execution boundary when metered APIs and client-bound CLI tools have to coexist: standardise on pure APIs and pay the premium, or run a dispatcher that governs official binaries? I have moved to the second, and I would like to hear from anyone who has run the first at team scale and made it survive an audit.

Two related posts from the same infrastructure work:

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Embedding Fonts So a PDF Looks the Same on Every Machine

Hoelee — Thu, 05 Nov 2026 04:25:03 +0000

Originally published on my blog — read it there with the full context, code and updates.

The requirement was one sentence

A report generator has to produce a document that looks the same everywhere:
Chinese headings in a serif, Chinese body text in a sans, Latin headings and
cover numerals in a display serif, Latin body text and tables in Inter.
Weights matter — medium for emphasis, semibold for headings — across three
languages and two PDF renderers.

And the fonts must travel with the document. Not "the server has them
installed", not "the reader probably has them" — embedded, so a print
shop in another country sees the same pages.

The old pipeline failed on the last clause in a way that was invisible: it
depended on fonts installed on the server by a package manager, and it only
ever embedded a single CJK face. The second weight did not exist, so "bold"
was the renderer's synthetic approximation, and the only way to notice was
to open the PDF and look at what was actually inside it.

Rule one: read the fonts back out of the finished PDF

"Embedded" is a claim about the output, so I measured the output. The same
document, rendered before and after:

	before	after
zh page count	23	23
en page count	28	29
bm page count	30	30
fonts actually embedded	one CJK face + Roboto	Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports
render time / peak memory	baseline	same order (±10%, noise)
file size	baseline	roughly +3%

The glyph counts per face are the receipt: in the Chinese report, 15,988
glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading
serif medium, 222 in its semibold. That is the difference between "the font
is on the server" and "the font is in the file" — a reader in another
country sees the same four faces, and a page count that moved by exactly one
page in one language (the Latin metrics are wider than what they replaced)
is the honest cost.

Rule two: static instances only, and the reason is in the parser

The obvious modern choice — ship one variable font per family — is wrong
here, for two measured reasons:

The PDF engine's font parser reads glyf/loca only. Feed it an OpenType file with PostScript/CFF outlines and it refuses: Fonts with postscript outlines are not supported. A variable font is not the problem; CFF outlines are.
It does not read the fvar table at all, so a variable font silently resolves to its default instance — and the default instances of the CJK families are the thinnest ones (Thin 100 and ExtraLight 200, visible in the name table). Every page would render in a hairline weight, with no error anywhere in the pipeline.

So the build step takes the upstream variable fonts and instantiates
static cuts: it runs the instancer, then rewrites name,
OS/2.usWeightClass and head.macStyle so the file identifies itself as
the weight it now is (the source's own name still says "Thin"). A manifest
records family, weight class, sha256, glyph count and code-point count for
every file, and the licence files ship next to them.

Rule three: the engine speaks two weights, so map onto two slots

The renderer's CSS handling honours exactly normal and bold — its source
says so in as many words ("Does not support: bolder, lighter, 100..900").
Numeric weights are therefore meaningless to it, while the headless-Chromium
renderer in the same pipeline honours them properly.

The mapping that works: each family declares an R slot and a B slot
(regular-ish and heavier), font-weight: 500 lands on the R slot of the
intended family, and 600 is rewritten to bold on the PDF side only,
which lands on the B slot. Chromium keeps 600 as written. Two tests pin
it: the B slot must be heavier than the R slot (read from usWeightClass,
not assumed), and font-weight: 600 must never appear in the stylesheet
handed to the PDF engine.

One subtlety worth the paragraph: a "medium" weight is not an emphasis
mechanism in this pipeline, it is a family member. Chinese headings use
the serif's medium as their regular and its semibold for emphasis; Latin
headings do the same with the display serif. Getting that wrong is invisible
on a screen and obvious in print.

Rule four: do not subset against a character set you do not control

The first attempt subset the CJK fonts to the union of two national
character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran
the name probe and found these were cut:

犇  骉  頔  祎  廸  玧  祾  赟  堃

Every one of them is a real given name character. The report prints a
customer's name on the cover, and the name is free input — so a subset-based
build will eventually put a tofu box on the most prominent page of the
product. Full coverage means keeping ~31k code points — roughly 50 MB of static font assets in the repository (adding only ~3% to each finished PDF after engine subsetting); that is a fine price for never showing a customer an empty rectangle.

Rule five: make missing glyphs a build failure, not a surprise

The gate runs three checks and exits non-zero on any of them:

every font file declared in the manifest is present, and its sha256 matches the file on disk;
every character in the content library (all three languages) plus the brand strings, plus probe classes — simplified, traditional, CJK punctuation, full-width forms, mixed-script, digits, Latin extended, symbols — plus the name characters, has a glyph in the face that will actually print it;
the lookup uses the PDF engine's own font parser rather than a third-party library, so the check cannot disagree with the renderer.

The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔)
exist in none of these text fonts and the engine cannot draw colour emoji,
so they are kept out of the report body. The display serif has only 974
glyphs and does not draw symbols like ① ★ ℃, which is why the
content-library check covers all slots rather than just the body. And CJK
characters inside a Latin report must go through the dedicated CJK slot —
two tests keep the renderers from disagreeing about that.

What I would do differently

Choose faces and weights before styling anything. Retrofitting a second weight after the layout exists means re-checking every page count by hand.
Read the renderer's source before choosing a font format. The variable-font decision looked like a taste question and was actually a parser question — two files settled it in ten minutes.
Treat everything you ship inside a document as a licence artifact. Font files enter the repo with their licences, or they enter someone else's legal review.
Measure what got embedded. Page count, glyph-per-face counts, file size, render time: all four moved, and three of them moved in ways I would have guessed wrong.

The result

Before: one CJK face and a UI font, with synthetic bold, depending on what
the server had installed. After: four families and eight static faces
embedded in the file, weights that are real members of their families, page
counts within one page of the previous layout, and a lint gate that fails
the build when a character has no glyph — so the failure mode moved from "a
customer sees a box" to "CI refuses the release". Which is the same trade as
everywhere else in this pipeline: make the mistake impossible to ship rather
than easy to notice.

I build automated document generation and multi-lingual PDF publishing pipelines — from layout engine customisation and compliant font embedding to CI visual verification gates. If your business reports, invoices, or client deliverables need to look identical on every machine, get in touch: WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

How I Vet an Open-Source Dependency Before Betting On It

Hoelee — Wed, 04 Nov 2026 04:26:01 +0000

Originally published on my blog — read it there with the full context, code and updates.

I was handed a 1,306-line architecture spec for a partner program. It read like a
plan: consistent, cross-referenced, confident. It named a self-hosted
open-source engine as the attribution and commission layer, and it assumed three
things about that engine — that it spoke OIDC so my existing identity provider
could log partners in, that its commission rules covered the five shapes the
business needed, and that its white-label feature was shipped.

Ten minutes of curl and grep later, two of those were false and the third was
half-true. None of that is unusual. The README describes the product; the
repository describes the software. The gap between those two is exactly where
integration projects die — and it is cheap to measure before you write any code.

The problem, phrased as the question I'd actually search for

How do I know an open-source project does what its documentation claims before
I build on it?

A spec is a claim about other software. The more polished the spec, the more its
assumptions read like facts — this one cited sources, defined stable IDs, and had
a source-of-truth matrix. What it never did was check whether the engine down the
chain implemented the features being assigned to it. So before installing
anything, I went looking for those specific features.

What I tried first, and why it wasn't enough

1. Read the README end to end. It sells the pipeline well:
click → identity → event → attribution → commission → payout. It never once
says which authentication protocol the app speaks. For a feature you need,
silence isn't neutral — but an absent mention is easy to skim past when the rest
of the page is convincing.

2. Searched the web. Several of my queries returned nothing usable. The
project is a few months old and its SEO surface is thin. For software that's
fine — the source is the documentation. It just means the reading has to happen
in the repo, not in a blog post.

3. Found a 78 KB feature design doc and assumed the feature shipped. It was
labelled "FINAL, post-review", listed three reviewers, and carried a date. I took
that as proof. That was the worst inference of the three, and the easiest to check.

The six checks that actually settled it

1. Read the repo's vital signs before any feature claim

curl -s -H "User-Agent: hermes" https://api.github.com/repos/<owner>/<repo> \
  | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"'

What came back: MIT, created 2026-04-23, last push 2026-09-10, 8 stars, 3 forks,
15 open issues, not archived. Read the same as: real, still moving, and very
small — one vendor, which also sells a hosted tier. Now I know how much of my
business I want resting on it before I start grading its features.

2. Grep for the feature instead of reading for it

grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l

Zero hits. The app authenticates with its own magic-link sessions and
per-persona API keys; it does not speak OIDC at all. That one line of output
killed a phase of the plan — "put the portal behind our SSO provider" — which
would otherwise have surfaced weeks later, mid-integration, with a partner
already invited. Reading for a feature you can't find is how you lose an
afternoon; grepping makes the absence loud.

3. Read the target branch line at the top of a feature doc

head -5 docs/white-label-custom-domains.md

Target branch: multi-tenant. The doc is a design for a branch that is not the
default branch — it describes work in flight, so the feature is not in the code
you would install. A finished-looking document with reviewers and dates can
document something unshipped. That isn't dishonesty; it's what design docs are.

4. Read the data model, not the feature list

The commission section named exactly two rule types — percent (with a recurring
flag) and fixed. The business spec had enumerated five. That gap isn't a bug;
it's a modelling job I'd have to do in configuration instead of expecting from
the product:

Spec assumed	What the engine actually has
first-order bonus	a `fixed` rule on the first event
event-specific rules (`lead` vs `invoice_paid`)	one rule per campaign — so: two campaigns
recurring commission	`percent` + `recurring: true`
percentage	`percent` ✅
fixed	`fixed` ✅

Two rule primitives, five behaviours — workable, but only if you design for it up
front instead of discovering it during implementation.

5. Check which paid tier gates the feature you need

For the database tool in the same stack, the edition table was blunt:

Feature I needed	Community (free)	Enterprise (paid)
API tokens, so I can automate it	✅	✅
SSO / OIDC	❌	✅
Page designer, hide branding, 2FA	❌	✅

The thing I wanted for the partner-facing door (SSO) sits behind the commercial
gate, while the thing I wanted for automation (an API token) is free. Knowing
which side of the wall each requirement falls on changes the design — before
you've designed around it.

6. Check that the money rail works in your country

The payout model was stripe_connect | manual, and self-hosted mode was
documented as "operator manages out-of-band" — i.e. you move the money yourself.
Stripe Connect is not how Malaysian businesses pay partners, so that one line
decided the entire deployment shape: self-host mode, manual payouts, and event
ingestion pushed from my own order system rather than leaning on the provider's
payment integration. That line is never on a marketing page, and it mattered more
than any feature.

Then: ask the fallback question before you need it

"What if the feature I need isn't there?" For the missing SSO the answer was:
don't put login in the app, put it in front of the app. authentik's proxy
provider exists to protect "applications that do not support native
authentication protocols such as OIDC, SAML, or LDAP", and it injects
X-authentik-username and X-authentik-groups headers upstream, so the reverse
proxy can do the gating. That turned "no SSO support" from a disqualification
into an afternoon of config.

What I'd do differently

Write the requirement list first, then go look for it. Five lines — the features I need — and search for those. Starting from the README's feature list means grading the software on the vendor's exam.
Treat "a doc exists" as different from "a feature ships". Read the branch line.
Grep, don't read. Presence is cheap to prove; absence is the thing you're hunting.
Decide the fallback before the decision, so a missing feature becomes a design change rather than a surprise.
Pin the commit. This project's README says its API is "stable but unversioned" and there is no OpenAPI description anywhere in the tree. There's no contract to compile against, so the commit hash is the contract — the endpoints I use go into my own docs, and upgrades get a regression pass.

The result

Six checks and about ten minutes of curl and grep corrected all three faulty assumptions in the spec before a single line of integration code existed. The project stayed viable — but
reshaped: no OIDC, manual payouts, a pinned commit, and a plan that no longer
depends on a feature sitting on an unmerged branch.

The spec wasn't wrong to be optimistic. It was wrong to be unverified, and that's
the cheapest thing on the whole project to fix.

Building on a self-hosted stack, and want a second pair of eyes on whether
those features will actually be there when you integrate? That's the kind of
review I do.

WhatsApp +60 12-797 2969 ·
me@hoelee.com ·
hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You

Hoelee — Tue, 03 Nov 2026 04:25:38 +0000

Originally published on my blog — read it there with the full context, code and updates.

If you self-host NocoDB and search for "nocodb sso self-hosted", you'll find a documentation page describing OIDC single sign-on, a set of environment variables, and forum answers telling you it works if you set them. All of that is technically true. None of it tells you the part that matters.

So here it is: NocoDB's OIDC SSO is a paid feature. On an unlicensed self-hosted build, the code path exists but abandons the process — and the failure isn't a polite "SSO is not enabled on this instance". The whole server exits.

Why this matters

Two audiences get bitten here.

If you enable it in production, you've built a denial-of-service trigger. The route that handles the OIDC callback is reachable without authentication — it has to be, that's how a login flow starts. On an instance where that route crashes the process, anyone who requests it takes your database UI down. Not degrades it, not returns an error page: kills the container. You don't need to be a target; a single curious visitor, or a link scanner that follows URLs, is enough.

If you're evaluating NocoDB as a platform, this is a licensing signal you should read correctly. The feature is present in the open-source image, the variables are enforced, the code ships. What's missing is permission. A feature that is implemented but not licensed looks identical to a feature that works, right up until you try it — and that's the whole trap.

What the docs say, and what the image does

NocoDB's own docs are clear if you read the licensing line: "OIDC SSO is available on NocoDB Cloud (Business plan and above) and licensed self-hosted deployments (Business plan and above)." The other half — that the unlicensed binary will not degrade gracefully — isn't documented, and can't be discovered from the docs, because it's a runtime fact about the image.

The environment variables are real. The ones I confirmed against the shipped image:

NC_SSO: oidc
NC_SSO_OIDC_ISSUER: https://auth.example.com/application/o/your-app/
NC_SSO_OIDC_AUTHORIZATION_URL: https://auth.example.com/application/o/authorize/
NC_SSO_OIDC_TOKEN_URL: https://auth.example.com/application/o/token/
NC_SSO_OIDC_USERINFO_URL: https://auth.example.com/application/o/userinfo/
NC_SSO_OIDC_CLIENT_ID: ...
NC_SSO_OIDC_CLIENT_SECRET: ...
NC_OIDC_PROVIDER_NAME: Hoelee SSO

Only NC_SSO and the provider-name variable appear in the public documentation. The six per-provider variables are undocumented but enforced: start the container with NC_SSO=oidc and no URLs and it refuses to boot:

Open ID SSO is enabled but missing required env keys

That's the detail that makes the trap convincing. Validation at startup means the feature is wired in, not vestigial. Nothing about that error message suggests the code path is unreachable without a licence — it reads like a configuration mistake you can fix.

How to test it without risking anything

Never test this on the instance you care about. A throwaway container with an in-image SQLite meta store is enough, and it takes about a minute:

docker run -d --name nc-sso-test -p 10399:8080 \
  -e 'NC_SSO=oidc' \
  -e 'NC_SSO_OIDC_ISSUER=https://auth.example.com/application/o/your-app/' \
  -e 'NC_SSO_OIDC_AUTHORIZATION_URL=https://auth.example.com/application/o/authorize/' \
  -e 'NC_SSO_OIDC_TOKEN_URL=https://auth.example.com/application/o/token/' \
  -e 'NC_SSO_OIDC_USERINFO_URL=https://auth.example.com/application/o/userinfo/' \
  -e 'NC_SSO_OIDC_CLIENT_ID=test' \
  -e 'NC_SSO_OIDC_CLIENT_SECRET=test' \
  -e 'NC_OIDC_PROVIDER_NAME=Test SSO' \
  nocodb/nocodb:<your-tag>

Note the deliberate omission of NC_DB — with no external database configured, the container spins up its own SQLite store, so it needs no credentials, touches no production data, and can be deleted with docker rm -f nc-sso-test.

Watch the startup, then request the SSO route once:

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:10399/auth/oidc
docker ps -a --filter name=nc-sso-test

What I got, twice — the second time with a ?workspaceId= parameter to rule out a bad request shape:

### UNCAUGHT EXCEPTION ###
unhandledRejection TypeError: _0x418c23 is not a function at docker/index.js:1:27483406

...and the container is no longer running. Exit code 1, from an unauthenticated GET. Reproduced on 2026.09.0 with the same result both times. It isn't a configuration error, and it isn't a request the client got wrong.

Note the log line that explains it, from startup:

No license key found — running in CE mode

The other half of the licensing story

If your reaction is "fine, I'll buy the licence", read this before you budget for it, because there's a prerequisite that isn't obvious either:

Instance ID unavailable — PostgreSQL is required for enterprise licensing

The community edition I run uses MySQL as its metadata store, and MySQL cannot generate the instance ID that licensing depends on. So a licence isn't something you activate on the instance you have — it's a licence plus a metadata migration from MySQL to Postgres, which is its own project with its own risk, done on a live instance. NocoDB publishes a migration guide for exactly this, which tells you how common the situation is.

That isn't a complaint about the pricing model. It's the evaluation I wish I'd had before spending an afternoon: the cost of "just buy it" is licence plus a storage-engine migration for anyone running the default MySQL setup.

The fork that search results recommend — check it's alive first

Search results for NocoDB SSO without a licence point at brunostjohn/nocodb-oidc, a community fork that adds OIDC to the open-source build. It's a drop-in image and it's exactly the thing you want. It's also abandoned, and this is checkable in about two minutes:

# what version of NocoDB is the fork based on?
curl -s https://raw.githubusercontent.com/brunostjohn/nocodb-oidc/main/packages/nocodb/package.json \
  | python -c "import sys,json; print(json.load(sys.stdin)['version'])"
# 0.255.2

# when was it last touched?
curl -s 'https://api.github.com/repos/brunostjohn/nocodb-oidc/commits?per_page=1' \
  | python -c "import sys,json; print(json.load(sys.stdin)[0]['commit']['author']['date'])"
# 2024-10-29

A version from 2024 against an instance on a monthly release cadence is not a drop-in — it's a two-year downgrade, taking with it the schema migrations, the security fixes and the API behaviour your automations already depend on. The advice is only as good as the fork is maintained, and search results don't go stale when a repo does.

Check the fork's base version and last commit date before you plan around it. Two curl calls, and they replace a rollback you'd otherwise do at 1 a.m.

What to do instead

Decide whether you need SSO in the app, or in front of it. If what you want is "the internet can't reach this login form", a forward-auth proxy in front of the hostname gets you there with the open-source build. It's not single sign-on in the app, and the app still asks for its own password — I wrote up that construction, and the ingress trap that made my first attempt look successful while being a no-op, in The Forward-Auth Gate That Verified Perfectly — and Wasn't Live.
Keep the app's own login and lock the door at the edge. For an internal tool with a handful of users, this is what most people actually need.
If a licensed build is genuinely required, price the migration, not just the licence. Confirm which metadata store you're on first.

What I'd do differently

When a feature's availability is ambiguous, read it out of the image instead of the docs. The docs describe intent; the image is the truth. One docker exec ... printenv | grep -iE 'NC_|OIDC', one grep of the shipped bundle, and one throwaway container answered in twenty minutes what an afternoon of documentation reading couldn't.

And a blunter rule that generalises past NocoDB: an undocumented-but-enforced environment variable is a licensing boundary, not a feature flag. If a variable rejects your startup when it's incomplete, but the documentation hides half its siblings, the missing piece is almost never configuration.

The result

Five minutes to reproduce on a throwaway container, twice, with the container dead both times. One accidental finding I'd rather not have learned in production: an unauthenticated request to the SSO route is enough to stop the instance.

The upside is that I stopped trying to make the app do SSO, put the decision in front of the hostname where it belongs, and can now answer "can NocoDB do SSO for free?" in one sentence instead of an afternoon.

Want this for your business?

If you're evaluating self-hosted software, need single sign-on in front of internal tools, or want someone to check what your stack actually does versus what its documentation claims, that's the work I do.

WhatsApp: +60 12-797 2969 — tap to chat
Email: me@hoelee.com
Website: hoelee.com

I set up authentik single sign-on, self-hosted Docker stacks and reverse proxies for small businesses in Malaysia — and I write up what I learn while doing it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

The Forward-Auth Gate That Verified Perfectly — and Wasn't Live

Hoelee — Mon, 02 Nov 2026 04:25:39 +0000

Originally published on my blog — read it there with the full context, code and updates.

I finished the gate, ran the check, and it passed.

HTTP/1.1 302 Found
location: https://app.example.com/outpost.goauthentik.io/start?rd=...
x-powered-by: authentik

An unauthenticated request was being pushed to my authentik login page. Branded, correct, working. I was about to move on — except that my own rule for anything user-facing is to hit it the way a user hits it, from outside, over the public internet. So I did.

GET https://app.example.com/          → 200
x-powered-by: Express

The app answered directly. No login page, no redirect, no gate. The exact same hostname, one request path gated and one wide open, depending on where the request entered my network.

That was the whole day: not the SSO setup, which worked, but the fact that a hostname can be served by more than one ingress layer, and I had only gated one of them.

Why this matters beyond my one app

If you self-host anything behind a tunnel, a reverse proxy, or both, you have probably already made this assumption: "the reverse proxy is what serves my domains." It's an assumption that holds right up until it doesn't, and it fails in the most dangerous direction — everything you change looks correct, because the path you're inspecting is correct.

The failure mode is silent by construction. There is no error, no log line, no hint. Your configuration check passes because you're checking the configuration you edited. Meanwhile the real traffic — the traffic you were protecting — never touches it.

I've since found two other gates on the same host with the same latent problem, built the same way, waiting for someone to trust them.

The setup: SSO for an app that has no SSO

The app was a self-hosted database UI on my NAS (Docker, published on a host port, sitting behind nginx and a Cloudflare Tunnel — the usual shape of the infrastructure I manage). I wanted a login wall in front of it so that "reachable from the internet" wouldn't mean "anyone who guesses the hostname sees the login form and can start guessing passwords".

The app's own documentation advertises SSO. I'll keep this post focused on the ingress trap and put the licensing story in its own write-up, but the short version is that the advertised SSO is a licensed feature, the open-source build ships the code path but not the permission to use it, and the community fork that used to fill the gap hasn't been touched in two years. Twenty minutes of research, two abandoned approaches, one conclusion: the app would never do SSO for me, so the gate had to live in front of it.

Which is a normal, well-trodden thing to do. authentik calls it a proxy provider, and the flow is:

A login flow for this app — its own title, its own background, so a user who lands on the login page knows which system they're being asked to authenticate for.
A proxy provider — the object that knows the app's external hostname (https://app.example.com) and its internal address (http://app-container:8080).
An application linking the two, attached to the embedded outpost. The outpost is the component that actually receives traffic, checks for a session, and proxies the request to the app's internal host after login.
The ingress — pointing that hostname at the outpost instead of at the app.

Steps 1–3 live in authentik and are configuration. Step 4 is the one that decides whether any of it is real, and it's the one I got wrong.

Two smaller things from steps 1–3, since they cost me time:

A partial PATCH on a proxy provider is rejected. Updating one field returns:

{"internal_host": ["Internal host cannot be empty when forward auth is disabled."]}

The serializer re-validates the object, and a proxy provider without an internal host is invalid — so a one-field update fails as if you'd cleared a field you never touched. Send the identity fields together with the field you're changing:

curl -s -X PATCH "$AUTH/api/v3/providers/proxy/$PK/" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"mode":"proxy","external_host":"https://app.example.com",
       "internal_host":"http://app-container:8080","skip_path_regex":""}'

authentik API tokens can expire mid-session. After working for an hour, every call started returning 403 {"detail":"Token invalid/expired"}. Note the status code: 403 means "we know who you are, and you may not do this" — the response body is the only thing that tells you the token died. If you cache a token in a script, re-read it from the database when calls start failing instead of assuming a permissions problem.

The trap: I verified the layer I changed

Here is the check I ran, from the host itself:

curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}' \
  -H 'Host: app.example.com' https://localhost/
# 302 -> https://app.example.com/outpost.goauthentik.io/start?rd=...

This is a good check. It proves the nginx vhost serves the outpost, that the outpost recognises the hostname, that the provider is linked to the application, and that the login flow is alive. Everything I had configured was proven correct in one command.

It says nothing whatsoever about whether public traffic reaches that vhost.

Here's why. My hostnames are served by two independent ingress paths:

Path	What serves it	Where it lands
Direct WAN IP → nginx (443)	the vhost I edited	the outpost ✅
Cloudflare Tunnel (`cloudflared`)	a rule inside the tunnel's own config	the app's port, directly ❌

The tunnel's ingress is a small ordered list evaluated per hostname:

app.example.com     -> http://app-container:8080     ← straight to the app
*                   -> http_status:404

That rule was written years earlier, when the goal was simply "make this reachable". It never mentioned nginx, because it never needed to. So my beautiful vhost sat off to the side of the real request path, gating a route that public traffic doesn't take.

The two paths even answer on the same hostname, so nothing about the URL hints at the difference.

What finally made it obvious was a response header. The gated check returned x-powered-by: authentik. The public check returned x-powered-by: Express. Same URL, same second, two different servers answering. When you suspect you're looking at the wrong layer, read the headers — they name the software that answered, which is exactly the question you're asking.

To see your own tunnel's rules rather than guessing (Cloudflare's API, not the dashboard):

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/cfd_tunnel/$TUNNEL_ID/configurations" \
  -H "Authorization: Bearer $CF_TOKEN" \
  | python -c "import sys,json; [print(r.get('hostname','*'), '->', r.get('service','')) for r in json.load(sys.stdin)['result']['config']['ingress']]"

git.example.com   -> http://10.0.0.10:6880
app.example.com   -> http://app-container:8080
bot.example.com   -> http://bot-container:8080
*                 -> http_status:404

There it is, in one line, after a day of looking at nginx.

The permission wall on the way to the fix

Editing that rule needs a token with Account → Cloudflare Tunnel → Edit. Mine didn't have it, and no zone-scoped token can substitute: tunnels are account-scoped objects, so a Zone-scoped token fails the read as well, with 1001 Not authorized. If you only have zone-level tokens, open the dashboard instead: Zero Trust → Networks → Tunnels → your tunnel → Public Hostnames and change the rule's service to your outpost.

For me the correct target was the outpost on the same Docker network as cloudflared:

app.example.com  ->  http://authentik-server:9000

The outpost reads the Host header (which the tunnel preserves) to pick the application, then proxies to the internal host from the provider config. That's the whole change: one service value. Everything I'd built in authentik was already correct — it was simply never on the path.

Optional SSO: gate the UI, leave the API alone

Once the gate is really in front of the traffic, the next problem is the API. A hostname-level gate can't tell a script from a browser, so token-only API calls get redirected to a login page — they don't fail with a JSON error, they get HTML.

If you want SSO for humans and no SSO for machines, authentik's proxy provider has a per-path exemption. The UI calls it Unauthenticated Paths; the API field is skip_path_regex:

{"skip_path_regex": "^/api/.*"}

Verified result on my instance, same hostname, same session:

Request	No session	Result
`/` and `/dashboard`	yes	`302` → authentik login
`/api/v1/health`	yes	`200` from the app
`/api/v2/tables/.../records` + API token	yes	`200` with real data

Two caveats worth knowing before you rely on it. This exemption applies at the outpost, so unlike a path split at the CDN it works on every ingress path. And it is not per-user: the gate applies to the request, not the person. There is no "some users can skip SSO" — either a path is exempt for everyone, or everyone needs a session.

I used a simpler form of the same idea first: move every machine caller off the public hostname before the gate exists. My automation was calling the app through the public URL and being routed out to Cloudflare and back into my own network for no reason. Pointing it at the container name (http://app-container:8080) shortened the path, removed a failure mode, and meant the gate couldn't break it. If a login wall takes down your integration, the integration was depending on the door being unlocked.

The honest part: this is not single sign-on

I have to be straight about what a forward-auth gate gives you, because "SSO" oversells it.

The outpost knows who you are. The app does not. Unless the app can consume an upstream identity — header-based auth, or native OIDC/SAML — it will show its own login form after the gate lets you through. So the user experience is: authentik login, then the app's login.

For a user who is already signed in to authentik, the first step is often silent. But it's still two systems, two session lifetimes, and two places a password can be wrong. In my case the app has no header auth and its own OIDC is a licensed feature, so the gate would have been an extra door bolted in front of the existing one — no reduction in logins at all.

That's a legitimate trade for some setups: keep the app's login for its own users, and stop the internet from reaching the login form. It's a poor trade if what you actually wanted was "my team types one password". Which meant the honest answer for me was to turn the whole thing back off.

Reverting it properly (and the arithmetic that saved ten other apps)

Undoing the gate is where I nearly caused a much bigger outage than the one I was preventing.

The nginx change was one line in a large generated vhost file. The obvious revert is to substitute the port back:

# DON'T — this is the version that would have hurt
sed -i 's#proxy_pass http://localhost:10000;#proxy_pass http://localhost:10380;#' vhost.conf

Before running it, I counted how many server blocks point at that same port:

grep -c 'proxy_pass http://localhost:10000;' vhost.conf
# 11

Eleven. My app was one of them. Ten other applications are gated through that outpost, and a blanket substitution would have silently un-gated every one of them at once — including the ones doing real authentication work. The "fix" would have looked successful and removed authentication from most of my infrastructure.

What I did instead:

Snapshot the current state first, so the revert was itself reversible: cp -a vhost.conf vhost.conf.bak-$(date +%Y%m%d-%H%M)-revert.
Find the exact line by diffing against the backup taken before the original change. That diff showed one changed line, which is proof of exactly how much I had touched.
Edit that line only, then re-diff against the original backup and require the result to be identical before going further.
Update the second file that mirrors the same setting — the NAS's own reverse-proxy database, which regenerates the vhost on DSM events. I edited the single key and asserted the hostname before writing, never restoring the whole file.
nginx -t then reload.

And then a detail that cost me another twenty minutes of doubt:

nginx -s reload && sleep 2 && curl -sk -H 'Host: app.example.com' https://localhost/
# still 302 -> /outpost.goauthentik.io/start

The revert looked like it had failed. It hadn't. A request issued microseconds after the reload can still be answered by the old worker while it drains. The re-test a few seconds later returned 200 x-powered-by: Express — direct to the app, gate gone.

If a config change appears not to apply, re-test after a beat before you go looking for a second cause. With a request that fast, "the config is wrong" and "the old process is still answering" are indistinguishable.

What I'd do differently

Enumerate every ingress path for a hostname before building anything on top of it. My checklist is now four questions, asked in this order:
- Is the hostname proxied by a CDN/tunnel? If yes, its rules are the real ones — check them first, not last.
- What does the reverse proxy vhost say?
- Does the platform mirror that vhost somewhere else that can regenerate it?
- Is there another hop (VPS, second proxy, vendor tunnel) with its own copy of the routing table?
Verify from outside, and verify which software answered. curl from the host with a Host: header proves your config is self-consistent. Only a request from the public internet, plus x-powered-by/server headers, proves it's in force.
Find out whether the app can consume an upstream identity before you build the gate. If it can't, you're adding a login, not removing one — a decision to make deliberately, not to discover afterwards.
Count before you substitute. Any port or hostname you're replacing globally probably appears in more places than you think. grep -c costs a second and prevents an outage.
Snapshot, diff, edit one line, re-diff. The backup's only job is to answer "how much did I change?" — and a diff answers it exactly.

The result

The gate worked. It was also invisible to every real user, and after working out what it actually delivered — a second login in front of an app that already had one — I turned it off again: one field cleared, one application unlinked, one line restored, verified by a request from outside my network.

The two checks that would have caught the whole thing on day one take about thirty seconds:

curl -sI https://app.example.com/ | grep -i '^x-powered-by'   # who is actually answering?
curl -s  https://app.example.com/ -o /dev/null -w '%{http_code}\n'  # what does a stranger get?

If the answer isn't what your configuration says it should be, you're editing the wrong layer. That's worth knowing before you spend the day on it — and definitely before you tell anyone the gate is live.

Want this for your business?

If you need single sign-on in front of internal tools, a self-hosted app exposed safely to the internet, or someone to audit which ingress path is actually serving your domains, that's the work I do.

WhatsApp: +60 12-797 2969 — tap to chat
Email: me@hoelee.com
Website: hoelee.com

I set up authentik single sign-on, hardened reverse proxies and Cloudflare tunnels, and self-hosted Docker stacks for small businesses in Malaysia — and I write up what I learn while doing it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Your Disk-Full Alert Is Lying: Stop Alerting on Percentages

Hoelee — Sun, 01 Nov 2026 04:25:47 +0000

Originally published on my blog — read it there with the full context, code and updates.

I built a monitoring stack, wrote a reasonable set of rules, and then spent an
afternoon deleting most of my own thresholds. Every one of them was a percentage, and
every one of them was wrong in the same specific way.

The three alerts that taught me

1. "Filesystem above 90% used." It fired on a 17 TB volume that had ~500 GB free.
Nothing was wrong. Nothing was about to be wrong. On a volume that size, a single
large backup can swing the percentage by several points — the ratio is noise, and the
fact that 500 GB is still available is the actual state of the world.

2. "Memory above 90% used." Fired constantly on a host with 4.8 GB available.
Linux uses free RAM for page cache and gives it back on demand; MemTotal - MemFree
describes your filesystem cache, not your risk. The number that predicts an OOM is
MemAvailable. (Even if one alerts on MemAvailable exceeding 90% capacity, on a 64 GB host it still fires an alert when 4.8 GB of usable headroom remains.)

3. "CPU steal above 25%." Fired on a VPS that had been running its mail stack
perfectly at 41% steal. Steal means the hypervisor is busy — it is a capacity signal,
not a failure signal, and 41% steal on a 7-vCPU box was simply what that host costs.
The rule produced alerts that were always true and never actionable.

The common thread: I was alerting on a ratio, and the ratio doesn't know how big the
thing is.

Alert on the consequence, not the ratio

The test I now apply to every threshold: if this condition persists, what will actually
happen? That answer is almost always expressible in absolute units.

Instead of	Alert on	Because
Filesystem > 90%	Free space < N GB	"Can I still write?" is the question
Memory > 90% used	`MemAvailable` < 512 MB	about to OOM, not "cache is warm"
CPU steal > 25%	Steal > 50%	capacity cost vs. starvation
Load average > N	`load1 / cores > 2`	load is meaningless without core count

In PromQL, before and after:

# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90

# AFTER — capacity left, on the volumes that hold data
min by (instance, mountpoint) (
  node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"}
) < 25e9

# BEFORE — page cache counted as "pressure" (misusing MemFree)
(1 - node_memory_MemFree_bytes / node_memory_MemTotal_bytes) * 100 > 90

# AFTER — the actual OOM precursor based on kernel-reclaimable MemAvailable
node_memory_MemAvailable_bytes < 512 * 1024 * 1024

Two details that make these rules survive contact with a real dashboard:

Keep the comparison in the threshold condition, not inside the expression. Query
min by (mountpoint) (node_filesystem_avail_bytes{...}), then let the alert rule
compare it to 25000000000. If you bake < 25e9 into the PromQL and leave an
inherited > 90 in the rule, the rule still evaluates — but it displays a nonsense
threshold, and the next person to read it has to reconstruct your intent from two
contradictory conditions.

Exclude the mounts that are views of other mounts. A "free space" rule is only as
honest as its selector. On my own setup, one NAS volume appears three times (as
/volume1, as /opt, and again on another host as a CIFS re-mount), and unRaid's shfs
union mount reports avail=0 — a rule without exclusions would either fire forever or
double-count capacity. One explicit selector beats a clever generic one:

node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"}

Noise is not harmless

The cost of a bad threshold isn't the alert itself — it's the training. Every alert
that fires while nothing is wrong teaches you to skim, then to ignore, and then the one
that matters arrives in a channel you've stopped reading. My rule count went down
while coverage went up: fewer, each tied to a named consequence.

Thresholds are personal, shapes are not

The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB
available before I care about memory, 50% steal before a VPS is being starved. Yours
will differ with your hardware and your tolerance.

What transfers is the shape:

absolute units over ratios,
the consequence written into the alert's summary,
a selector that states exactly which mounts it speaks for,
and a rule count small enough that you still read every message.

The result

The same stack that was producing three permanent, uninformative warnings now runs 16
rules that stay silent when the infrastructure is healthy — and one of them immediately surfaced a
NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been
hiding behind a 99%-full giant that I'd learned to ignore.

Want this for your business?

If your monitoring sends you alerts you've learned to ignore, or you have none and would
rather find out about a full disk from a rule than from a failed backup, I set up
self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email)
with thresholds tied to what actually breaks.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)

Hoelee — Sat, 31 Oct 2026 04:25:13 +0000

Originally published on my blog — read it there with the full context, code and updates.

You open the dashboard you use every day and it's empty. Not "one panel broke" empty —
every panel, top to bottom, says No data. You check Prometheus and everything is
healthy. This post is the specific, non-obvious cause I hit, and the way my own
verification convinced me the dashboard was fine while it wasn't.

The searchable question

My Grafana panels show No data but the data is in Prometheus — why?

The usual answers (wrong datasource, wrong time range, rate() on a counter that
resets, missing scrape target) did not apply. Here is the checklist that narrowed it
down, in the order that costs the least time:

# 1. is the target actually being scraped?
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}'

# 2. does the metric exist right now?  (bypasses Grafana entirely)
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric'

# 3. is the panel's expression sound?
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})'

All three were green: 7 targets up, 25k series ingested, and the panel's raw
expression returned data when I ran it by hand. So the problem was not the metrics,
the scrape, or the PromQL — it was the variable layer between them.

The actual cause: a variable that filtered on itself

The dashboard had a host picker. Two variables chained: $nodename (which machine)
and $node (its instance label). The definition of the first one was:

# BROKEN — the variable filters on its own value
label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename)

On first load $nodename has no value, so the selector becomes
nodename=~"" — which matches nothing. A variable whose query returns no rows has
no options, so it stays empty. $node is then defined against $nodename:

label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance)

…which is also empty, so every panel filtering on $node matches no series. Grafana
isn't lying when it says No data: the panel query genuinely has no data, because the
variable that scopes it resolved to nothing.

The fix is to stop the self-reference and give each variable an explicit saved default:

# FIXED — no self-reference, one hop per variable
# $nodename
label_values(node_uname_info{job="unraid-host"}, nodename)
# $node
label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance)

Then save current values for both so the dashboard opens populated instead of
depending on a click. (This is also why the same 41-panel layout can be cloned to
three hosts and just work.)

Why my verification missed it

This is the part worth stealing. I "verified" the dashboard by expanding each panel's
expression by hand, substituting the variable values I knew were correct
($nodename=unRaid, $node=host.docker.internal:9100), and asserting the query
returned points. Thirteen of those checks passed. The dashboard was still blank,
because the bug was upstream of the substitution: Grafana's own resolution of the
variables was empty.

A verification that substitutes the very value under suspicion cannot detect that
value failing to resolve. To catch it, read the values Grafana actually saved and
expand with those:

curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \
  | jq '.dashboard.templating.list[] | {name, current: .current.value}'

The correct check is on the variable query itself — run what Grafana runs:

# what does $nodename's label_values() actually return?
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result'

Empty there means the dashboard cannot possibly render, no matter how healthy the data is.

The second trap: `$__all` is not `.*`

While automating panel checks, a variable set to All reports its current value as the
sentinel string $__all — not a regex. Expanding panel expressions naively turns
name=~"$__all" into a literal that matches nothing, so a perfectly healthy
container dashboard "fails" every panel. Resolve $__all to the variable's allValue
(usually .*) before expanding, or you'll chase a bug that only exists in your checker.

Not every empty panel is a bug

After the fix, out of the 41 total layout panels, 21 of the 25 core metric panels returned data. The remaining four were empty by design and
worth naming honestly in the panel description:

PSI panels (Pressure, Pressure Stall Information) need /proc/pressure, which the on-premise NAS storage node's kernel doesn't expose. The metric cannot exist there — the same panels on a host that does export node_pressure_* render fine.
Root filesystem panels whose expression carries fstype!="rootfs" can never match on a host whose / is a ramdisk, which is what unRaid's root is.

Distinguishing "broken because of a bug" from "empty because the metric cannot exist on
this host" is the difference between a real fix and a wild goose chase.

What I'd do differently

Never define a template variable that filters on itself. One hop per variable.
Always save explicit current values for variables a dashboard depends on, so a fresh load doesn't rely on a populated dropdown.
Verify with Grafana's saved values, not my idea of them. The masking is invisible in the test result and obvious in the UI.
Describe intentional gaps in the panel description. Future-you will otherwise spend an afternoon "fixing" a panel that was never meant to work there.

The result

All three hosts now run the same instrumented layout — a 41-panel host dashboard (with 21/23/25 of the 25 core metric panels returning data respectively) and a
10-panel container dashboard each — and the two structurally-empty classes documented in place instead of silently
confusing whoever opens them next.

Want this for your business?

If you have Grafana dashboards that nobody trusts — or servers and a NAS with no
monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container
metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that
have quietly stopped showing data.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Stop Writing Agent Prompts for Deterministic Work

Hoelee — Fri, 30 Oct 2026 04:25:11 +0000

Originally published on my blog — read it there with the full context, code and updates.

One of my scheduled jobs failed 29 times in a row. Nobody told me, because
it was designed to speak only when it had something to report — and a run
that dies before it can speak looks exactly like a run that found nothing
wrong.

The job was a version check: ask npm for the latest published version of a
CLI I use, compare it with the one installed here, and tell me if there is a
newer one. That is the entire task. It is two strings and an inequality.

I had built it as an agent job, because that was the pattern I had in my
head at the time. Every tick, the model would read the injected script
output and decide whether to write a notification. It worked for a while.
Then my global default model changed, and the job — which records the model
it was created with — stopped running entirely. It printed [drift_skip]
and exited. Twenty-nine times.

The fix was not a better prompt. It was deleting the model from the loop.

The rule

If the stop condition is a comparison — a string, a number, a status code — a model in that loop can only add failure modes.

There is no judgment to make in "is 3.8.50 newer than 3.8.50". The model
was not deciding anything the code could not; it was adding a dependency (a
model route, a provider, a config snapshot, a token bill) and a new way to
fail. So I rewrote the checker as 125 lines of Python with an explicit exit
path for every branch, and flipped the job to script mode: the script's
stdout is delivered as the notification, with no LLM anywhere in the path.

It has run clean ever since. Anthropic's own guidance says the same thing in
one line — use scripts for deterministic work; running a script is cheaper
than reasoning through the steps — but the cost was never the interesting
part for me. The reliability was.

Where the model does earn its place

Of the eleven loops I run, eight now have no model in the path. The three
that do all handle something I genuinely cannot express as code:

Is this email important? A mailbox produces promos, DMARC reports, container alerts and a customer asking for a quote. Fetching mail is deterministic. Ranking it is not.
Is this shop voucher about to overspend? The arithmetic is trivial; deciding what a finding means for money I have already committed is not.
Is this project stalled? Reading two markdown files is a script's job. Noticing that the plan has not moved in three weeks is not.

The test I apply to a new loop: write down what the check would print on
success and on failure. If both answers are a number or a string, it is a
script. If the answer is "it depends what it says", it needs a model — and
then it needs a verifier I trust more than the agent, because I am no longer
able to predict the output.

The part that actually bit me

The failure mode was not the drift. It was the silence.

A loop that speaks only when something is wrong is the only kind of loop I
can tolerate — I do not want a daily status report from eleven jobs. But
"silent because healthy" and "silent because it crashed before it could
speak" are the same observation from my side of the phone. Twenty-nine runs
of evidence were sitting in the job's execution log the whole time; I was
trusting the absence of a notification instead of reading the history.

So every script in the fleet now has a second exit path: exit 0 means
measured, here is what I found, and exit 1 means I could not measure —
and exit 1 is never silent. Missing config, unparseable state, an unexpected
exception: all of them print a line and exit 1. That one change would have
surfaced this bug on the first run instead of the twenty-ninth.

If you have a loop that has been "quiet for a while", go and read its last
ten runs before you conclude it is working. That advice cost me three weeks.

The short version

Deterministic check → script. Judgment → model. Do not mix them for convenience.
Give the silent-on-success convention one exception: "I could not measure" must always be loud.
Read the runner's own execution history, not just your notifications.

Want a version of this for your own site or shop — uptime, certificates,
backups, mail that only pings you when it matters?
WhatsApp +60 12-797 2969 ·
me@hoelee.com ·
hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Running Production Infrastructure Solo: 162 Containers Across Three Hosts

Hoelee — Thu, 29 Oct 2026 04:25:39 +0000

Originally published on my blog — read it there with the full context, code and updates.

Why this matters

A business that sells software needs two things at once: an application that
does the job, and the machinery underneath that keeps it reachable, backed
up and secure. Agencies usually split those two between a developer and a
hosting company, which is how you end up with a developer who cannot see the
server and a hosting company that cannot read the code.

I run both. That means the person who wrote the application is also the
person who gets woken up when TLS renewal stops working, and who can fix the
application rather than blame the platform.

This is my own production footprint: three self-managed Docker hosts running
162 containers across 78 compose stacks, serving live public services —
client websites, a mail platform, CI/CD, workflow automation, analytics and
this blog. No cloud platform holds it together, and no team operates it.
Here is the architecture, and the two failures that changed how I check my
own work.

The shape of it

Host	What it carries	Containers	Compose stacks
Synology DSM	storage, SSO, the bulk of internal services	86	44
unRaid	CI runner, media pipeline, monitoring	37	21
Ubuntu 24.04 VPS	public entry point, mail platform, client sites	39	13

The split is deliberate rather than historical. The VPS carries what must
stay reachable and fast independent of the on-premise uplink, because that is
where public traffic and client mail land. The Synology carries what needs
disk and a stable filesystem, plus the identity provider that every other
service trusts. unRaid runs the CI runner and the heavy, spiky batch work
that would otherwise compete with the services for RAM.

What makes three hosts one system

One ingress pattern. Each host runs a Traefik reverse proxy with automated ACME TLS, so certificates renew themselves and no service owns a certificate file. Behind it there is one convention for routing and redirects, and Cloudflare sits in front of the public names.

One identity provider. authentik handles SSO with OIDC and proxy providers, so a new internal service is a new application in one place instead of a new password in a spreadsheet. Public services stay anonymous; everything else authenticates through it.

One monitoring stack. Prometheus, Grafana, cAdvisor and node-exporter across the hosts, in a single Grafana rather than three dashboards, because the interesting failures are the ones that cross host boundaries.

One delivery path. Self-hosted Gitea Action runners build and deploy. This blog ships through that pipeline — commit, runner picks it up, container rebuilds, Cloudflare serves the new build.

Backups that are tested rather than assumed. Snapshot-based backup on the NAS plus Duplicati and restic jobs, with restore procedures written down, because an untested backup is a feeling, not a backup.

Failure 1: the dashboard was lying to me

When I built the monitoring stack I trusted what Grafana showed me, and
three of the panels were wrong.

cAdvisor reported "containers" that were really cgroup pseudo-entries, which
inflated the container count. My total-storage panel counted one NAS volume
three times, because three near-identical mount points were being scraped as
separate filesystems — the number looked plausible, which is exactly what
made it dangerous. And one virtual machine exported no CPU frequency at all,
so its panel stayed empty and I read that as "no data yet" instead of "this
exporter is misconfigured".

None of those broke a service. All three would have made me report something
false to a client, because a wrong number on a dashboard is
indistinguishable from a right one. I wrote the whole thing up in
One Prometheus for unRaid, a Synology NAS, and a VPS.

Failure 2: one missing config file, three failure modes

Passbolt, my team password manager, started returning 504s and its cron job
appeared to hang. It turned out to be three separate failure modes with one
root cause: a configuration value that existed inside the running container
but never in the compose file.

A null GPG fingerprint stalled mail delivery, and because the mail send sat
inside the scheduled job, the whole job hung and took the UI with it. The
fix worked — until the container was recreated, which wiped it and restored
the original symptom, so the problem appeared to come back on its own. Then
ssl.force behind a reverse proxy produced a redirect loop that I had
introduced myself while chasing the other two.

Three symptoms, three sessions, one lesson: configuration that is not in
version control is not configuration. The full sequence is in
Passbolt UI Kept Hanging — Three Failure Modes From One Missing Config File.

How I keep the numbers honest

Numbers on a CV go stale quietly. Mine are re-derivable: a small script
reads the live Docker endpoints through the Portainer API and counts
containers and compose stacks per host, so "162 containers across 78
compose stacks" is re-measured on demand rather than remembered from the day
it was true. The same script prints the list of services that must never
appear in anything client-facing.

The rule behind it is simple. A number that cannot be re-checked becomes a
claim nobody can defend — in an interview, in a proposal, or on a call when
a client asks how you got it. If I cannot re-measure a figure, it does not
go in the document.

What I would do differently

Every configuration value in the compose file from day one, including the ones that feel like secrets of the machine rather than of the service. The Passbolt relapse existed only because a fix lived in a container.

Alert on the absence of data, not just on bad data. A blank panel and a healthy service look identical on a dashboard. If nothing arrives from an exporter for ten minutes, that is the alert.

Measure availability with something built to measure availability. A metrics pipeline tells you how a system is behaving; it does not count successful requests against total requests. If a percentage is going to appear in front of a client, it needs a probe checking endpoints on a schedule and recording the result — otherwise the honest answer is "monitored", not a number. That distinction matters more than the number.

The result

Three hosts, 162 containers, 78 compose stacks, one operator — with
automated TLS, central SSO, cross-host monitoring, tested backups and
self-hosted CI/CD. It has run client websites, a mail platform, a Telegram
support bot, scheduling automation and this blog at the same time, and it is
the reason I can take a project from "here is the requirement" to "it is
live, monitored and backed up" without handing the second half to somebody
else.

Want this for your business?

If you need someone who can build the application and run the
infrastructure it lives on — Docker and Traefik, automated TLS, SSO,
monitoring, mail, backups, CI/CD — I do exactly this, and I would rather
talk about your setup than send a brochure.

WhatsApp: wa.me/60127972969
Email: me@hoelee.com
More: hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Why Chrome Forgets Its Tabs in a Container — And How I Fixed It

Hoelee — Wed, 28 Oct 2026 04:25:38 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run a real Google Chrome inside a Docker container on my NAS. It is not a scraping instance and not a screenshot service — it is a browser I actually use: signed into the accounts a headless bot should never touch, watchable through a web desktop from any device, and drivable over the Chrome DevTools Protocol when I want to automate part of a job.

It works well, with one exception. Every time the container restarts, the browser comes back with a single blank tab.

Chrome has a setting for exactly this — "Continue where you left off". It was already enabled. It did nothing. What follows is three fixes that failed, the actual root cause, the two landmines I hit on the way, and the ~70-line keeper that finally made restarts survivable.

Why this matters

When a container browser is your workspace, the tabs are the state. Half-finished forms, a marketplace listing in progress, a dashboard filtered to the right date range, three tabs open on a customer's infrastructure — that is the work. A browser that opens on about:blank means the first ten minutes after every restart are spent rebuilding context you already had.

I looked for this answer more than once and found nothing but "set restore_on_startup" advice that does not survive contact with a container. If you have hit the same wall, this is the wall explained.

The setup

A single Docker stack on a Synology NAS, two browsers in it (real Chrome and Brave), each behind my own SSO gate:

Piece	Value
Image	`lscr.io/linuxserver/chrome` — real Google Chrome 154
Desktop	Selkies web desktop, container `3000`/`3001` → host `3030`/`3031`
Automation	CDP published at `http://<nas>:9232/json/version`
Profile	a bind mount, `/config/profile` inside the container
Launch flags	`--remote-debugging-port=9222 --user-data-dir=/config/profile --restore-last-session`

The profile is a bind mount, so the logins, cookies and history genuinely do persist across a container rebuild — that part was never broken. Only the session (which tabs were open) was lost.

Attempt 1: the flag

Chromium ships a switch that sounds like the answer: --restore-last-session. It is documented as "restore the last session after an unexpected exit".

I added it via the image's CLI env var and confirmed it was actually applied to the running process, because these images build the command line from a wrapper script:

docker exec chrome sh -c 'tr "\0" " " < /proc/$(pgrep -f "google-chrome --" | head -1)/cmdline' | tr ' ' '\n' | grep -E 'restore|user-data-dir|remote-debug'
# --remote-debugging-port=9222
# --user-data-dir=/config/profile
# --restore-last-session

Restart. Blank tab. The flag was there; it just had nothing it was willing to restore.

Attempt 2: edit the profile's preferences

Chrome stores "Continue where you left off" in the profile's Preferences JSON as session.restore_on_startup: 1. So I stopped the container and edited it directly, along with the exit_type marker:

import json
p = '/config/profile/Default/Preferences'
d = json.load(open(p))
d.setdefault('session', {})['restore_on_startup'] = 1   # 1 = restore last session
d.setdefault('profile', {})['exit_type'] = 'Normal'
json.dump(d, open(p, 'w'))

This is the classic advice you will find online, and it half-worked: after the next boot, Chrome had kept restore_on_startup: 1 in the file — and still opened a blank tab. What it did not keep was my exit_type: it had already rewritten that to Crashed at startup.

That rewrite turned out to be the first real clue. Chrome treats startup state as a security boundary — the settings a hijacker would most like to change (homepage, search provider, startup pages) are MAC-checked against a separate Secure Preferences file. A value edited on disk does not automatically become a value Chrome obeys.

Attempt 3: a managed policy

The supported way to force browser behaviour is a managed policy, which is authoritative and not subject to that MAC check. On Linux, Chrome reads /etc/opt/chrome/policies/managed/*.json:

{ "RestoreOnStartup": 1 }

Two details before you bother: Chrome ignores a policy file that is writable by the user running the browser, so it needs root:root and mode 644 — and on a Synology share, chmod refuses to run even under sudo for some directories, so verify the mode you already have instead of assuming you changed it.

Mounted into the container, verified present after boot:

docker exec chrome sh -c 'ls -l /etc/opt/chrome/policies/managed/; cat /etc/opt/chrome/policies/managed/hoeleepolicy.json'
# -rw-r--r-- 1 root root 28 hoeleepolicy.json
# { "RestoreOnStartup": 1 }

Restart. Blank tab. Still.

Attempt 4: prove the exit is ungraceful

At this point the question stopped being "which setting do I need" and became "what does this browser actually see when the container stops".

Chrome's restore machinery needs a session it considers interrupted or resumable. It rolls Current Session into Last Session on a clean exit. I asked the CDP endpoint to close the browser gracefully instead of pulling the container out from under it:

# browser-level target from /json/version
ws.send(json.dumps({"id": 1, "method": "Browser.close", "params": {}}))

Chrome exited — and the desktop session relaunched it within seconds. exit_type was back to Crashed immediately.

That is the whole trick, and it explains all three failures: the browser never exits cleanly. When you stop the container, the desktop session is torn down and the browser is killed with it. No clean exit means no Last Session rollover, which means there is nothing for "continue where you left off" — or a RestoreOnStartup policy — to restore. The flag, the preference and the policy were all working correctly and all had nothing to work with.

The data was never missing. The profile's session directory was being written continuously, seconds apart, the whole time:

-rw------- 1 hoelee users 40722 Session_13434833255312492
-rw------- 1 hoelee users 58879 Tabs_13434833757632480
-rw------- 1 hoelee users 79361 Tabs_13434833833646133

The fix: stop configuring, start keeping

If the browser cannot be trusted to remember its own session, the container can remember it instead. Two jobs, both tiny:

On start: wait for the browser, and if it came up with no real page open, re-open the URLs it had last time.
Forever after: every 60 seconds, snapshot the open page URLs.

That is a background script launched from the image's container-init hook. The version I run, trimmed to the useful parts:

tabs_keeper.py — Tab restore & snapshot loop (42 lines)

#!/usr/bin/env python3
"""Restore last session's tabs on start, then snapshot open tabs every 60s."""
import json, os, time, urllib.parse, urllib.request

CDP   = "http://127.0.0.1:9222"          # loopback inside the container
STORE = "/config/tabs-last.txt"          # lives in the persistent profile mount

def open_tabs():
    """http(s) page URLs, or None when the browser isn't answering yet."""
    try:
        with urllib.request.urlopen(CDP + "/json/list", timeout=5) as r:
            tabs = json.load(r)
    except Exception:
        return None
    return [t.get("url", "") for t in tabs
            if t.get("type") == "page" and t.get("url", "").startswith("http")]

saved = [u.strip() for u in open(STORE) if u.strip()] if os.path.exists(STORE) else []

# 1) restore — wait up to 3 minutes for the browser to come up
if saved:
    for _ in range(60):
        if open_tabs() is not None:
            break
        time.sleep(3)
    if open_tabs() == []:                      # nothing but a blank new tab
        for url in saved[:20]:
            req = urllib.request.Request(
                CDP + "/json/new?" + urllib.parse.quote(url, safe=""), method="PUT")
            urllib.request.urlopen(req, timeout=10).read()
            time.sleep(0.7)

# 2) snapshot — write even an empty list, so "I closed everything" is respected
while True:
    time.sleep(60)
    tabs = open_tabs()
    if tabs is None:
        continue
    tmp = STORE + ".tmp"
    open(tmp, "w").write("\n".join(tabs) + ("\n" if tabs else ""))
    os.replace(tmp, STORE)                     # atomic; a kill can't truncate it

Two design choices worth copying:

Restore only when the browser came up empty. If a future Chrome version does restore its own tabs, the keeper stays out of the way instead of duplicating every tab.
The snapshot is atomic and may be empty. Writing through a temp file plus os.replace means a container killed mid-write cannot leave a truncated list, and an empty snapshot is meaningful data: it means you closed everything on purpose, so the next start should stay empty.

The container-init hook that runs it:

#!/bin/sh
# clear stale singleton state (see below)
rm -f /config/profile/Singleton* 2>/dev/null
# long-running, so background it or you block container init
/usr/bin/env python3 /custom-cont-init.d/tabs_keeper.py >> /config/tabs-keeper.log 2>&1 &
exit 0

Landmine 1: a stale `Singleton*` stops Chrome from starting at all

While testing restarts I hit a worse failure than a blank tab: the container came up healthy with no browser in it. No google-chrome process, CDP dead, the relay's healthcheck reporting unhealthy, nothing in the logs beyond the desktop starting.

The cause is the profile's SingletonLock/SingletonSocket/SingletonCookie symlinks. They encode "this profile is owned by PID n on host h" — and inside a container, PIDs are reused across restarts, so the next boot can find its own PID already claiming the profile and exit immediately. Deleting them at container start, before the browser launches, is the fix. It is also why that rm -f sits in the hook above.

Landmine 2: `custom-cont-init.d` only runs executable files — and runs them with bash

The LinuxServer images execute every file in /custom-cont-init.d, but only if it is executable, and each one through /bin/bash. Which means a Python file you drop in there must not be executable, or it gets fed to bash and exits 2:

[custom-init] tabs_keeper.py: executing...
[custom-init] tabs_keeper.py: exited 2

The split that works: a .sh hook at 755 that backgrounds the long-running work, and the .py next to it at 644. And anything that does not return immediately has to be backgrounded, or container init waits on it forever.

The plumbing trap that came with it

The keeper talks to Chrome over CDP on loopback, which is why it needs no ports. If you also want to drive the browser from another machine, there is one more container in the picture, because Chrome ≥ 136 pins its devtools endpoint to 127.0.0.1:9222 and ignores --remote-debugging-address. The standard workaround is a socat sidecar sharing the browser's network namespace:

chrome-relay:
  image: alpine/socat
  network_mode: service:chrome            # must share the browser's netns
  command: socat TCP-LISTEN:9223,fork,reuseaddr TCP:127.0.0.1:9222
  healthcheck:
    test: ["CMD-SHELL", "nc -z 127.0.0.1 9222 || exit 1"]

The port mapping does not go on the relay (it has no namespace of its own) — it goes on the browser service: 9232:9223.

That sidecar has one behaviour you have to internalise: restart the browser and the relay dies with it. The namespace it was living in is rebuilt underneath it, socat keeps running and its sockets stop working, so CDP answers with connection reset while the container still looks fine. Fix the symptoms in this order:

docker restart chrome && docker restart chrome-relay   # order matters

The healthcheck above is what turns a silent failure into a visible unhealthy flag. If you want to drive a container browser yourself, I wrote the client side up separately in Scraping a Bot-Walled Marketplace With a Warm Browser Session.

What I'd do differently

Determine whether the process can exit cleanly before trusting any "restore on startup" mechanism. I spent three attempts configuring a feature whose precondition — an exit the browser notices — was never met. One Browser.close test would have collapsed the whole search.
Don't hand-edit browser preferences. They are MAC-checked against Secure Preferences; a plausible-looking value on disk is not a value Chrome obeys. If you need a switch to be authoritative, use a managed policy — and verify the policy is loaded instead of assuming the file is enough.
Check who is allowed to write the policy. A root-owned 644 file in /etc/opt/chrome/policies/managed is the shape; a user-writable one is ignored silently.
Treat "the browser is running" and "the container is running" as different questions. The stale-singleton failure looked like a healthy container from the outside.
Snapshot state instead of asking software to remember it. Anything with a hard kill in its shutdown path — containers, kiosks, desktop sessions — is a poor place to rely on clean-exit bookkeeping. A 60-second atomic snapshot is boring, and it works.
Watch the empty-snapshot case. Because step 2 writes even an empty list (that is how "I closed everything" gets respected), a browser that dies mid-session and gets relaunched blank by the desktop supervisor will have its empty tab state snapshotted 60 seconds later, erasing the last good file. The cheap guard is to record the browser's webSocketDebuggerUrl from /json/version and only overwrite the snapshot when it is the same instance you started.

The result

Tabs restored 2/2 on both browsers across a real container restart (restored 2/2 tabs in the keeper log), verified over CDP afterwards
Logins, cookies and localStorage confirmed surviving the same restarts — the persistent profile was never the problem
Three stacks (two browsers plus a throwaway experiment) collapsed into one, with the relay's healthcheck making the one restart-order trap visible instead of silent

Want this for your business?

If you want a browser that stays logged in — running on your own hardware, reachable from anywhere, behind SSO, with tab state that survives restarts and a CDP endpoint your scripts can drive — that is the setup I run and maintain.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; self-hosted infrastructure and automation is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

How do you manage persistent browser user-data-dir volumes and state across container restarts without profile lock corruption? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing

Hoelee — Tue, 27 Oct 2026 04:26:26 +0000

Originally published on my blog — read it there with the full context, code and updates.

The question I could not answer: which number do I ship?

My transcoding queue decides, per file, which encoder does the work: copy
when the source is already efficient, NVENC on the GPU, x265 on the CPU —
and now Intel Quick Sync. Each tier has two knobs that decide quality, and
every tier except QSV had numbers I had measured on my own library.

QSV had none. And the code refused to guess: a shipping default of
qsv_quality = 0 / qsv_preset = "" means nobody has calibrated this,
and the engine treats that as a hard exclusion — QSV can exist in the code
and never be selected on any machine, including the one that has the
hardware.

So the task was not "turn on QSV". It was "produce measured numbers for two
knobs, so the encoder becomes selectable honestly".

Problem one: the hardware is not in the machine where the app runs

The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through
and no Intel graphics of any kind — which is exactly why QSV had never been
measured: the obvious move is to run one test locally and read the numbers
off.

The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed:

00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770]
  driver: i915            refcount: 0
/dev/dri/card0  /dev/dri/renderD128        (present, world-accessible)
no running container declares a dri device
virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio)

Two conclusions, both useful. The VM's Error creating a MFX session is
expected and not a bug to chase — the VM can never see that iGPU. And
the only way to measure QSV is on the Linux side, in a container, on the
host.

Measure with the engine's own gate, not with ffmpeg

This is the part that makes the numbers worth anything. I did not write
ffmpeg commands and compute SSIM myself — the calibration runs the engine
that will do the work, with its own verification and its own accept/reject
gate:

python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \
  --qsv-quality Q --qsv-preset P --path <real source> --keep-local --overwrite

--keep-local means encode, verify, run the gate, and then do not land
the file — the verdict is what I am collecting, not a new library. Sources
came in read-only over an existing SMB mount, and I hashed each source
before and after: unchanged.

The image is linuxserver/ffmpeg (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17)
plus python3 — plus numpy, which is not optional. Without it the gate
prints flat-frame scan skipped: No module named 'numpy' and quietly
changes how near-flat frames are judged, which changes verdicts.

-global_quality maps to RateControlMethod: ICQ on hevc_qsv, which is
the knob being calibrated.

The ladder: on a thin source, low ICQ makes the file bigger

First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a
"thin" file, the kind that is already squeezed):

ICQ	engine verdict	mean SSIM	video = source
18	FAIL size — 176.8%	—	—
20	FAIL size — 141.8%	—	—
22	FAIL size — 107.6%	—	—
24	ok	0.9967	81.1%
26	ok	0.9959	61.0%
28	ok	0.9951	46.4%
30	FAIL quality	0.9942	—
32	FAIL quality	0.9942	—

Three things fall out of that table.

The counterintuitive one: low ICQ does not mean safe. ICQ 18–22 produce
files larger than the source, so the engine's size ceiling (100% of the
source's video bitrate) rejects them. QSV is not alone in this — NVENC at a
low cq does the same thing on thin inputs — but the boundary sits higher
here: the usable range starts at 24, not 18.

Second, there is a cliff between 28 and 30. 30 and 32 fail on quality,
and they fail on the first attempt. That phrase matters, because the
engine retries a failed quality attempt at global_quality - 2 — lower
ICQ, higher quality — and logs the retry. A line reading "ok at quality+2"
is a first-attempt failure, not a pass. I read that line backwards in my
own notes the first time; the rule is to read the first attempt, not the
recovery.

Third, the floor is where the choice gets made: with a 0.995 mean-SSIM
floor, q28 passes with 0.0001 of margin and q26 with 0.0010.

Full files confirm the ladder — and one shortcut survives

Excerpts (180-second lossless slices) are how you rank ten candidates
without ten full encodes. They are only legitimate if they agree with
full-file runs, so I ran the real files end to end:

source	ICQ	preset	video = source	file	SSIM min / mean	encode
254 MB, 2023 s, 872 kbps	26	medium	61.0%	168,435,266 B (66%)	0.9938 / 0.9960	360 s ≈ 5.6× realtime
the same file	28	medium	46.3%	54%	0.9927 / 0.9952	330 s
315 MB, 3280 s, 546 kbps	26	medium	50.4%	60%	0.9965 / 0.9977	600 s ≈ 5.5× realtime

The 180-second excerpt of the first file measured mean 0.9959; the full file
measured 0.9960. That is close enough to trust the shortcut for quality
ranking — and I now have the evidence to say so rather than hope so.

The preset that does nothing

The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast
30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick
veryfast, save 30 seconds per file.

Then the full files disagreed with it. The same source at ICQ 26, one run
with fast and one with medium:

fast:    168,435,266 B   enc 360 s   mean SSIM 0.9960
medium:  168,435,266 B   enc 360 s   mean SSIM 0.9960

Byte-for-byte identical, same encode time, same quality. The 30-vs-60-second
difference in the excerpt layer was per-run fixed overhead, not encoding
speed — and it is the kind of number that would have gone into a config file
and a blog post if I had stopped one step early.

So: excerpts rank quality, never speed. Speed has to be measured on full
files, where the fixed overhead disappears.

What I shipped

qsv_quality = 26, qsv_preset = medium — 61.0% of the source's video
bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4%
at mean 0.9977 on the thinnest file in the library. q28's margin over the
floor is 0.0002, and the cost of a quality failure is two full encodes plus
a park, so the extra 15 percentage points of savings are not worth the tail
risk.

Two supporting measurements worth recording, because they answer "is it QSV
or is it the file?":

Cross-encoder control. One 720p source with non-uniform cadence scored QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and the engine's alignment search reported no alignment explains the mismatch. The rejection belongs to the source, not to Quick Sync.
Concurrency. The engine refuses a second encoder on a shared work root: another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old). With separate work roots, two jobs run — and take 201 s where one takes 107 s. Twice the wall clock, zero throughput gain. One QSV job at a time.

Four traps, each of which cost a cycle

Never mount a shared library at /lib. I mounted the SMB share into the container at /lib — which is where the container's own dynamic loader lives. All eight jobs died in under a second with exec /bin/bash: no such file or directory, which reads like a broken image, not a bad mount. /library works.
Synthetic sources lie about ICQ. hevc_qsv's rate control is rejected by the driver for rgb24 input (Selected ratecontrol mode is unsupported) — so a testsrc/mandelbrot self-test concludes the opposite of the truth. Use a real file, or add format=nv12.
Already-HEVC sources cannot be calibrated at all. The gate answers SKIP already HEVC hvc1 and --overwrite does not lift it. The fat 1080p H.264 originals in this library had all been converted long ago; the remaining fat files are HEVC — so that tier has nothing left to measure on. That is a missing input, not a skipped task.
Read the first attempt, not the retry (see the cliff above).

What I would do differently

Measure speed on the artifact you will actually run, not on a slice of it. My one wrong number in this whole exercise came from trusting an excerpt's wall clock.
Make the machine prove it has the hardware before you plan the test. Six minutes of lspci and virsh dumpxml retired the "why does QSV fail in the VM" question that had been open for weeks.
Keep the gate in the loop. Because the engine's own verification judged every run, the tables above are decisions, not opinions — the same gate that will reject a bad file in production rejected my ICQ 30.
Write the honest scope line into the notes. These numbers are Linux iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the Windows MSDK path on the same silicon, and I cannot claim they agree. What this establishes is that the command runs on real hardware, where the quality↔size curve sits, and how the lock behaves — leaving a re-measurement on Windows+Intel as one table, instead of another afternoon.

The result

QSV went from "present in the code and never selectable" to a calibrated
tier with published numbers, and the safety rule changed with it: the test
that used to pin shipping defaults must mean uncalibrated now pins
shipping defaults must be measured values, while keeping every assertion
that an empty value is a hard exclusion. The doctor output on a machine with
no Intel GPU moved from an informational "waiting for hardware" line to a
warning — which is its own small lesson about what a calibrated encoder
failing self-check should look like, and why hardware the machine does not
have belongs in a summary row rather than a warning.

Need Resilient Hardware Video Pipelines & Automated Quality Gates?

If you manage GPU/iGPU transcoding fleets (NVENC, Quick Sync, VAAPI) or need strict frame-level quality assertions (SSIM/VMAF) to keep defective encodes out of production libraries:

WhatsApp: +60 12-797 2969
Email: me@hoelee.com
Website: hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

For video pipeline engineers: do you rely on hardware QSV/NVENC for bulk storage transcode or bite the CPU bullet for x265 crf? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App

Hoelee — Mon, 26 Oct 2026 04:26:02 +0000

Originally published on my blog — read it there with the full context, code and updates.

Can I trigger the browser's translate feature from JavaScript? No. That answer cost me an afternoon to accept — but the more interesting discovery came first: my app was already sabotaging the browser's translation, and it did it with a single attribute.

What I was working on

I run a private, invite-only reading platform at book.hoelee.com: a self-hosted reader (Kotlin/Spring backend, Vue single-page app) holding my library, with accounts for invited readers. The interface is Chinese-only. Not "Chinese-first" — only. I grepped every served bundle before believing it: no i18n library, no locale files, no language switcher. Just a Simplified/Traditional character converter and a list of English TTS voice names.

Then I started handing out invite cards to English-speaking readers. They could register, and then they met 书架, 浏览书仓, 确定, 设置 — every control in a language they don't read. That isn't a polish problem. For half the people I invited, the product was unusable.

One constraint shaped everything that follows: I don't modify the app. Its upstream repository is archived and the UI is a minified Vue bundle, so anything I patch inside the image dies the next time I swap the image. Whatever I build has to live one layer out — in the nginx gateway I already run in front of the app.

Finding 1: the app declares the wrong language, and that disables translation

Before writing a single line of code, I looked at what the browser sees:

$ curl -s https://book.hoelee.com/index.html | grep -o '<html[^>]*>'
<html lang="en">

The shell says English. The interface is 100% Chinese.

That matters more than it looks. Chrome, Edge and Safari decide whether to offer translation largely from the page's declared language. A page that declares English while rendering Chinese text doesn't get the "Translate this page?" prompt — the single most useful accessibility feature for my new readers was silently switched off, and right-click → Translate to English is a ritual most people never learn.

The fix is one line, applied to the response on the way out instead of inside the app:

location = /index.html {
    # sub_filter cannot rewrite a gzipped body -> ask upstream for plain HTML
    proxy_set_header Accept-Encoding "";
    sub_filter_once on;
    # the app declares lang="en" while its UI is Chinese -> browsers then never
    # offer Chinese -> English translation. Declare the truth.
    sub_filter '<html lang="en">' '<html lang="zh-CN">';
    proxy_pass http://hectorqin-reader:8080;
}

Verify the same way you found it: curl -s https://book.hoelee.com/index.html | grep -o '<html lang="[^"]*"' → <html lang="zh-CN". The declared language is what the translate prompt keys off, so with the page finally declaring Chinese the offer can fire correctly. (The exact prompt still depends on each browser and its language settings — what I verified on the wire is the attribute itself.) Because the rewrite happens at the proxy, it survives every app update.

Finding 2: there is no browser-translate API

With the page honest again, the obvious next question was whether I could offer translation myself — a button that does what the browser's menu item does. There is no such API. Browser translation is a UI-level feature; a page cannot invoke it, and no web-exposed surface exists to script it.

So I worked through the alternatives and rejected them one by one:

Approach	Why I dropped it
Rewrite the app's JS bundles to swap strings	Couples me to minified internals; the next image breaks it. Also violates the no-modify rule in spirit.
Proxy the site through `translate.goog`	It's a different origin. The app keeps its auth token in `localStorage`, which is per-origin — readers would land logged out, and the service worker/PWA breaks. Every API call would also round-trip through Google.
Tell readers to right-click → Translate	Works for book text, but it's a per-visit, per-browser ritual that does nothing for someone who doesn't know it exists. Not good enough for a product where I control the whole stack.
Fork the app and add i18n properly	Weeks of work on an archived codebase, for an interface I'd have to re-translate on every upstream change.

What's left is the unglamorous one: translate the interface myself, in the browser, with a dictionary — and wrap it in a button that only exists inside the app.

The fix: a dictionary and a button, injected at the gateway

The gateway already serves my front page, so it can serve one more static file and inject a script tag into the app shell only:

location = /gate-en.js {
    root /usr/share/nginx/html;
    add_header Cache-Control "public, max-age=300" always;
}

location = /index.html {
    # ... the lang rewrite above ...
    sub_filter '</head>' '<script src="/gate-en.js" defer></script></head>';
    proxy_pass http://hectorqin-reader:8080;
}

That's the whole deployment: a 40 KB script, loaded only on /index.html, so the button exists inside the app and nowhere else. The script does three things.

1. Harvest the vocabulary from the app's own bundles

The UI strings are already in the minified JavaScript as quoted literals. Extract every one that contains a CJK character and count them:

import re, collections
CJK = re.compile(r'[\u4e00-\u9fff]')
count = collections.Counter()
for bundle in bundles:
    for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle):
        if CJK.search(m.group(1)):
            count[m.group(1)] += 1
print(len(count))   # 840 unique strings, most frequent first

840 unique strings came out: menus (书架, 书源管理), confirmations (确认要删除所选择的书籍吗?), settings labels (段落行距), error messages (本地书籍源文件不存在). The dictionary ended up at 871 entries — the harvest plus everything I found by running the app in English mode and watching what stayed Chinese.

2. Match like the DOM actually is, not like you wish it were

Naive substring replacement mangles Chinese compounds: 书架 sits inside 加入书架, and single characters are worse (页 inside 页面). Three rules fixed it:

Whole-label match first, on whitespace-stripped text. The app stores labels like " 书架 " with padding, so a dictionary key of 设置 has to match " 设置 " in the DOM.
Then longest-first substring replacement, restricted to keys of two or more characters.
One-character keys are whole-label only (章, 页, 条), so they can never corrupt a longer word.

function tr(s) {
  if (!s || !CJK.test(s)) return s;
  var hit = EXACT[s.replace(/[\s\u3000]/g, '')];   // whole label, whitespace-normalised
  if (hit) return keepWS(s, hit);
  var t = s;
  for (var i = 0; i < KEYS.length; i++) {          // KEYS: longest-first, length >= 2
    if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]);
  }
  // tidy count phrases: 共58个可用书源 -> "58 usable sources"
  return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2');
}

3. Keep up with the framework

It's a Vue app, so the DOM is rewritten constantly — a translation pass that runs once is a translation pass that's half wrong a second later. A MutationObserver with a 150 ms debounce re-runs the walk after any change.

The pass is idempotent: once a label is English it contains no CJK, so nothing is rewritten and there is no write loop. Text nodes plus placeholder, title, aria-label, alt and input value attributes all get translated, which is why tooltips and the search box come along too.

The UI is a small pill in the bottom-right corner: EN when off, 中文 when on, remembered in localStorage, with a "Hide this button" link for readers who prefer their own translator.

What it deliberately does not translate

Book text. The reader renders EPUB content inside an iframe; my walker skips iframes by design, and I would never machine-swap a novel's prose anyway.

Also left alone: book titles and authors, book-source names, and the group names I created myself (梯子, 精品, 正版), plus the app's poem tagline. Those are data, not interface. Translating the chrome is a courtesy; silently rewriting a user's own content is a different and much worse idea.

For book text the browser's own translation is still the right tool — and it now actually gets offered, thanks to Finding 1. The button's popup says exactly that.

Measure it, don't eyeball it

"Screenshots look mostly English" is not a result. I counted CJK-containing text nodes in the live DOM before and after toggling the button:

let cjk = 0, total = 0;                          // walker skips the injected UI + script/style
while ((n = walker.nextNode())) {
  const t = n.nodeValue.trim();
  if (!t) continue;
  total++;
  if (/[\u4e00-\u9fff]/.test(t)) cjk++;
}

Screen	Before	After	Translated
Login / register dialog	80	12	85%
Bookshelf + settings	259	60	77%
Reading page chrome	214	31	88%

Everything still counted as Chinese on those screens is data — titles, authors, source names, my own group names, the tagline — which is exactly what should stay Chinese.

What I'd do differently

Don't rewrite an 871-entry dictionary by hand. I restructured mine once and silently dropped 25 keys. The symptom looked like a logic bug: 设置 stayed Chinese while every sibling label translated correctly. The matcher was fine — the entry was gone. Diff the key sets programmatically before you trust a rewrite:

python -c "print(sorted(set(old_keys) - set(new_keys)))"

Keep automated UI checks short. My headless runner hard-times-out at 60 seconds per call. A six-book loop came back HTTP 408 with an empty body, and my JSON parser reported a parse error rather than a timeout — I debugged the wrong thing for a while.

Learn the SPA's real routes before debugging it. #/reader/1 renders a blank shell and makes you think the app is broken; the actual route is #/reader?bookUrl=<urlencoded>.

Mind your own bounce logic. My gateway sends any reload back to the front page unless a sessionStorage pass flag is set. Switching the interface back to Chinese reloads the page — so the button sets that same flag first, or the reader gets thrown out of the app for changing a language setting.

Result

An English-only reader now opens the platform, taps one button, and reads the interface in English. The app itself is untouched: the whole feature is a 40 KB script served by a sidecar nginx container, so the next image update cannot break it, and the browser's own translation has a page that declares its real language to work with.

The lesson I'm keeping: when the platform won't give you an API, look one layer out. A reverse proxy is a legitimate place to add behaviour you can't add in the app — and it's usually the only place that survives upgrades.

I build and self-host platforms like this one — web apps, private libraries, internal tools for small businesses in Malaysia. If you need something similar: WhatsApp +60 12-797 2969, me@hoelee.com, or hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

When retrofitting i18n into monolithic legacy SPAs without backend API support, what architectural trade-offs do you make? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

DEV Community: Hoelee

When a Self-Hosted Model Gateway Tripped Trust & Safety

The error that sent me the wrong way

Why a shared token is not a shared key

Task dispatch instead of routing

The economics survived, because the workload splits

💬 Let's Compare Notes

Embedding Fonts So a PDF Looks the Same on Every Machine

The requirement was one sentence

Rule one: read the fonts back out of the finished PDF

Rule two: static instances only, and the reason is in the parser

Rule three: the engine speaks two weights, so map onto two slots

Rule four: do not subset against a character set you do not control

Rule five: make missing glyphs a build failure, not a surprise

What I would do differently

The result

How I Vet an Open-Source Dependency Before Betting On It

The problem, phrased as the question I'd actually search for

What I tried first, and why it wasn't enough

The six checks that actually settled it

1. Read the repo's vital signs before any feature claim

2. Grep for the feature instead of reading for it

3. Read the target branch line at the top of a feature doc

4. Read the data model, not the feature list

5. Check which paid tier gates the feature you need

6. Check that the money rail works in your country

Then: ask the fallback question before you need it

What I'd do differently

The result

NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You

Why this matters

What the docs say, and what the image does

How to test it without risking anything

The other half of the licensing story

The fork that search results recommend — check it's alive first

What to do instead

What I'd do differently

The result

Want this for your business?

The Forward-Auth Gate That Verified Perfectly — and Wasn't Live

Why this matters beyond my one app

The setup: SSO for an app that has no SSO

The trap: I verified the layer I changed

The permission wall on the way to the fix

Optional SSO: gate the UI, leave the API alone

The honest part: this is not single sign-on

Reverting it properly (and the arithmetic that saved ten other apps)

What I'd do differently

The result

Want this for your business?

Your Disk-Full Alert Is Lying: Stop Alerting on Percentages

The three alerts that taught me

Alert on the consequence, not the ratio

Noise is not harmless

Thresholds are personal, shapes are not

The result

Want this for your business?

Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)

The searchable question

The actual cause: a variable that filtered on itself

Why my verification missed it

The second trap: $__all is not .*

Not every empty panel is a bug

What I'd do differently

The result

Want this for your business?

Stop Writing Agent Prompts for Deterministic Work

The rule

Where the model does earn its place

The part that actually bit me

The short version

Running Production Infrastructure Solo: 162 Containers Across Three Hosts

Why this matters

The shape of it

What makes three hosts one system

Failure 1: the dashboard was lying to me

Failure 2: one missing config file, three failure modes

How I keep the numbers honest

What I would do differently

The result

The second trap: `$__all` is not `.*`

Landmine 1: a stale `Singleton*` stops Chrome from starting at all

Landmine 2: `custom-cont-init.d` only runs executable files — and runs them with bash