DEV Community: Sarvar Nadaf

I built an offline AI that knows your last frost date, no internet, no API

Sarvar Nadaf — Fri, 09 Oct 2026 12:44:55 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

The most useful number in a vegetable garden is the last spring frost date. Get it wrong by a week and one cold night turns your tomato seedlings to mush.

Every tool that gives you that number has the same flaw. It lives on a server. You look it up on your phone, standing in the one corner of the allotment that gets a bar of signal, hoping the page loads before the rain starts.

So I built FrostWise: it predicts your last frost and tells you what to plant, and it does the whole thing on your laptop with the Wi-Fi off. Two open-weight models, no API key, no account, nothing leaves the machine.

I built it for a specific person: my friend who keeps a plot at a community garden on a hill where the signal drops to nothing. Every spring she texts me "is it safe to plant yet" because she cannot load a frost-date site from the plot itself. FrostWise is the answer she can keep on her own phone and open with no bars.

What I Built

FrostWise answers one question honestly: when does the cold let go where you are, and what should you do about it.

You pick a location (or type your own climate numbers), choose a crop, and it returns three things:

A date. Your predicted last spring frost, with an honest range, not a fake single-day promise.
The advice. Plain language for your exact crop. Wait until after the frost for tender things like tomato and basil. Sow a few weeks early for hardy things like peas and kale.
A reason to trust it. The number comes from a model you can inspect and retrain, and the range tells you the risk.

It gets people off the screen in the most literal way. The screen is the ten seconds you spend before you walk outside and put a seed in the ground. The whole point is to make that part short.

Who it is for: anyone with a patch of soil and a spotty signal. Community plots, hillside allotments, a herb bed at a trail-head cabin. The places where a cloud API is useless are exactly where people garden.

Demo

Try it now: d2damz5gbd2rbh.cloudfront.net

That hosted link is a static site (S3 plus CloudFront, near zero cost) that serves real, pre-computed TabPFN-v2 and Gemma results for the preset cities, so you can click through actual model output. The live models themselves run on your own machine. For custom climate input, fully offline, clone the repo and run python run.py.

Here is the full flow across three very different climates: Miami (effectively frost-free), Chicago (a real May frost), and Fargo (frost that lingers into summer). Same code, three honest answers.

Here is what one answer looks like: a predicted last frost, the honest range around it, and planting advice written locally by Gemma for the exact crop.

The frost date animates in, then a local Gemma model writes the planting advice live. Nothing in that video calls out to the internet.

Code

simplynadaf / frostwise

Know when the frost lets go. An offline-first planting planner: TabPFN-v2 forecasts your last spring frost and Gemma 3 writes the advice, fully local, no internet, no API. Built for the Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass.

🌱 FrostWise: Know When the Frost Lets Go (2026)

An offline-first planting planner. An open-weight model forecasts your last spring frost and tells you what to plant, on your own laptop, with no internet, no account, and no API bill. The frost date is predicted by a tabular foundation model, the advice is written by a local language model, and nothing you type ever leaves your machine.

Hacktoberfest Open-Source AI Challenge · Week 1: Touch Grass

🔗 Live demo: d2damz5gbd2rbh.cloudfront.net · 💻 Run it live: Getting Started (the full offline models run locally) · 🎬 YouTube demo: coming soon

Note

About the live demo. The hosted preview is a static site (S3 + CloudFront) that serves real, pre-computed TabPFN-v2 + Gemma results for the preset cities, so you can click through actual model output. The live models themselves run on your machine with no server and no internet. For custom…

View on GitHub

Clone it, python run.py, and it is on localhost:8077. The first forecast caches the model weights once; after that you can pull the network cable and it still works.

How I Built It

The whole thing is two open-weight models doing one honest job each, locally.

The forecast: TabPFN-v2

TabPFN is a tabular foundation model from Prior Labs. It is the strange and wonderful part. Instead of training a model on my frost data, I hand it the data as context and it predicts in a single forward pass. No training loop, no hyperparameter search, no GPU.

from tabpfn import TabPFNRegressor
from tabpfn.constants import ModelVersion

reg = TabPFNRegressor.create_default_for_version(ModelVersion.V2, device="cpu")
reg.fit(X_train, y_train)          # "fit" is just loading the context
pred = reg.predict(X_test)         # one forward pass, a few seconds on CPU

I use the v2 weights on purpose. The default TabPFN-3.5 weights are non-commercial and want a browser login. The v2 weights are the Prior Labs License (Apache-2.0 plus attribution), so the whole stack stays genuinely open and runs headless. For an "open innovation" project that distinction is the point, not a footnote.

The features are the things that actually drive frost timing: latitude, elevation, the February and March mean temperatures, the coldest winter night, and an ENSO index for that winter. The target is the last-frost day-of-year. On a plain 8-core CPU it predicts in about 3 seconds, and I read the 10 to 90 percent quantiles for the range.

For a Chicago-like input it lands on May 20, range May 15 to 26. For Fargo it says June 19. Those match the real climate norms, which told me the model was learning the physics and not memorizing noise.

One honest note on the data. The 400-row dataset in the repo is grounded but synthetic: I generated it from documented last-frost norms for 20 real United States locations, so the relationships are real, but it is a teaching dataset, not a live weather feed. That is a feature, not a dodge. Point fit at your own weather-station CSV and the same model forecasts your actual backyard. The open stack is what makes that swap a one-line change instead of a support ticket.

The advice: Gemma 3, local via Ollama

The date is a number. A gardener wants a sentence. So a local Gemma 3 model, served by Ollama, turns the forecast plus the crop into guidance.

payload = {
    "model": "gemma3:1b",
    "prompt": f"{SYSTEM}\n\nLocation: {station}. Last frost: {date}. Plant: {crop}.",
    "stream": False,
    "options": {"temperature": 0.3, "num_predict": 160},
}

I started with gemma3:270m because it is tiny. It was too tiny: it ignored the frost date and chatted. gemma3:1b (about 815 MB) follows the instruction cleanly and still runs comfortably on CPU. One honest line in the design: the model is told the date, it never invents one. The number is always TabPFN's.

The fallback: when the model is not there

A tool that promises "works offline" has to survive the model being unreachable too. If Ollama is down, a deterministic tender-vs-hardy rule writes a correct tip from the same date. The app never shows a frightened spinner that never ends.

def advise(plant, forecast):
    try:
        return {"text": call_gemma(plant, forecast), "source": "gemma"}
    except Exception:
        return {"text": rule_based(plant, forecast), "source": "offline-fallback"}

The shell

A small FastAPI server wires the two models together and serves a single-page UI (Three.js for a quiet night-sky background, no build step). The server only ever talks to localhost. There is no outbound call anywhere in the request path once the weights are cached.

What the Data Actually Says

Once the model was fitting cleanly, I did the thing a forecast tool rarely does: I asked it what it had learned. I swept each feature across its real 10th-to-90th-percentile range and measured how many days the predicted last frost moved. Every number here is reproducible with python scripts/analyze.py.

The result surprised me.

What changes	Realistic swing moves the last frost by
February mean temperature	50 days
March mean temperature	32 days
Coldest winter night	27 days
Latitude	24 days
Elevation	12 days
El Nino / La Nina (ENSO)	10 days

Gardeners obsess over latitude. "I'm up north, so I plant late." But in this data a cold-versus-mild February swings your last frost more than twice as hard as how far north you are. The month most people ignore, the dead one before anything grows, is the strongest single tell. Latitude is a slow backdrop; February is the actual signal.

Two more things fell out of the sweep:

The El Nino signal is not shared equally. I expected it to shift everyone. It barely touches the cold north: at a Fargo-like station the La Nina to El Nino swing was about 0.1 days, basically nothing. At a Houston-like station it was 8 days. The climate oscillation you hear about on the news moves the mild south and leaves the frozen north alone.

Elevation is real but modest. At 40 degrees north, climbing from sea level to 1600 meters (think Denver) pushed the last frost about 13 days later. Real, worth knowing, but a quarter of February's pull.

None of this is on a frost-date website, because those sites look up a static average by ZIP code. They cannot tell you what moves your number. A model you can interrogate can. That is the difference between a lookup table and something that has actually learned the shape of the problem.

Why Does Open Innovation Matter?

I kept asking: would a closed, hosted model have made this better? Every time the answer was no.

Offline is the entire premise. The place you most need a frost date is the plot with no signal. A closed API cannot reach it. Open weights on your own disk work there. This is not a nice-to-have for FrostWise, it is the reason it exists.

Your location is yours. "Where do you garden" is personal data. With open weights there is no server to send it to, no account to create, nothing logged. A hosted model turns your garden into someone else's row in a database.

It costs nothing. Open weights mean zero per-request fees. A community garden can run it a thousand times and the bill is still zero. A metered API would make the same tool expensive at exactly the moment you want to share it.

You own the brain. You can read how TabPFN predicts, retrain it on your own weather-station records, and swap Gemma for any open-weight model Ollama can run. Nothing is locked to a vendor you cannot inspect. If a forecast looks wrong, it is my data and my model to fix, not a prompt I have to beg a black box to honor.

A closed API would have been faster to wire up for about an hour, and then wrong forever for the one person standing in a field with no signal.

My Agent Session

I built FrostWise with an AI coding agent, and I kept it honest the same way the app keeps its forecasts honest: the agent proposed, I verified every claim against a real run before it shipped.

The useful parts of that session, in order:

Research before code. The agent read the challenge rules, the official template, and two past winning posts, then wrote the "why open beats closed" argument from the prompt's own bullets instead of guessing.
A feasibility spike first. Before a line of product code, it proved both models actually run on a plain CPU box: TabPFN-v2 predicting a frost date in about 3 seconds, and gemma3:1b answering a planting question offline. Only then did we build.
Catching my own overclaim. During review the agent flagged that the dataset is synthetic and made me say so in this post, rather than let "matches real climate norms" imply a live feed. That note is in "How I Built It" now because the agent pushed back.
Real bugs, found by running it. A static-path mistake made the UI load blank; a weak gemma3:270m ignored the frost date. Both were caught by actually running the thing, not by reading the diff.

The through-line: the agent is fast, but nothing shipped until a real run backed it up. That is the same contract as the product. The model proposes, the evidence decides.

Did It Touch Grass?

I ran it for my friend's plot before she planted. FrostWise put her last frost in early May; she waited, set her tomatoes out the week after, and did not lose a single seedling to a late cold night. The screen part took about ten seconds. The rest of the afternoon she was in the dirt. That is the whole idea: the tool gets out of the way and sends you outside.

Prize Categories

Best Use of Gemma. Gemma 3 runs locally via Ollama and writes every piece of planting advice. It is the whole language layer, offline, swappable, told the date so it cannot invent one.
Best Use of TabPFN. TabPFN-v2 (open weights) is the forecaster. It predicts the last-frost day-of-year from climate features in one CPU pass, with a calibrated range, no training required. The date on screen is always its number.

Secure Self-Hosted n8n on AWS: The 2026 Hardening Checklist

Sarvar Nadaf — Wed, 07 Oct 2026 13:16:38 +0000

A friend messaged me last month. He had spun up self-hosted n8n on a cheap cloud box over a weekend, wired it to Stripe, his Postgres, Slack, and a couple of Google APIs, and it was humming along beautifully. Then he asked the question that made my stomach drop:

"Wait, is port 5678 supposed to be open to the whole internet?"

It was. He had followed a popular tutorial. The tutorial told him to publish 5678:5678 in his Docker Compose file, skip the encryption key, and get on with building workflows. It worked on the first try. That is exactly the problem.

Here is the thing nobody says loudly enough: people treat n8n as an automation tool. It is really a credential aggregator. One instance can hold the keys to Stripe, your database, Slack, Google, HubSpot, and a dozen other services. Compromise the n8n box and you do not get one API key. You get all of them, at once.

In 2026 that stopped being theoretical.

Why 2026 was a rough year for n8n

This is not fearmongering. n8n had a genuinely bad year for security, and a lot of it hit self-hosted instances specifically:

Multiple critical CVEs, including a couple rated CVSS 10.0 (sandbox escape leading to full server takeover).
Leaked API tokens exposing live instances to credential theft.
Weak encryption keys recovered from public artifacts. Researchers found internet-exposed instances running with known weak keys, which means the "encryption" protecting stored credentials was effectively decorative.

The one I want to anchor on, because it is clean and easy to understand, is CVE-2026-65589.

CVE-2026-65589, in plain English

Type: information disclosure (CWE-532, sensitive data written to records).
Affected: n8n before 1.123.64 on the 1.x line, fixed in 1.123.64. The 2.x line is patched separately (fixed in 2.29.8), so check your major version and upgrade to the fixed release for whichever line you run.
What happens: when you pass credentials as custom HTTP headers inside certain LLM sub-nodes (OpenAI, Anthropic, and others), n8n masks them in the UI but writes the plaintext API key straight into the workflow execution records. Any authenticated user who can view or export executions can read them. Exported execution archives carry the same exposure.
Severity: CVSS 5.1 (medium). Not known-exploited, no public proof-of-concept.
Source: the n8n advisory GHSA-89gh-3pgc-v5h2.

Read that again. You mask the key in the UI, you feel safe, and n8n logs it in cleartext where it persists in the database. That is the kind of bug that does not need a hacker in a hoodie. It just needs one more user account with read access to executions.

Content on these CVEs was rephrased from vendor advisories for compliance.

Who this is for

If you self-host n8n, or you are about to, this is for you. If you have not stood n8n up yet, start with the previous episode, Self-Host n8n on AWS EC2 with Docker, which gets it running from an empty box to first login. This article picks up from there: it shows you the risky default setup first (so you can recognize it), then hardens it on AWS with a concrete, verifiable checklist. Everything here is runnable. I put the full insecure-vs-hardened pair, the IAM policy, the Caddy config, and a verify-from-outside script in one repo, linked at the end.

I am deliberately doing this on AWS, not on a generic VPS, and I will explain why the AWS primitives make this easier and safer than the usual guide.

The scoreboard (before vs after)

This is the whole article in one table. Same n8n, locked down. Every row was verified live on a real EC2 instance.

Item	Before (insecure)	After (hardened)
Port 5678 to the internet	OPEN (HTTP 200 from public IP)	closed (connection refused)
n8n version	`1.100.0` (CVE-affected)	`1.123.64` patched
Encryption key	none (derived default)	AWS Secrets Manager
DB password	plaintext	AWS Secrets Manager
IAM	broad	least-privilege role
Editor UI	public	IP-restricted + TLS (Caddy)

Now let me walk through how you get from the left column to the right one.

Step 0: see the problem

The insecure Compose file looks harmless. That is what makes it dangerous. The red flags:

services:
  n8n:
    image: n8nio/n8n:1.100.0     # old, CVE-affected
    ports:
      - "5678:5678"              # published to the host = open to the world
    # no N8N_ENCRYPTION_KEY
    # telemetry on, no execution pruning

Bring it up on a fresh Amazon Linux 2023 box and, from a completely different machine, run:

curl -s -o /dev/null -w 'n8n on public IP -> HTTP %{http_code}\n' http://<PUBLIC_IP>:5678

I got back:

n8n on public IP -> HTTP 200

HTTP 200. Reachable by anyone on the internet who runs a port scan, which is to say, reachable by every automated scanner on the internet within hours. Do not leave this running. Tear it down immediately:

sudo docker compose -f insecure/docker-compose.yml down

One honest note on scope: I am describing the CVE-2026-65589 class accurately from the advisory. I did not stage a live key-leak on camera, and you should not either. The point is to recognize the misconfiguration and fix it, not to build a working attack.

The 4 fixes

Fix 1: patch, and stop publishing 5678

Two changes in the hardened Compose file:

services:
  n8n:
    image: n8nio/n8n:1.123.64    # patched: fixes CVE-2026-65589
    expose:
      - "5678"                   # internal only, NOT published to the host

expose instead of ports is the part people miss. It makes n8n reachable only to other containers on the Docker network. Nothing lands on the host's public interface. Combine that with the AWS security group (next), and 5678 is closed at two layers.

Fix 2: put secrets in AWS Secrets Manager

A .env file on disk is one cat away from leaking. So the encryption key and the database password never touch the repo and never sit in plaintext on the box. Generate them, store them in Secrets Manager, inject them at runtime:

# run once, from an admin session
ENC_KEY="$(openssl rand -hex 32)"
aws secretsmanager create-secret \
  --name "n8n/encryption-key" \
  --secret-string "$ENC_KEY" \
  --region us-east-1

The N8N_ENCRYPTION_KEY matters more than it looks. Without it, n8n derives a default key, which means your stored credentials are effectively unencrypted. Set it, and back it up separately. Lose that key and you lose access to every credential n8n has stored. There is no recovery.

(The repo's create-secrets.sh wraps this in a create-secret || put-secret-value fallback so it is safe to re-run, which is why the instructions ask for both CreateSecret and PutSecretValue on the admin session.)

At deploy time, a small script pulls both secrets into the shell environment (not a file) and starts the stack:

export N8N_ENCRYPTION_KEY="$(aws secretsmanager get-secret-value \
  --secret-id n8n/encryption-key --query SecretString --output text --region us-east-1)"

Fix 3: a least-privilege IAM role

Here is where AWS earns its keep. The EC2 instance reads those two secrets through an IAM role that can read only those two secrets. No static credentials on the box. No broad secretsmanager:*. Just this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOnlyN8nSecretsFromSecretsManager",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:n8n/encryption-key-*",
        "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:n8n/postgres-password-*"
      ]
    }
  ]
}

If the box is ever compromised, the blast radius from this role is two secrets, read-only. Compare that to a .env file, which hands over everything the moment someone gets shell access. The trailing -* matches the random suffix Secrets Manager appends to secret ARNs.

Fix 4: TLS and an IP-locked editor with Caddy

n8n publishes no ports. Only Caddy publishes 80 and 443, terminates TLS with an automatic Let's Encrypt certificate, and splits traffic:

n8n.example.com {
    # Webhooks must stay public so external services can call them.
    handle /webhook/* {
        reverse_proxy n8n:5678
    }

    # Everything else (editor UI + REST API) is restricted to your admin IP.
    handle {
        @blocked not remote_ip 203.0.113.10
        respond @blocked "Forbidden" 403
        reverse_proxy n8n:5678
    }
}

Your editor and REST API are now reachable only from your IP. Your webhooks stay public, because they have to. Everyone else gets a 403.

Step final: prove it

Do not trust that hardening worked. Verify it from outside the instance:

curl -s -m 8 -o /dev/null -w 'port 5678 -> HTTP %{http_code}\n' http://<PUBLIC_IP>:5678

Output:

port 5678 -> HTTP 000

HTTP 000 means curl could not even open a connection: refused or timed out. That is exactly what you want. The door that was wide open is now a wall.

Why "on AWS" beats a generic VPS guide

Every generic hardening guide tells you to install UFW, edit an nginx config, and put secrets in a chmod 600 file. That works. But AWS gives you managed primitives that do the same jobs with less to get wrong:

Security group is a managed, default-deny firewall. You allow 22 (locked to your IP), 80, and 443. You never open 5678. It is stateful and it is not a config file you can fat-finger.
Secrets Manager is a managed secret store with rotation available as a managed feature. No plaintext env file on disk to leak.
IAM least-privilege role scopes access with zero static credentials. If n8n later needs to call an AWS service, you give it a tighter role, separate from any human's keys.

That last point is the setup for where this series goes next. Once your secrets and identity live in AWS, giving n8n a scoped execution role to call something like Amazon Bedrock is a small, safe step instead of pasting an API key into a header (the exact thing CVE-2026-65589 punished).

The checklist to keep

If you remember nothing else, work down this list:

Patch first. n8n >= 1.123.64.
AWS network. Security group allows only 22 (your IP), 80, 443. Never 5678.
Server/OS. SSH keys only, no root or password login, unattended security updates.
n8n config. Real N8N_ENCRYPTION_KEY, disable public signup, WEBHOOK_URL=https://..., session timeout, prune executions, telemetry off.
Reverse proxy and TLS. No published n8n ports, TLS at Caddy, editor IP-restricted, only /webhook/* public.
Secrets. Encryption key and DB password in Secrets Manager, read via a least-privilege IAM role.
Credentials inside n8n. Use built-in credential objects, not custom LLM headers. Prefer OAuth. Rotate.
Monitoring. Review the Executions tab for odd sources, add an external uptime check, centralize logs.

What I would tell my friend now

The weekend tutorial got him running. It just skipped the part about making him safe to leave running. Those are different jobs, and the second one is the one that matters once real credentials are involved.

It took me an afternoon to build the hardened version. Closing 5678, patching the image, moving two secrets into Secrets Manager, scoping an IAM role, and putting Caddy in front. Compared to the cost of leaking a Stripe key, that is the cheapest afternoon you will ever spend.

The full repo has the insecure and hardened Compose files, the IAM policy, the Caddyfile, the scripts, and a hardening checklist with the reasoning behind every item: github.com/simplynadaf/secure-n8n-on-aws.

This is the hardening episode of a series on running n8n properly on AWS. The previous episode got n8n running from an empty box to first login; this one locked it down. Next up: running a real AI agent inside n8n with Amazon Bedrock, using that scoped IAM role instead of an OpenAI key.

Question for you: if you self-host n8n, is your port 5678 published to the host right now? Go check your Compose file. I will wait. Tell me what you find in the comments.

I Built an AI That Reads Eviction Notices and Refuses to Lie to You

Sarvar Nadaf — Mon, 05 Oct 2026 13:27:11 +0000

This project was built for the AWS Zero to Shipped hackathon. Category: Social Good. Lane: Community. I built the whole thing over a weekend with an AI coding agent connected to AWS.

I spent the weekend building one small app for the AWS Zero to Shipped hackathon. It reads an eviction notice and tells you the one date you cannot miss. Here is how the idea started, what I used, who it is for, and the one rule that shaped every line of code.

How I got the idea

I was not planning to build this.

My first idea was an "AWS account report card", a tool that grades your account on security and cost. I researched it for a few hours and killed it. Prowler, 6WAF, and AWS Trusted Advisor already do that, and the judges are AWS Solutions Architects who know those tools cold. Low originality, bad odds.

So I scraped the field. Over 240 projects were already submitted. The crowd was DevOps tools and student study assistants. The gap was something else: a trust-first app for people who get hit by a scary document and have no one to help them read it.

Then I found the number that would not leave me alone. Roughly 1 in 4 eviction cases end in a default judgment (MLRI, The Default Project). The tenant does not lose on the merits. They lose because they never responded in time. The same research says those tenants are more likely to have a disability, to speak a language other than English, and to have little access to technology.

The Furman Center summed it up in one line: "half the battle is just showing up."

So the job was simpler than a lawyer-bot. Make one fact impossible to miss: the deadline on the letter.

What it does

You paste the eviction notice you received. DueDate gives you back, in your language:

The one deadline you cannot miss.
A dated checklist of what to do before it.
Your rights, but only the ones your notice actually supports.
An honesty panel: "what your notice does NOT say", so nothing is assumed.
Links to real free legal aid near you.

No login. No account. It runs on AWS and it is read-only.

The one rule that shaped everything

An eviction notice is read by someone who often cannot check whether the answer is right. If my app says "you have until January 12" and the real date is January 9, I have not helped. I have hurt them.

So the whole system has one rule: the AI model is never allowed to decide a fact.

Here is the opinion that drove it, and I think most document-reading AI apps get this backwards. The usual pattern hands the whole document to the model and asks it to "extract the deadline and explain it." That makes the model the source of truth for a fact a frightened person cannot check. Wrong job for the model. The model should own language, not facts. Code should own facts.

Dates, dollar amounts, the notice type, the computed deadline. All of that comes from plain code, not from a language model. The model only translates and simplifies the facts after the code has already found them. If the code cannot find a fact, the model is not allowed to fill the gap. It has to say the notice does not state it, and point to free help instead.

That rule is the product. Everything else is how I enforced it on AWS.

One honest limit, because the claim is strong: Textract can still misread a photo of a crumpled notice. So DueDate shows the exact line it read and its confidence, and lets you click any answer to see the line behind it. It cannot invent a fact, and when it is unsure of what it read, it shows you so you can check against the paper in your hand.

How it works

Four steps. Only one of them is allowed to be creative.

Browser (CloudFront + S3)
   |
   |  POST { text, language }   (HTTPS, no login)
   v
Lambda Function URL
   |
   |  1. Amazon Textract       read the notice line by line
   |  2. deterministic engine  find the facts (code, no model)
   |  3. Amazon Bedrock        narrate + translate (guardrailed)
   |  4. legal-aid directory   route to real free help
   v
JSON: cited facts + plain-language summary + dated checklist

Textract reads the notice and gives each line an id (L1, L2, L3) with a confidence score. Those ids are the citation anchors. No id, no citation, nothing on screen.

The engine is code. It classifies the notice by matching rules, pulls out the amount and the parties, and computes the deadline. The tricky case is a notice that says "within THREE (3) days" and "dated January 6, 2026". There is no deadline date on that paper. A lazy build asks the model to guess it. Mine reads the day count from one line, the notice date from another, adds the days by a documented rule, and labels the result as computed with the rule shown on screen. If either input is missing, it produces no date at all.

Bedrock only narrates. I used Nova 2 Lite, cheap and multilingual. It rewrites the facts in plain language at about a sixth-grade reading level, in English or Spanish. After it responds, I throw away any sentence whose fact id is not real. A prompt is a request. A filter is a guarantee.

The last step is not AI. It is a small directory of real free legal aid, picked by the state read off the notice.

I wrote tests that try to make it lie, and it refuses:

def test_R7_never_invents_a_date_absent_from_the_notice():
    vague = [L(1, "NOTICE TO PAY RENT OR QUIT"),
             L(2, "Rent is overdue. Please contact the office.")]
    keys = {f.key for f in extract_facts(vague)}
    assert "deadline" not in keys   # no date in the paper, no date on screen

Eight of these pass with no AWS calls, so anyone can clone the repo and prove the guarantee in seconds:

$ python3 tests/test_engine.py
PASS  test_R10_refuses_when_only_a_day_count_but_no_start_date
PASS  test_R7_never_invents_a_date_absent_from_the_notice
PASS  test_analyze_surfaces_missing_fields
PASS  test_classifies_pay_or_quit_with_citation
PASS  test_computes_deadline_from_days_and_marks_it_computed
PASS  test_every_fact_has_a_citation_or_is_explicitly_absent
PASS  test_extracts_amount_only_when_present_and_cited
PASS  test_unknown_notice_is_honest_not_guessed

8/8 passed

The AWS services I used

The whole thing is serverless and scales to zero.

Amazon CloudFront + Amazon S3 serve one static HTML page from a private bucket (Origin Access Control, so the bucket is never public).
AWS Lambda (Function URL) is the only compute. No API Gateway, no servers. The Function URL owns CORS, so there is one source of truth for headers.
Amazon Textract reads the notice line by line.
Amazon Bedrock (Nova 2 Lite) handles plain language and translation, under guardrails.
AWS IAM keeps the Lambda role to exactly textract:DetectDocumentText and bedrock:InvokeModel, nothing else.
AWS CDK defines all of it. One cdk deploy prints the live URL.

At idle it costs nothing. Each analysis is one Textract page read ($0.0015 at $1.50 per 1,000 pages) plus one short Nova 2 Lite call (a few hundred tokens in and out, well under a tenth of a cent at $0.30 per 1M input and $2.50 per 1M output). Call it under half a cent per notice. A demo run a few hundred times stays under a dollar a month. A tool for people with no money cannot be expensive to keep alive.

Who it is for

Tenants who get an eviction notice and have no lawyer. People who read English as a second language. People who miss the response window not because they have no case, but because the letter was confusing and the clock ran out.

The scale is not small. Landlords file about 3.6 million eviction cases in the US in a typical year (Eviction Lab), and in the ones that end in a default judgment, the tenant lost without ever being heard. The single cheapest intervention is making the deadline impossible to miss. That is the whole job.

It is not legal advice, and it says so plainly. It explains your notice, cites your own paper, and hands you to free legal aid. That boundary is the point.

Two bugs that cost me real time

Bedrock threw AccessDeniedException the moment the Lambda tried to narrate, even though it worked from my laptop. Nova 2 Lite's us. model id is a cross-region inference profile, so the IAM policy needed the inference-profile ARN and the foundation-model ARNs across regions, not just one model ARN in us-east-1.

Then the app loaded but every call failed with Access-Control-Allow-Origin header contains multiple values. I was setting CORS in the Lambda and the Function URL was also adding it. Two sources, duplicated header, browser rejects it. I deleted the manual headers. One source of truth fixed it.

Neither bug is exotic. Both only show up when you actually ship instead of demoing on localhost.

Try it and see the code

The live app, no install and no account:

d3pjdlu332prje.cloudfront.net

The full source, the CDK, the EARS spec, and the tests:

simplynadaf / duedate

Understand your eviction notice before the deadline. Upload the notice, get the one date you cannot miss, what to do, and your rights, cited to your own paper and in your language. Built on AWS (Textract + Bedrock + Lambda + CloudFront). Not legal advice. AWS Zero to Shipped.

📬 DueDate: Understand Your Eviction Notice Before the Deadline (2026)

Paste the eviction notice you received and get, in your language, the one date you cannot miss, exactly what to do before it, and your rights - with every statement pinned to the exact line of your own notice, and a hard refuse-to-invent rule so it never tells a frightened person something their paper does not actually say.

AWS Zero to Shipped · Category: #social-good · Lane: #community

▶ Watch the 2-minute demo

Important

DueDate is not legal advice and not a lawyer. It explains what your notice says, cites the exact line it read, and points you to real free legal aid. It never states a right or a step your notice does not actually contain. If it cannot ground an answer in your paper, it says so and routes you to free help. See The Honest Take.

…

View on GitHub

The Builder Center project, with the full write-up and the agent-to-AWS proof:

AWS Builder Center

Connect with builders who understand your journey. Share solutions, influence AWS product development, and access useful content that accelerates your growth. Your community starts here.

builder.aws.com

What I would tell you to copy

The pattern is not specific to eviction notices. It works for any case where an AI reads a document for someone who cannot check the answer: medical bills, debt letters, benefit terminations, immigration notices.

Split the work. Let code own the facts. Let the model own the language. Put a filter between them that drops anything ungrounded. Then write a test that tries to make it lie, and do not ship until that test passes.

The result is an AI you can hand to a frightened person, because the worst thing it can do is say "I do not know, here is who can help."

If you found this useful, I would love a like, a comment, and a share. What document would you want it to read next: medical bills, debt letters, or benefit-termination notices? Tell me in the comments.

Which AWS limit is actually current? An agent that proves it, 32 vs 5 vs 16

Sarvar Nadaf — Thu, 01 Oct 2026 20:38:01 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Which AWS quota is actually current? You copied a default vCPU limit off the AWS docs, shipped it, and it broke in production. I have done this. The number was stale, and nothing warned me.

Here is what makes it nasty. For one AWS fact, three official pages can each give you a different number. An old User Guide says one thing. The Service Quotas console says another. A pricing page says a third. All look official. None of them tells you which is live today.

So I built an agent that answers "which AWS value is current?" and then proves it. It reads typed facts from a Sanity Context Knowledge Base, picks the winner with a deterministic rule the model is not allowed to override, and shows both the current value and the one it replaced, each with its source. Then it does the part most content agents skip: it calls the live AWS API (read-only) to check whether even the reconciled record is still true. On the EC2 vCPU quota it turns up three different numbers, docs 32, record 5, live 16, and puts all three on the screen.

Stack: Amazon Nova Pro on Bedrock, Strands Agents (Python), a Sanity Context Knowledge Base read over the hosted Context MCP, and a read-only live AWS cross-check.

Why a keyword search returns the wrong AWS value

The challenge sets a hard bar: "the strongest submissions show an agent that only works because the content was structured. If a keyword search would have gotten you the same answer, aim higher."

Fair. So I built a fact where keyword search gets it wrong on purpose. Real one too: the max IOPS per volume for an EBS general purpose SSD. The current number is 80,000 (gp3, from the EBS console). The old number is 16,000, the legacy gp2 ceiling, still sitting in an older SSD guide. That old guide repeats the exact phrase you would type into a search box: "maximum IOPS per volume for a general purpose SSD EBS volume." It is stale and wordy, so keyword scoring loves it.

Watch it pick the wrong answer. Same content, real TF-IDF, the question a person would actually ask:

Keyword/TF-IDF baseline for: "maximum IOPS per volume general purpose SSD"

  0.8626  EBS/limit: Maximum IOPS per volume for a general purpose SSD ... = 16000   <-- WRONG (old)
  0.2026  EBS/limit: Maximum provisioned IOPS per gp3 volume            = 80000   <-- right, ranked lower
  0.0218  S3/price:  S3 Standard storage, first 50 TB / month           = 0.023
  0.0185  EC2/quota: Running On-Demand Standard ...                      = 5
  ...

The stale 16,000 wins by a mile. Keyword search hands you the wrong number and sounds sure of it.

Demo

Run the credential-free core yourself in under a minute (public dataset over anonymous GROQ, local reconcile, read-only live AWS check, no model or token needed):

# 1. clone + install
git clone https://github.com/simplynadaf/aws-source-of-truth-agent
cd aws-source-of-truth-agent
pip install -r requirements.txt

# 2. the credential-free path (public dataset + local reconcile + live AWS check)
python -m agent.reconcile_offline --service EBS --type limit --region us-east-1
# -> Verdict: 80000 IOPS (serviceQuotasConsole wins over officialDocs)

# 3. the keyword control that returns the WRONG answer:
python -m agent.baseline "maximum IOPS per volume general purpose SSD"

For the full LLM agent through the Context MCP, copy .env.example to .env and add a Bedrock region plus a Sanity Context Viewer token (the README walks through it). There is a JSON mode too, for piping into something else:

python -m agent.ask --json "What is the maximum IOPS per volume for a gp3 EBS volume?"

The real run, Nova Pro, us-east-1, read-only throughout:

Fact	Reconciled (from the record)	Superseded	Live AWS	Result
EC2 On-Demand Standard vCPU quota	5 (console)	32 (old user guide)	16	DRIFT
EBS gp3 max IOPS per volume	80,000 (console)	16,000 (old SSD guide)	unavailable	trusted (no such live quota)
S3 Standard $/GB-mo	0.023 (pricing page)	0.021 (stale blog)	0.023	AGREE
RDS PostgreSQL oldest major	13 (release notes)	11 (old tutorial)	11	DRIFT
Lambda concurrent executions	1000 (dev guide)	n/a	unavailable	trusted
Graviton4 (R8g) availability	Available (instance types)	n/a	Available	AGREE

Look at the EC2 row. Docs say 32. The reconciled record says 5. The live account says 16. Three numbers for one quota, and the agent shows all three with their sources instead of picking one and hoping. The drift is not a failure. It is the honest answer: this is the current record, and here is where reality has already moved past it.

Code

simplynadaf / aws-source-of-truth-agent

🌊 AWS Source of Truth

When your AWS docs, pricing page, and Service Quotas console disagree, an agent that knows which one is telling the truth, then checks the live API to see if even that record has drifted.

Sanity project id: 0q5ohtvv · ⭐ If a stale AWS number has ever bitten you in production, give this a star.

The Problem • Why Search Fails • How Structure Fixes It • The Twist • Getting Started • FAQ

📖 Table of Contents

🤔 The Problem

You copied a limit straight out of the AWS docs…

View on GitHub

Full source, plus the credential-free path judges can run with no token.

How I Used Sanity

What I pointed Sanity Context at: my own Sanity content. Every fact is a typed awsFact document in the production dataset, not a wall of prose:

{
  "_type": "awsFact",
  "service": "EBS", "factType": "limit", "region": "us-east-1",
  "key": "Maximum provisioned IOPS per gp3 volume",
  "currentValue": "80000", "unit": "IOPS",
  "effectiveDate": "2026-01-15",
  "source": { "name": "EBS gp3 volume limits (Service Quotas / EBS console)",
              "kind": "serviceQuotasConsole", "url": "..." }
}
// ...plus a separate record for the old 16,000 (kind: officialDocs, 2020).

source.kind and effectiveDate are real fields, so the rule that picks the winner is dull and readable:

highest source precedence wins (console and pricing page beat changelog, which beats official docs, which beats a blog), and the newest effectiveDate breaks ties.

Prose cannot do this. The schema is the whole trick.

The Knowledge Base: a Sanity Context Knowledge Base indexes these facts. The build reads them ahead of time and writes short cited entries. Where two records fight, the entry keeps both numbers and both sources next to each other.

Which Context tools the agent used: the agent pulls facts over the hosted Context MCP with knowledge_base_search (ranked lookup) then knowledge_base_read (full entry). That is the proof the answer came from Sanity and not from the model's memory. (It can also fall back to anonymous GROQ over the public dataset for the no-token path.)

What the agent does with what it retrieves: Nova Pro runs the tools and writes the sentence, but it does not get to decide the number. A plain function reconciles the value, and a guard checks the model's answer against it. If they disagree, the guard throws out the model's prose and ships the deterministic answer instead. On one EBS run Nova muddled its own wording, the guard caught it, and swapped in the correct answer with no help from me. The model cannot invent the number even if it tries.

The tool trail proves the path through Sanity on every question:

- [Sanity Context MCP] knowledge_base_search('EC2 quota us-east-1 ...') -> top='ec2/quotas_and_limits'; knowledge_base_read(['ec2/quotas_and_limits'])
- fetch_candidate_facts(service='EC2', fact_type='quota', region='us-east-1') -> 2 rows
- reconcile_facts(n=2) -> current=5
- verify_live(EC2/quota) -> drift

Then: what if even the reconciled record is stale? (live AWS drift)

Reconciling the sources gives you the best answer the documents can offer. But documents rot. So the agent does one more thing a pure content agent will not: it calls the live AWS API, read-only, and asks whether the reconciled value is still true right now. Service Quotas, the Price List API, EC2, RDS. The results are in the Demo table above. The live layer is additive and clearly labelled; it is not part of Sanity and it is account-specific.

What didn't work (the honest part)

I planned to screenshot a resolved conflict in the dashboard. The build never gave me one. The Knowledge Base read the typed records, reconciled the 32-vs-5 fight into a clean entry with both numbers cited, and left the Issues queue empty. That is the KB doing its job, but it means there is no "I resolved an Issue" screenshot to show, so I am not claiming one.
Nova dropped a tool argument on me. fetch_candidate_facts returned a row, then Nova passed an empty string to reconcile_facts, which saw zero facts. The guard fail-closed to "Not verified" instead of guessing. I fixed it by caching the last real fetch server-side, so the facts always come from Sanity, never from whatever the model relayed.
A wrong factType guess used to lose a real fact. Nova guessed instanceType when the field value was regionalAvailability, the typed filter matched nothing, and the fact vanished. Now a zero-row typed fetch retries on service and region alone.
The live check is not part of Sanity, and it is account-specific. A judge on a different AWS account will see different live numbers. That layer is additive and labelled as such. Two facts (EBS max IOPS, Lambda) have no matching Service Quotas entry, so the agent says "unavailable" rather than fake a check. The seed is labelled demo data at the top.

Reusing the pattern beyond AWS

Drop the AWS parts and the shape is generic: type your sources, index them into a Knowledge Base, reconcile by precedence and date, guard the answer so the model cannot fake the winner, and check a live system of record when one exists. It fits API version support, pricing, compliance clauses, legal terms. Anywhere the real question is "which version is current?"

The thing I took away: the win here was not a smarter model. It was structured content, plus the nerve to say "the record says 5, but reality says 16."

Sanity Project Details

Sanity project ID: 0q5ohtvv
Dataset: production (public)
Public dataset query (no token): https://0q5ohtvv.api.sanity.io/v2023-05-03/data/query/production?query=*%5B_type%3D%3D%22awsFact%22%5D

If your docs and your console have ever disagreed with each other, tell me which number bit you.

AI Agent Governance on AWS: Block Agents, Prove EU AI Act Compliance

Sarvar Nadaf — Tue, 29 Sep 2026 13:27:38 +0000

Every fact here is verified against an actual run (Amazon Nova Pro us-east-1 on-demand, Strands Agents, and traccia 0.1.29 $0.0008 per 1K input and $0.0032 per 1K output; EU AI Act Annex III credit-scoring obligations apply 2 December 2027). The demo system is SYNTHETIC throughout.

Observability told me the agent approved a loan it should have declined. It showed me the run that leaked an applicant's email into the logs. Then it did nothing, because watching is the only thing it does.

Part 1 of this series put a real observability layer under a multi-agent AWS crew and caught a run that looked healthy while it silently overspent. This part is about the two things observability cannot do: stop the bad run, and hand a regulator the paperwork afterward. So I built a synthetic multi-agent loan-decision crew on Amazon Bedrock, wired it to Traccia, and made four things happen on real infrastructure: a prompt injection hard-blocked before any model call, applicant PII redacted across every sub-agent's trace, EU AI Act evidence stamped on every span, and the platform itself denying a runaway agent with a real, cited decision.

Then it did not work. Two of the policies I wrote denied nothing, and not because I misconfigured them: two of the three policy types structurally had no signal to match in this crew. Working out why is the most useful part of this article, because it is the part nobody writes down.

How I tested this: the build, the afternoon the platform refused to block, the two real bugs I fixed, and the critique near the end are all mine. Everything on screen is real (real Nova Pro calls, real governance decisions, real exported evidence), the crew is synthetic on purpose, and it produces evidence, not compliance - a distinction that runs through the whole piece. The $0 SDK layer needs no account. After Part 1, the 14-day free trial on the premium platform tier (@govern enforcement and the Governance Hub) is what pushed me to dig in and pull out something meaningful for the community, so everything here is reproducible at no cost.

This is for people already running agents on AWS (Strands, CrewAI, LangGraph, or your own loop on Bedrock) who have heard "EU AI Act" enough times to be nervous, and who want runtime guardrails plus audit evidence without rewriting the app around a compliance framework. Where the governance vocabulary is new, I define it the first time it shows up.

If you have two minutes, skip to the platform payoff, and why it did not work at first. The build-up matters, but that section is the point.

Contents

Why agent governance is a different problem
The crew, and why it is synthetic
Two layers, and only one needs a key
Prerequisites
The three $0 governance beats
The platform payoff, and why it did not work at first
The evidence a regulator asks for
Real-time enforcement vs scheduled detection
The timing that makes this worth doing now
An honest take on Traccia governance
Honest caveats
FAQ

Why agent governance is a different problem

Observability answers "what did the agent do, and what did it cost." Governance answers a harder pair: "can I stop it from doing the wrong thing, and can I prove what happened." Those are not the same layer, and a trace alone gives you only the first.

For a deterministic service, governance is mostly access control and input validation. Both are loud, and both are enforced at the edge before any work starts. An AI agent breaks that. It decides its own control flow, so a block has to interrupt a reasoning loop that is already running. It calls tools in an order you did not hardcode, so a runaway is a genuine failure mode, not a hypothetical.

And when it is a multi-agent crew, the failure surface multiplies three ways. A block has to stop the whole crew, not one sub-agent. Redaction has to cover every span in the tree, not just the entry point. And the agent that actually burns the budget may be three delegations deep from the one you wrapped.

There is regulation attached to this now, not just good practice. The EU AI Act names credit scoring as a high-risk use (Annex III, point 5(b)) and attaches concrete obligations: record-keeping (Article 12), transparency to deployers (Article 13), human oversight (Article 14), post-market incident reporting (Articles 72 and 73), and for some deployers a Fundamental Rights Impact Assessment (Article 27). Article 50 transparency, telling a person they are dealing with AI, is already enforceable. So "governance" here is concrete: a specific list of things a specific regulator will ask you to show.

A quick vocabulary anchor, because the rest leans on it. A guardrail inspects an input, an output, or a tool call and decides whether it is allowed. A policy is a rule the platform enforces at runtime (a spend cap, a tool-call cap, an allowed-model list). Enforcement is what happens when a policy matches: the call is denied. Evidence is the durable record of all of it. The whole article is putting the right guardrails and policies on the right agents, then reading the evidence back.

The crew, and why it is synthetic

A supervisor loan officer delegates to three specialists, using the AWS Strands agents-as-tools pattern, each calling Amazon Nova Pro (amazon.nova-pro-v1:0).

Intake parses the applicant's free text into an id and a requested amount. PII lands here first, which is why redaction gets tested here (though it covers every span).
Credit & Risk (credit-risk) calls a mock credit_score tool and a region-restricted pull_bureau_report. This is the agent that makes the tool calls. Remember it, because it is the whole plot of the platform-block section.
Policy applies a deterministic, explainable lending rule (income-to-debt and requested-amount-to-income, never a protected attribute) and returns approve, refer, or decline.

Two more guardrails sit at the boundary alongside the injection block: an output-validation check that rejects any recommendation making an absolute approval claim, and a fairness check that rejects a decision if a protected attribute drove the score. Fairness carries real weight here, because it is the whole reason credit scoring is high-risk under the EU AI Act. So the crew asserts on the scoring signals and blocks if a protected attribute ever appears.

Multi-agent earns its place too. It makes governance harder in exactly the ways that matter, and a single-agent demo hides all three problems: the block has to stop the whole crew, the redaction has to span the whole tree, and a crew can genuinely run away, so a loop cap is believable rather than contrived.

Now the honesty that has to be up front: the credit score is a mock. There is no real bureau, no real model decision, no real applicants. The scoring is a deterministic heuristic that uses only legitimate financial signals and, by construction, never touches a protected attribute (there is a unit test that asserts exactly that). A real high-risk credit system needs a legal conformity assessment; this project produces the evidence substrate such a system would generate, not the compliance itself. I label everything "synthetic" on screen for the same reason: a governance demo that pretends to be a real lender is the opposite of trustworthy.

Two layers, and only one needs a key

Traccia's governance splits into two layers, and getting the split right is most of this article.

Layer	Needs a key?	What it does
SDK / in-process	No ($0, offline)	Guardrail detection, an explicit hard-block, PII redaction, EU AI Act stamping, transparency evidence, an integrity hash. All run as OpenTelemetry span processors before export.
Platform	Yes (`TRACCIA_API_KEY`)	`@govern` networked enforcement (Spend Cap, Model Boundary, Loop Cap), plus the Governance Hub: system registry, human review, incidents, evidence packs, FRIA.

@observe gives you observability on any OTLP backend, no key. @govern adds runtime policy enforcement and needs the platform. The load-bearing point, stated plainly because it is easy to blur: the local hard-block and PII redaction run in-process regardless of the key. The key does not power the block. What the key adds is the networked enforcement and the audit hub. So the honest framing is "the SDK governs locally; the platform enforces and records," not "you have to pay to block anything."

One more piece of honesty about the SDK layer, because I read the source to be sure: a Traccia guardrail detects, it does not enforce. The three-tier engine writes findings to spans. The "block" in Beat 1 below is my own code raising on a guardrail result, not the guardrail engine stopping anything. Keeping that distinction straight is the difference between describing the tool accurately and overselling it.

Prerequisites

Three things, and only the third is for the platform payoff.

Python 3.10+ and the SDKs (strands-agents, strands-agents-tools, traccia, boto3). The repo pins tested versions in requirements.txt (traccia 0.1.29, strands-agents 1.56.0, boto3 1.43.99).
AWS credentials with bedrock:InvokeModel on Nova Pro. The repo ships a least-privilege policy at iam/bedrock-invoke-policy.json that allows InvokeModel on amazon.nova-pro-v1:0 only. You also enable Nova Pro once in the Bedrock console under Model access (the console grant is not an IAM permission, so you need both). "$0 Traccia" still means real Bedrock calls, so this is required to run the crew at all.
A Traccia platform key (TRACCIA_API_KEY) only for the @govern enforcement and the Governance Hub. Everything else runs without it.

Getting it running is clone, set up, and go:

git clone https://github.com/simplynadaf/ai-agent-governance-aws.git
cd ai-agent-governance-aws
ls -a                # no .env here - only the shipped .env with keys commented out
cat .env             # proves the $0 beats need no key
make setup           # venv + pinned deps + git hooks
make test            # 10/10 unit tests: pure logic, no AWS, no key

A fresh clone runs the $0 beats with no key. The tracked .env ships with the key lines commented out, so cat .env on camera is the proof that "$0, no key" is real rather than a footnote. Your real key goes in that same file locally; a pre-commit hook blocks any commit that contains an uncommented key.

The three $0 governance beats

These run with nothing but the install and AWS credentials, writing traces to a local traces_gov.jsonl file.

Beat 1: hard-block a prompt injection

A detector decorated with @observe(as_type="guardrail") returns a bool, and the SDK auto-sets guardrail.triggered from it. Your own code raises on that. The crew stops before the supervisor ever calls the model.

@observe(as_type="guardrail", attributes={"guardrail.category": "prompt_injection",
                                           "guardrail.enforcement_mode": "block"})
def injection_check(text: str) -> bool:
    return any(kw in text.lower() for kw in INJECTION_KEYWORDS)

# in the crew entrypoint:
if injection_check(text):
    raise BlockedByGuardrail("Prompt injection detected. Crew blocked before any model call.")

The proof is not the log line, it is the trace: the blocked run has zero sub-agent spans. Intake, Credit & Risk, and Policy never ran. I verified this by counting spans in the exported file: the blocked trace contains only the injection check and the crew root. A block that truly stops a multi-agent crew leaves nothing downstream, and the trace is where you confirm that instead of trusting a print statement.

One naming detail worth being honest about: the SDK's real signal is guardrail.triggered (auto-set from the guardrail's bool return). The crew-stopped flag demo.crew.blocked is my own attribute, not an SDK one. I keep the two distinct so nobody thinks the SDK is doing something it is not.

Beat 2: redact PII across every sub-agent

init(redact_pii=True) adds a regex processor that masks email, phone, and SSN to [REDACTED_EMAIL], [REDACTED_PHONE], [REDACTED_SSN] on every span in the tree, before export. I verified it the only way that counts: searched all span attributes for the literal test email and SSN. Zero matches, across the whole multi-agent tree, not just the intake entry point.

Honest limit, out loud: this is best-effort regex, not ML NER. It catches labeled patterns like emails and SSNs; it will miss names, addresses, and unlabeled ids, and it can over-redact. In a real system you say that plainly and layer a real PII classifier on top.

Beat 3: stamp EU AI Act evidence

init(compliance={"frameworks": ["eu_ai_act"], "risk_tier": "high"}) puts eu_ai_act.risk_tier=high on every span automatically. Credit scoring is a textbook Annex III point 5(b) case, so I stamp eu_ai_act.annex_iii_category="5b_creditworthiness" by hand. That distinction is deliberate and verified against the SDK source: the SDK auto-writes risk_tier, but the Annex III key is a reserved constant that is never auto-populated, so if you want it, you set it. disclosure() records Article 50 transparency, and every span carries an automatic governance.integrity_hash.

A plain-language report reads straight from the trace file, so a compliance reviewer never has to open raw span JSON:

AGENTS THAT RAN: loan-prescreen, intake, credit-risk, policy
GUARDRAIL TIERS:  A explicit YES · B provider-native "not observed" · C heuristic YES
EU AI ACT:        risk_tier N/N spans · annex_iii YES · Art.50 YES · integrity_hash YES
PII LEAK CHECK:   raw test PII found in 0 places  (must be 0)

Here is the honesty note most tutorials skip, and my verify_trace.py now prints it. Traccia's guardrail engine has three detection tiers: A (explicit, your @observe guardrails), B (provider-native, fires when the model itself returns a safety or stop signal), and C (heuristic, fires when a tool span errors with a denial keyword). I show A and C firing (the injection block is A; the region-restricted bureau pull that errors for an EU applicant is C). Tier B does not fire on a clean run, and that is expected, not a gap. So the honest claim is "A and C demonstrated; B fires on the model's own safety signals," never "all three fire."

The platform payoff, and why it did not work at first

Here is the afternoon I lost, and the lessons that came out of it.

I set @govern(fail_open=False) on the crew entrypoint, put the key in .env, and ran it. Result: ALLOWED. No block.

Before any policy could even be evaluated, there was a setup gotcha to clear. @govern needs a tracing endpoint, not just a key. Without TRACCIA_ENDPOINT set, the SDK logs "Traccia endpoint not found," silently skips enforcement, and returns ALLOWED. I set TRACCIA_ENDPOINT=https://api.traccia.ai/v2/traces, and the warning went away. Now enforcement was actually running, so the real diagnosis could start. It still returned ALLOWED.

Then I created a Model Boundary policy that only permitted gpt-4o (the crew uses Nova Pro), scoped it to the crew agent loan-prescreen, set it to Block, and activated it. Ran again. Still ALLOWED. That was the first policy that should have blocked and did not.

The Decision Log in the dashboard is what cracked it. It groups per-call policy checks by trace, and every row said the same two things: No Match, and the check was attributed to agent credit-risk, not loan-prescreen.

The Decision Log is the screen that cracked this for me. Each row is a per-call check tied to a trace. The No Match rows told me why my first policies failed; the Denied row (Loop Cap, credit-risk, "tool calls 2 exceed 1") is the block that finally worked. Every check is attributed to credit-risk, not the @govern entrypoint.

Two lessons fell out of that, and they are the content the docs and the reference paper do not have.

Lesson 1: scope the policy to the agent that actually makes the calls. My crew wraps each sub-agent in a Strands run_identity, so the per-call policy check runs under the sub-agent's identity (credit-risk), not the @govern entrypoint's (loan-prescreen). A policy scoped to the entrypoint never matches, because no call is ever attributed to it. Scope to the sub-agent, or org-wide.

Lesson 2: match the policy type to what the spans actually carry. I re-scoped a Spend Cap with a $0 budget to credit-risk. Still No Match. The two span checks in each trace are the two tool calls (credit_score and pull_bureau_report), and their cost is about zero, so a spend rule has nothing to trip. A Model Boundary needs an auto-patched LLM client to inspect the model in use, and a Strands BedrockModel is not one of the clients Traccia auto-patches. So neither policy type can match this crew, for two different reasons.

A Loop Cap counts tool calls per run. The crew makes exactly two. So I set Max Tool Calls Per Run to 1, Block, scoped to credit-risk, activated it, and ran:

PLATFORM BLOCK - AgentBlockedError:
   reasons             : ['tool calls 2 exceed 1']
   decision_id         : b5a3a3e4-6b62-4c81-9c07-358773735d44   (varies per run)
   remaining_budget_usd: None

The active Loop Cap: Max Tool Calls Per Run = 1, scoped to credit-risk, enforcement Block. This is the only one of the three policy types whose signal actually exists in this crew's spans.

That is a real, networked deny with a reason and a decision id (the id is fresh on every run). A subtle but important detail I verified in the SDK source: those populated fields (reasons, a real decision_id) come from the per-call deny path. A block from the lagged status check raises a bare AgentBlockedError with empty reasons and no id. So a block that shows a real reason string and a decision id is proof the per-call policy engine denied it, not that an auth failure or a status breaker tripped. remaining_budget_usd is None because this is the loop path, not the spend path (only a Spend Cap deny populates that field).

This is not a case of "Loop Cap good, the other two bad." Runtime enforcement depends on where the model call is visible to the governance plane and which agent identity carries the call. For a tool-heavy Strands crew with near-zero LLM cost, a Loop Cap is the policy type whose signal actually exists in the spans. The Decision Log's "N span checks, No Match versus Denied" tells you both facts at once, which is why it is the screen I kept open the whole time.

I fixed two real bugs along the way, both committed: the crew entrypoint was forwarding the whole prompt sentence as applicant_id (a KeyError), and the platform init was passing the key without the endpoint. Neither was exotic; both are the kind of thing you only find by running it for real against the platform.

The evidence a regulator asks for

Once a real block exists, the Governance Hub turns it into paperwork. Working from the single blocked trace, I registered, reviewed, logged, and exported.

The Compliance Hub after the run: Readiness 85, one registered AI System, one pending review, one open incident. These are real, non-zero, and every one traces back to the single blocked run.

From that one blocked trace I built the paper trail a regulator expects. I registered the crew as a high-risk AI System (Article 12 record-keeping) with risk tier High, three agents linked, and an Annex III 5(b) intended purpose, which moved the registry from 0 systems to 1. I requested a human review on the blocked trace (Article 14 human oversight), and the Reviews tab went to Pending 1. I logged an incident (Articles 72 and 73), "Loop Cap block: credit-risk exceeded tool-call cap (2 > 1)," carrying the decision id and trace, and the Incidents tab went to Open 1. Finally I exported an audit bundle (Article 12 / Annex VIII), which is the payoff artifact: one hash-sealed JSON, about 10 KB, that ties everything together.

The Audit Bundle is one hash-sealed JSON that snapshots the registered system, the policy decisions, the human review, the incident, and the admin audit trail. Verified contents: the High-risk system with three linked agents, review_requests_count 1 with the blocked trace id inside it, incidents_count 1, audit_events_count 110, and a top-level integrity_hash.

Turning on the EU AI Act module in Settings then unlocked the EU-specific artifacts: a FRIA draft (Article 27), plus Instructions for Use (Article 13), an Annex IV technical-documentation outline, and an EU registration pre-fill (Annex VIII). Each is a real export tied to the registered system.

The FRIA wizard (Article 27), tied to the registered high-risk system. It exports a draft JSON with the real ai_system_id and a completion percentage, another artifact a deployer of a high-risk system is expected to produce.

Honest limits again, because a governance tool that overclaims is worse than no tool:

@govern defaults to fail_open=True. If the platform is unreachable, execution continues. I set fail_open=False for a high-risk crew, but that only hardens the before-run status check. The per-call policy check is always fail-open, with no override, so a network blip mid-run lets that individual call through. The on-camera block is a real per-call deny, but it is not a guarantee the platform can never miss a call.
integrity_hash is a plain unkeyed SHA-256. It is tamper-evidence, not a signature. Anyone who can rewrite the bundle can recompute the hash.
The EU registration export is a pre-fill, and it says so: annex_iii_category comes back null because the registry field is not auto-populated from the SDK span stamp, and the export carries a disclaimer that Traccia is not the official EU registry.
Evidence substrate is not legal compliance. A lawyer or a notified body still decides conformity. This produces the record; it does not sign off on it.

Real-time enforcement vs scheduled detection

One thing confused me on the dashboard, and it is worth 30 seconds because it looks like a bug and is not. The Policies page showed "Open Violations 0" even though I had real blocks. The reason is that Traccia has two policy families that surface in different places:

Preventive (per-call) policies (Loop Cap, Spend Cap, Model Boundary) act in real time, before each call, and show up instantly in the Decision Log as "Denied" (plus the AgentBlockedError). My block is one of these.
Detective (monitoring) policies (Output Token Limit, Cost Spike Alert) evaluate after the run, on a schedule (shortest cadence is hourly), and feed the Open Violations tile.

So "Open Violations 0" next to Denied rows is correct: the block is a real-time preventive deny, and a detective violation only opens on the next scheduled tick. I added a detective Output Token Limit policy to demonstrate the second path, and after the hourly evaluation Open Violations went to 1. The lesson for anyone building this: do not wait on camera (or in a demo) for the hourly tick, and narrate the distinction. Real-time block is the Decision Log; scheduled detection is the violations tile.

For completeness on the live numbers, after running the crew a few dozen times the registered agent showed the following.

Per-agent cost and posture for the crew: 37 executions, a 43% error rate, and a 7-day cost around $0.002. The error rate is high on purpose, because every demo run includes a deliberate injection attempt and every @govern deny raises an error by design. A high error rate here is the guardrails firing, not a defect.

The 37 executions and the near-zero cost are honest, not throttled or trimmed: each run is a short sequence of Nova Pro calls, and I did not fake a single one of those numbers to look better. Treat the 43% error rate the same way, as the guardrails and denies doing their job, not a reliability problem.

The timing that makes this worth doing now

Article 50 transparency obligations became enforceable on 2 August 2026. The Annex III high-risk obligations for credit scoring apply from 2 December 2027. That is a preparation window, not a free pass, and building the evidence layer before the deadline is far cheaper than retrofitting it after. (Verify these dates on your publish day; regulatory timelines move, and the kind of omnibus adjustments that shift them are exactly the thing to double-check.)

An honest take on Traccia governance

I shipped a real crew against this and read the SDK source to understand the behavior, so here is the assessment grounded in that, the parts that earned their keep and the parts that cost me an afternoon.

What earned its keep, first. The two-layer split does real work at $0: you get real governance in-process (block, redact, stamp) without a key, and the key adds networked enforcement plus the audit hub. The free tier is not a demo-ware teaser; it does the real work with no account attached. The Decision Log is the best diagnostic in the product. "N span checks, No Match versus Denied, attributed to agent X" told me both why my policy failed and how to fix it, and without it the two lessons above would have been a much longer afternoon. The evidence hub also maps cleanly to real EU AI Act articles: registry, review, incident, evidence pack, FRIA, and the EU exports line up with Articles 12, 13, 14, 27, 72/73, and Annexes IV and VIII, which is worth a lot to a team that needs to show something. And it does not oversell itself in the exports; the disclaimers ("not the official registry," the unkeyed hash) are baked into the product output, not just something I added in this article.

Now where it made me work, and where it should be better:

The endpoint requirement is silent. @govern skips enforcement without an endpoint (at the time of writing). Set only the key and you get ALLOWED with a log line most people miss. A hard error, or a louder warning, would save the confusion.
The always-fail-open per-call check deserves a louder callout. fail_open=False reads like "this will always stop," but it only hardens the status check; the per-call engine still fails open. That is a defensible design (a network blip should not hard-fail a production call), but it is exactly the kind of nuance a high-risk deployer needs stated up front.
The per-call identity attribution is a genuine gotcha for multi-agent crews. That the check runs under the sub-agent's run_identity, not the @govern entrypoint, is correct behavior, but it is undocumented for this case and it is exactly where a multi-agent build goes wrong first.
Documentation is the real gap, same as in Part 1. I learned the identity precedence, the endpoint requirement, the always-fail-open per-call rule, and the policy-matching logic by running the platform and reading the source, not from docs. The capability is there; the guidance for a multi-agent Strands build is not written down yet, which is precisely the gap this article fills.

Net: the governance is real and the evidence structure is genuinely useful. The gaps are documentation and a couple of developer-experience papercuts, and I would not wave all of them away as stage-of-product growing pains. For a tool whose entire value is trustworthy audit evidence, shipping always-fail-open per-call behavior that is not documented is more than cosmetic; a high-risk deployer needs that surfaced, not discovered. The capability is there and it works. The guidance and a few defaults have not caught up to it yet.

Honest caveats

The loan crew is synthetic. The credit model is a mock, the applicants are invented, no output is a real decision. This produces evidence, not compliance.
The local block and PII redaction run in-process with or without the key. The key adds networked enforcement and the hub, not the local guardrails.
A Traccia guardrail detects; it does not enforce. The Beat 1 block is my own control flow raising on a finding.
PII redaction is best-effort regex, not ML NER. It misses names and addresses and can over-redact.
integrity_hash is an unkeyed SHA-256: tamper-evidence, not a signature.
@govern defaults fail_open=True; the per-call check is always fail-open, even when you set fail_open=False.
I show guardrail Tiers A and C; Tier B is provider-native and does not fire on a clean run. Do not claim all three fire.
Every dollar and token count is real, but they shift run to run. Treat the relationships (a crew makes two tool calls; a Loop Cap of 1 blocks it) as the lesson, not the exact decimals.
The block that works is a Loop Cap scoped to credit-risk, not a Model Boundary or Spend Cap, for the reasons in the payoff section. Your policy choice depends on what your spans carry.

The takeaway is not "buy a governance tool." It is that watching a failure and stopping one are different layers, that in a multi-agent crew the enforcement has to target the agent that actually makes the calls, and that the audit evidence is something you generate on purpose, before a regulator asks.

FAQ

What is AI agent governance, and how is it different from observability?
Observability records what an agent did and what it cost. Governance decides whether the agent is allowed to do it (guardrails and runtime policies) and produces a durable audit record. Observability watches; governance blocks and proves.

Can I block a runaway agent without a paid platform?
You can block locally with an in-process guardrail that raises (the injection block here runs at $0, no key). What needs the platform is networked enforcement: a Loop Cap, Spend Cap, or Model Boundary the platform evaluates on every call and denies with a decision id you can cite.

Why did my @govern policy not block anything?
Three usual reasons, all of which I hit. You did not set TRACCIA_ENDPOINT, so the SDK skipped enforcement silently. You scoped the policy to the @govern entrypoint agent, but in a Strands crew the per-call check is attributed to the sub-agent's run_identity. Or you picked a policy type whose signal is not in your spans (a Spend Cap on a ~$0 crew, or a Model Boundary on a non-auto-patched client). Open the Decision Log; "No Match versus Denied" tells you which.

How does this map to the EU AI Act?
Credit scoring is Annex III point 5(b), high-risk. The demo produces record-keeping (Art. 12), transparency to deployers (Art. 13) and to users (Art. 50), human oversight (Art. 14), incident logging (Arts. 72/73), a FRIA draft (Art. 27), and Annex IV / VIII exports. It is evidence substrate, not a compliance sign-off.

Is Traccia open source?
The SDK (traccia-py) is Apache-2.0 and OpenTelemetry-native, so the spans are standard OTel and the local governance runs with no account. The @govern enforcement and the Governance Hub are the hosted, commercial part.

Does this work with Strands out of the box?
The $0 SDK governance (block, redact, stamp) works with any code you can decorate. The platform @govern works too, but the per-call policy matching has the multi-agent identity nuance above: scope to the sub-agent that makes the calls, and pick a Loop Cap for a tool-heavy crew.

Try it

Prefer to watch it happen first? The full walkthrough is on YouTube: Watch the demo.

The full code (the crew, the three $0 beats, the platform @govern script, the 10 unit tests, and a Makefile) is on GitHub: ai-agent-governance-aws. Clone it, run make demo to see the local block, redaction, and EU AI Act stamping at $0, then drop a Traccia key into .env, add the endpoint, and run make platform to get the real AgentBlockedError.

If you run agents in production: do you have a real block in the request path, or an alert that fires after the money (or the bad decision) is already gone? That is the difference this whole article is about.

I Followed the n8n AWS Docs and It Broke at the First Command

Sarvar Nadaf — Fri, 25 Sep 2026 14:09:50 +0000

You SSH into a fresh EC2 box, run sudo dnf install -y docker, then the very next command from the guide you're following:

docker compose up -d

And it dies:

docker: 'compose' is not a docker command.

I hit this on camera. Followed the steps, watched n8n never come up, and spent a few minutes convinced I'd broken something. I hadn't. Almost every "self-host n8n on AWS" tutorial has this exact hole in it, because the authors tested on Ubuntu or a VPS where Docker installs differently. On Amazon Linux 2023, the default AMI for EC2, the command they tell you to run gives you half of what you need.

This post is the fix, and the full path from a bare EC2 instance to your first n8n login. Two containers, one paste-ready compose file, no reverse proxy yet (that's the hardening step, and it's its own article). By the end you have n8n on Postgres running on a box you own.

The gotcha, up front

On Amazon Linux 2023:

sudo dnf install -y docker

installs the Docker engine. It does not install the Compose v2 plugin. They're separate now. Compose stopped being a standalone docker-compose binary years ago and became a plugin that lives under Docker's CLI, and dnf's docker package doesn't bundle it. So the engine runs fine, docker run works, and then docker compose throws 'compose' is not a docker command because the plugin isn't there.

On Ubuntu you'd install docker.io plus docker-compose-plugin from Docker's apt repo and never notice. On AL2023 the plugin is on you. Here's the whole install, plugin included:

sudo dnf install -y docker
sudo mkdir -p /usr/local/lib/docker/cli-plugins
sudo curl -fsSL \
  https://github.com/docker/compose/releases/download/v2.29.7/docker-compose-linux-x86_64 \
  -o /usr/local/lib/docker/cli-plugins/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
sudo systemctl enable --now docker

That third command is the one the other guides skip: it drops Docker's official Compose plugin binary into the directory the CLI actually looks in. Now both answer:

sudo docker --version
# Docker version 25.0.14, build 0bab007
sudo docker compose version
# Docker Compose version v2.29.7

Two version banners is how you know the box is actually ready. If you only check the first one, you find out the plugin is missing at docker compose up, which is the worst time to find out.

One catch on the download URL: it ends in x86_64, which is right for a t3 (Intel) box. If you launched a Graviton/ARM instance (t4g and friends), grab the aarch64 binary instead by swapping the filename to docker-compose-linux-aarch64. The wrong architecture installs cleanly and then fails with an exec-format error the moment you run it, which sends you hunting in the wrong place. Pick the binary that matches your instance. And v2.29.7 is just the version I pinned here; check Docker's releases and use the current one if you'd rather not lag.

Launch the box first

Before any of that, you need the instance:

AMI: Amazon Linux 2023.
Size: t3.small or larger. n8n plus Postgres want about 2 GB of RAM. Skip t2.micro and t3.micro (1 GB) unless you like watching containers get OOM-killed mid-run.
Security Group: SSH (22) from your IP only. For this private first test you can open 5678 to your IP only too. Do not open 5678 to 0.0.0.0/0. An open n8n editor on the public internet is an editor anyone can find and claim.
SSH in: ssh -i your-key.pem ec2-user@<PUBLIC_IP>

Everything below runs on that box as ec2-user.

Why bother self-hosting at all

n8n is a credential aggregator. One instance can hold your Stripe key, your database password, your Slack token, your Google OAuth, all in one place. On n8n Cloud that pile lives on someone else's server, priced per seat, capped on executions. Self-hosting moves it onto a box inside your own AWS account: no seat fees, no execution caps, your data stays home, and you pick the version.

The trade is honest. You now own the patching, the backups, and the security. This article gets it running. The hardening article (next in the series) closes the door behind it.

The stack: two containers

That's the whole thing.

n8n runs the editor and the workflow engine.
Postgres stores workflows and executions so they survive a restart.

Not SQLite. n8n defaults to SQLite, which is fine for a five-minute look, but it handles concurrent executions poorly and migrating off it later is an afternoon you won't enjoy. Start on Postgres.

Clone the repo so you have the compose file on the box:

git clone https://github.com/simplynadaf/self-host-n8n-on-ec2.git
cd self-host-n8n-on-ec2/compose

Here's the compose file:

services:
  n8n:
    image: n8nio/n8n:1.123.64
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_SECURE_COOKIE=false
      - N8N_DIAGNOSTICS_ENABLED=false
      - N8N_PERSONALIZATION_ENABLED=false
      - N8N_ENCRYPTION_KEY=change-me-to-a-long-random-string-please
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=change-me-strong-db-password
    depends_on:
      - postgres

  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_DB=n8n
      - POSTGRES_USER=n8n
      - POSTGRES_PASSWORD=change-me-strong-db-password
    volumes:
      - pg_data:/var/lib/postgresql/data

volumes:
  pg_data:

Three lines decide whether this works.

The image is pinned to 1.123.64, not latest. Pinning makes the build reproducible, and this version patches a real issue: CVE-2026-65589, an info-disclosure bug where credentials passed as custom headers in LLM sub-nodes could land in execution records. Run latest and you're one silent restart away from a version you didn't choose.

N8N_ENCRYPTION_KEY encrypts every credential n8n stores. Set a real 32-plus character random string and save it somewhere safe right now. Lose it and every saved credential is unrecoverable. A restored backup without this key is a database full of workflows whose logins can't be decrypted.

The DB password appears twice (DB_POSTGRESDB_PASSWORD in the n8n service, POSTGRES_PASSWORD in postgres) and the two values must match. If they don't, n8n can't reach its own database and the container just restarts in a loop while you wonder why the editor never loads.

Bring it up

Edit the two passwords and the encryption key, then:

sudo docker compose up -d
sudo docker compose ps

First run pulls both images and starts them. Give it twenty to forty seconds, then check n8n is answering:

curl -s -o /dev/null -w 'n8n -> HTTP %{http_code}\n' http://localhost:5678
# n8n -> HTTP 200

200 means n8n is up and serving. Once Compose was actually installed, this came back green on the first try.

First login

Open http://<PUBLIC_IP>:5678 in a browser. A fresh instance shows the setup wizard. Create your owner account with an email and a strong password, submit, and you land on the canvas.

Do this immediately after the box comes up, not tomorrow. The first account created on a fresh n8n becomes the owner, and until someone submits that form, it's open to whoever reaches it first. On a private Security Group that's only you. It's still a habit worth keeping.

One line in the compose file explains itself here: N8N_SECURE_COOKIE=false. That's only so first login works over plain HTTP on a raw IP while testing. It's a development shortcut, not a keeper. In production n8n sits behind HTTPS and this setting goes away.

Before this goes anywhere near the internet

The compose here publishes port 5678 directly. That's fine for a private test where the Security Group only lets your IP in, but it falls apart on the open internet, where certificate transparency logs announce every new HTTPS host within minutes and scanners find fresh boxes fast.

Before you point a domain at this or widen the Security Group:

Keep n8n patched (1.123.64 or newer).
Stop publishing 5678. Use expose so it's only reachable inside the Docker network.
Put Caddy in front for automatic TLS, and restrict the editor to your admin IP. Leave only /webhook/* public.
Move the encryption key and DB password into AWS Secrets Manager, read through a least-privilege IAM role.
Security Group: allow 22 (your IP), 80, 443. Never 5678 to 0.0.0.0/0.

That whole checklist is the hardening article in this series. If this box is going public, that's your next read.

Stop and start without losing data

Your data lives in the pg_data volume, so you can stop the stack safely:

sudo docker compose down     # stop, keep the data
sudo docker compose up -d    # start again

down stops the containers and keeps the volume, so your workflows and account are still there on the next up. To also wipe the data, that's down -v, and only when you mean it.

What you have now

n8n running on an EC2 box you control, backed by Postgres.
A pinned, patched image instead of a moving latest target.
An encryption key you actually set and saved.
A clear line for what to do before this faces the internet.

And you know the one thing most AWS n8n guides get wrong: on Amazon Linux 2023, installing Docker does not install Compose, and the fix is one curl into the plugin directory.

Where the series goes

Get it running (this one): bare EC2 to first login, past the Compose gotcha.
Harden it: TLS, closed editor port, Secrets Manager, least-privilege IAM.
Give it a brain: wire this same n8n to Amazon Bedrock and build a real AI agent on the canvas, model running in your account, no OpenAI key anywhere.

The compose file, the install and verify scripts, and the full setup notes are in the repo: github.com/simplynadaf/self-host-n8n-on-ec2.

Per-Agent Cost Tracking for Multi-Agent AI on AWS

Sarvar Nadaf — Wed, 23 Sep 2026 13:41:46 +0000

Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance monitoring you already run: uptime, latency, error rate) says 200 OK, latency fine, everything green.

And you were silently billed about 1.4x what you should have been.

That is the part nobody shows you. Nested traces and per-agent cost are becoming common; the primitives are easy to find now. What stays rare is a data model that lets you act on them: catch the run that looks completely successful while it burns money in the middle. The paper "Why Do Multi-Agent LLM Systems Fail?" (MAST, arXiv:2503.13657) hand-annotated 150 traces across 7 state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa=0.88, and measured failure rates from 41% to 86.7%. The uncomfortable finding: many of those failures do not crash. They complete. They look fine.

In this article I build a small read-only "AWS Account Investigator" crew, wire real cost into every trace span, and then reproduce three silent-waste patterns with real Amazon Nova Pro dollars. You can run the whole thing for $0 locally. Nothing gets created, modified, or deleted in your AWS account.

If you only have two minutes, jump straight to the unique part: catching silent waste. The build up to it matters, but that section is the payoff.

I spent about a week on this against a real AWS account: a few days probing the SDK's behavior before I trusted it, then several more building the crew, watching the trace design break twice, and reading the SDK source when the docs ran out. What follows is written from that, not from a quickstart. The scars are in here on purpose, because they are the part that saves you the week.

This is for people already building AI agents who have never put a real observability layer under them. You know agents, tools, and crews. Where the tracing vocabulary (spans, traces, OpenTelemetry) is new, I define it the first time it shows up.

Contents

Why agent observability is a different problem
Picking the instrumentation
Prerequisites
Adding Traccia to your code
The stack: AWS native and read only
The cost bridge and one gotcha
Modeling a multi-agent crew in traces
The unique part: catching silent waste
Watching it happen: the live control panel
Build your own, at zero cost and read only
An honest take on Traccia
Honest caveats
FAQ

Why agent observability is a different problem

Traditional application monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that is enough, because the work is deterministic and the failure modes are loud. An AI agent breaks all three assumptions. It decides its own control flow at runtime, it calls tools in an order you did not hardcode, and it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it is a trend, not an event.

So agent observability has to record things classic APM never needed: how many reasoning cycles an agent took, which tools it called versus which it was allowed to call, the token count and dollar cost of each step, and which agent in a multi-agent crew did what. Those attributes are what make an invisible regression visible.

This is not a fringe opinion. AWS's own Well-Architected Agentic AI Lens frames the baseline state (its "Level 1") as exactly this problem: agent costs are visible only at the account level, Cost Explorer cannot separate agents or workflows, and "teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing." The whole point of what follows is to move off Level 1: to make spending "attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level," which is AWS's own words for the target.

A quick vocabulary anchor, since the rest of the article leans on it. A span is one timed step with attributes attached (one LLM call, one tool call, one AWS read). A trace is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes (status, latency). Agent observability is the same trace structure carrying agent-specific attributes: cycle count, tokens, cost, and which agent owned the step. That is the whole idea; everything below is just putting the right attributes on the right spans.

At one run, a 1.4x overspend is a rounding error. At enterprise scale it is a budget line and a governance problem, and it shows up in four concrete ways:

Cost control. A 1.03x-to-1.4x silent overspend per run (the real range I measured across three waste patterns), multiplied across thousands of daily runs and dozens of agents, is real money leaking with no alarm attached. Per-agent, per-tool cost on the trace is the only way to attribute and cap it.
Accountability. When a crew misbehaves, "which agent, owned by which team, cost what" needs to be answerable. Trace-level ownership metadata turns a vague incident into a routed ticket.
Regression detection. Agents change when prompts, models, or tools change. A known-good baseline plus per-run deltas catches the day a prompt tweak silently doubled token usage, before finance does.
Auditability. In regulated environments you need a record of what the agent read, what it decided, and what it cost. A trace is that record.

The theme throughout: a correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose. Here is the before and after in one line. Before, a typical demo gives you one lump token count for the whole run, and an inefficient run looks identical to an efficient one. After, every reasoning step and every AWS read is a span carrying real cost, tokens, cycle count, and the owning agent's identity, so two runs that both return the correct answer and both show 200 OK are no longer indistinguishable when one of them costs 43% more.

Picking the instrumentation

Once you know you need per-agent cost, cycle counts, and tool-call attributes on every span, the next question is what to write them with. You could do a lot of this with raw OpenTelemetry, and I nearly did. The reason I did not is that agents need a vocabulary plain OTel does not ship: token counts turned into dollars, a span-level agent identity so one process can render as a real fleet, ownership metadata, and a way to view per-agent cost grouped by session. You end up building all of that yourself, or you find an SDK that already speaks it.

There are options here: LangSmith, Langfuse, and Arize Phoenix all do LLM tracing, and each is worth a look depending on your stack. I went looking for one I would trust in a codebase, which for me means two hard requirements: I can read the source, and I am not locked in. Traccia cleared both cleanly. The SDK is open source, Apache-2.0 licensed, and built on OpenTelemetry (OTel, the vendor-neutral open standard for traces and metrics, the reason you are not locked into any one backend). The spans it produces are standard OTel, the file exporter works with no account and no network, and I could read exactly what it does to my data before committing to it (I did, and the source-grounded critique later in this article is the result). It runs at $0 locally; the hosted dashboard at app.traccia.ai is optional and only comes in when you want the visualization. An open, inspectable SDK with an optional commercial backend is a split I am comfortable adopting, because the instrumentation does not trap me.

That is the real reason it is in this build: agent-native plumbing I did not want to hand-roll, source I could audit, and a real $0 offline path. It also has sharp edges, and I hit several of them; those are documented in full near the end rather than glossed over.

Why not Amazon Bedrock AgentCore Observability or Langfuse, the two obvious AWS-native alternatives? Both are good, and for many teams either is the right call. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on the AgentCore runtime, but AWS's own Well-Architected lens is blunt about the cost gap: "cost reporting stops at the AWS account level, so teams can't separate supervisor overhead from worker execution." Per-agent dollars are something you still assemble. Langfuse is the strong open-source incumbent and I would happily use it; it just was not the tool I was asked to put through its paces here. The point of this build is not "Traccia beats them." It is that whichever tracer you pick, the per-agent cost attribute and the baseline-delta detection are things you wire on purpose, and this article shows exactly how.

Prerequisites

Nothing exotic. Three things to run this yourself:

Python 3.10+ and the SDKs (strands-agents, strands-agents-tools, traccia, boto3). The repo pins the exact tested versions in requirements.txt.
AWS credentials with read-only permissions for the services the crew reads (Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, GuardDuty), plus bedrock:InvokeModel so the agent can actually call the model. The repo ships a ready-to-use policy at iam/read-only-policy.json; AWS's managed SecurityAudit + ViewOnlyAccess cover the reads, but you still add bedrock:InvokeModel on top of them.
Amazon Nova Pro, which is two separate steps: (a) enable model access once in the Bedrock console (us-east-1, amazon.nova-pro-v1:0) under Model access, and (b) allow bedrock:InvokeModel in your IAM policy. The console grant is not an IAM permission, so you need both.

No Traccia account is required. With no API key it writes traces to a local file, which is the $0 path used throughout this article.

Getting it running is four commands:

git clone https://github.com/simplynadaf/ai-agent-observability-aws.git
cd ai-agent-observability-aws
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

Then create the least-privilege policy once (with your own admin credentials) and attach it to whoever runs the crew:

aws iam create-policy \
  --policy-name AgentObservabilityReadOnly \
  --policy-document file://iam/read-only-policy.json

Adding Traccia to your code

Before the crew, here is the smallest version of what "wire cost into a span" actually means, because that is the one non-obvious step. Traccia auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, so on those stacks you get most of this for free. It does not yet hook Strands or raw Bedrock, so you stamp the cost yourself. It is a short function, and one attribute name will bite you (more on that below).

def stamp_llm_cost(span, result, model_id="amazon.nova-pro-v1:0"):
    usage = result.metrics.accumulated_usage
    in_tok, out_tok = usage["inputTokens"], usage["outputTokens"]
    cost = (in_tok / 1000 * 0.0008) + (out_tok / 1000 * 0.0032)  # Nova Pro, us-east-1

    span.set_attribute("llm.model", model_id)   # REQUIRED. wrong key = silently zero
    span.set_attribute("span.type", "LLM")
    span.set_attribute("llm.usage.prompt_tokens", in_tok)
    span.set_attribute("llm.usage.completion_tokens", out_tok)
    span.set_attribute("llm.cost.usd", round(cost, 6))

That is the whole idea: read the token usage the SDK already gives you, turn it into dollars against real pricing, and attach it to the span. Everything else in this article is applying this same move across a multi-agent crew and then reading the numbers back. The full wiring (init, the per-agent identity, the tool spans) is in src/crew.py in the repo.

The stack: AWS native and read only

The crew runs on Amazon Nova Pro (amazon.nova-pro-v1:0) through AWS Strands Agents using the agents-as-tools pattern. A supervisor named investigation_run delegates to three specialist sub-agents. Each specialist is a real separation of concerns, owns several read-only tools, and every one of those tools opens its own live span, so in the dashboard you see the agent, then each AWS read nested under it with a real duration. Here is the full fleet and exactly what each agent does.

1. AWS Account Investigator (supervisor, `investigation_run`)

The orchestrator. It does not touch AWS directly; it reads the user's question, decides which specialists are in scope, delegates to them, and synthesizes one report. On its trace span it records agent.delegated_to (which specialists it called this run), and it stamps the shared session.id that ties the whole investigation together.

The delegation is intent-routed, not fan-out-everything. The supervisor's instructions are strict: call ONLY the specialist whose domain the user actually asked about. Ask only about cost and it delegates to the Cost Analyst alone, while Health & Ops and the Security Auditor never run. Ask only about security and only the Security Auditor fires. Only a whole-account question ("what's running, any risks, and where is my spend going?") lights up all three. This matters for the trace and the bill: agent.delegated_to shows exactly which specialists ran, and a scoped question costs a fraction of a full sweep because the agents you did not need never spent a token. You can watch this live in the control panel: a cost-only prompt lights up one agent and leaves the other two idle.

Task	Tool	AWS read-only API
Plan + delegate	`cost_analyst`, `health_ops`, `security_ops`	none directly (delegates)

The supervisor's trace in Traccia: agent.delegated_to records which specialists ran this run, and the shared session.id links the four agents into one investigation.

2. Cost Analyst (`cost_analyst`)

A read-only FinOps specialist. It builds a full spend picture with three tools, and all three show up as separate tool spans in its trace.

Task	Tool	AWS read-only API
Month-to-date total, month-end forecast, top 5 services	`cost_forecast`	`ce:GetCostAndUsage`, `ce:GetCostForecast`
Last full month's total for month-over-month change	`last_month_cost`	`ce:GetCostAndUsage`
Daily cost series to catch a spike	`daily_cost_trend`	`ce:GetCostAndUsage`

What it reports on a real run: actual MTD spend, the account's forecasted month-end total, the top services by spend, the month-over-month direction and rough percentage, and the single most expensive day compared against the daily average (a possible spike).

The Cost Analyst trace: three Cost Explorer tool spans nested under the agent, each with its real read duration and the agent's own per-agent cost. This per-agent cost figure is exactly what turns a "successful" run into a caught overspend later.

3. Health & Ops (`health_ops`)

A read-only SRE specialist. It inventories the account and reads health signals with five tools, so it is usually the heaviest agent on input tokens (it chains the most reads).

Task	Tool	AWS read-only API
List running EC2 instances	`running_instances`	`ec2:DescribeInstances`
Read CPU utilization per instance	`cpu_utilization`	`cloudwatch:GetMetricStatistics`
Find unattached (idle) EBS volumes	`list_volumes`	`ec2:DescribeVolumes`
Inventory Lambda functions	`list_functions`	`lambda:ListFunctions`
Inventory S3 buckets	`list_buckets`	`s3:ListAllMyBuckets`

What it reports: running instances with their CPU, an inventory of volumes, functions, and buckets, and any notable health finding such as an unattached EBS volume.

The Health & Ops trace: five read-only tool spans and, on this run, the highest token count of the fleet (4,772) because it chains the most reads.

4. Security Auditor (`security_ops`)

A read-only security specialist. It runs four independent checks, each its own tool span.

Task	Tool	AWS read-only API
Security groups open to the internet (0.0.0.0/0)	`open_security_groups`	`ec2:DescribeSecurityGroups`
MFA gaps on the root account and IAM users	`mfa_findings`	`iam:GetAccountSummary`, `iam:ListUsers`, `iam:ListMFADevices`
S3 buckets missing a public-access block	`public_s3_buckets`	`s3:ListAllMyBuckets`, `s3:GetPublicAccessBlock`
Whether GuardDuty is enabled	`guardduty_enabled`	`guardduty:ListDetectors`

What it reports: each finding stated plainly with its risk, and it explicitly says so when a check comes back clean.

The Security Auditor trace: four independent read-only checks, each its own tool span with a real duration.

Each specialist stamps its own identity onto its trace span, so from a single crew run the dashboard shows four distinct agents with their own token and cost profiles, not one agent logged four times. Each agent runs as its own top-level trace, tied to the others by a shared session.id, and carries production ownership (type, owner, team) from a catalog file. Every call is a describe or get. There is no create, no modify, no delete. The worst thing this agent can do is read a bit too much, which, as you will see, is exactly the waste we want to catch.

Observability comes from Traccia, an OpenTelemetry-native agent-observability SDK.

pip install traccia

Runs $0 by default using a local file exporter. If you set TRACCIA_API_KEY, it pushes spans to app.traccia.ai. No key, no network, no cost. (The repo pins the exact tested version in requirements.txt; the prose stays unpinned so it does not age.)

Nova Pro pricing, pulled live from the AWS Price List API (effective 2026-08-01, us-east-1):

Token type	Price per 1K
Input	$0.0008
Output	$0.0032

Every dollar figure below is computed from real token counts against these two numbers.

The cost bridge and one gotcha

Here is the part most tutorials skip. Traccia auto-instruments several stacks out of the box (LangChain including BedrockChat, CrewAI, OpenAI Agents, and raw OpenAI/Anthropic/Gemini), and it ships a cost engine with a bundled pricing snapshot that covers Nova and Claude. But there is no Strands integration yet, and it does not hook raw Bedrock converse calls. So for this specific stack, Strands agents-as-tools calling Bedrock directly, you wire the cost in yourself. That is a fair amount of the value proposition for supported frameworks arriving for free, and real manual work for an unsupported one.

Strands hands you the token usage after a run. You read it, compute the cost, and stamp it onto the span, which is exactly the stamp_llm_cost function from earlier. About 40 lines once you handle all four agents and the tool spans; the full version is in src/crew.py.

The gotcha cost me a confused afternoon, and reading the SDK source explained exactly why. Traccia's cost-annotating processor only computes cost for a span when three things are all true: span.type is LLM (or unset), an llm.model attribute is present, and both token counts are set. Miss any one and the processor simply returns, with no error and no warning. I first set llm.request.model (which felt more semantically correct) instead of llm.model, so the processor silently skipped every span, and the "LLM Calls" and "Total Tokens" tiles read zero while my spans clearly had tokens on them. Set llm.model, and the tiles light up. The forgiving fail is reasonable; the fact that it is invisible is the trap. A one-line debug log ("skipping cost: no llm.model") would have saved the afternoon.

No double-counting across agents

The other question that always comes up: if the supervisor calls two sub-agents, and I sum everyone's tokens, am I counting the sub-agent tokens twice?

I wrote probes/probe_doublecount.py to check instead of guessing. Strands runs each sub-agent in its own event loop with its own metrics object. A supervisor's accumulated_usage is exclusive of its sub-agents' tokens. So the arithmetic is clean:

crew total = supervisor + sum(sub-agents)

No subtraction, no overlap, no double-count. Verified, not assumed.

Modeling a multi-agent crew in traces

Once the bridge is in, you get per-agent cost. But here is a design decision worth being explicit about, because most demos hand-wave it: how do you model a supervisor and its specialists in a trace?

You have two reasonable options. You can nest everything under one trace (supervisor is the root, sub-agents are child spans). Or you can give each agent its own top-level trace and tie them together with a shared session.id. I went with the second, because it is what a real production fleet looks like: the Cost Analyst, Health & Ops, and Security Auditor are independently owned, independently operated services. On the Traces page they show up as their own executions, each with its own cost, tokens, and duration; "Group by session" folds them back into one investigation when you want the whole picture.

session 4f4c1ade...  (one investigation, four independent traces)

  investigation_run   AWS Account Investigator   $0.006   delegated -> 3
  cost_analyst        Cost Analyst               $0.004
  health_ops          Health & Ops               $0.005   (highest total tokens: 4,772)
  security_ops        Security Auditor           $0.005
                                          crew total   $0.021

(These are the real per-agent figures from the exported run shown in the trace screenshots above, rounded to the dashboard's own cost tiles; they shift run to run with token usage. The crew total is the investigation_workflow roll-up span, which equals the supervisor's own synthesis plus the three sub-agents, no double-count. Health & Ops carries the highest token count because it chains the most reads, while the supervisor costs about the same because it writes the long final synthesis. The CLEAN and CONTEXT BLOAT numbers later in the article come from separate, labeled runs, so do not expect them to tie back to this one.)

Each agent's own trace still nests its tools underneath it (agent -> tool:running_instances -> the real boto3 call), so you keep the drill-down without pretending four separate services are one call stack. On the Traces page, "Group by session" folds all four agents from one run back into a single investigation, so you can move between the fleet view and the per-agent view without losing either.

(This is a different run from the CLEAN baseline used later; token counts and therefore dollars shift run to run. The point is the per-agent breakdown, not the absolute number.)

Good. Useful. Per-agent cost on its own is becoming common. The reason it matters here is not the number itself but the data model underneath it: once every step carries cost, tokens, cycle count, and an owning agent, you can build the thing that is still rare, which is catching a run that overspends while looking perfectly healthy. That is what the primitives let you build next.

A few things here are easy to get subtly wrong, and I hit them in roughly this order over a couple of days before the trace design held. First, all four agents come from a single crew run in a single process. Traccia bakes the agent identity into the OpenTelemetry resource at init, which is process-level, so my first version labeled every trace with one agent name: three identical "AWS Account Investigator" rows in the dashboard. Reading the SDK's enrichment processor showed that a span-level agent.id / agent.name attribute takes precedence over the process default, so stamping each agent's span with its own identity makes it show up as its own agent. Static ownership (type, owner, team, org) comes from an agent_config.json catalog the SDK auto-discovers, so the dashboard shows a real fleet with owners and teams, not four anonymous rows. No extra processes, no fake agents.

Second, separate traces need a real correlation key or they look disconnected. Every agent stamps the run's session.id, and the supervisor additionally records agent.delegated_to (which specialists it called this run). That is the explicit link that makes four independent traces read as one orchestrated investigation.

Third, the "each agent is its own trace" bit did not happen by wishing, and this one cost me a rebuild. Traccia's span_scope(parent=None) still inherits the current span if one is active, so my agents silently collapsed back into one trace until I detached the OpenTelemetry context before starting each agent's span. One small helper, verified by counting distinct trace IDs in the exported spans.

Fourth, the first time I looked at the timeline every tool span was 0ms, because I was reconstructing tool spans after the fact from the metrics object. The fix was to wrap the real boto3 call in a live span while it runs, so the timeline shows each AWS read's true duration. A 0ms bar is the kind of thing that makes a viewer distrust the whole trace, and it is worth chasing down. Then a subtler follow-on bit me: those live tool spans inherited the process-level default identity, so every tool bucketed under the supervisor and the specialists looked trace-thin. I had to stamp each tool span with its calling agent's identity too. Nothing about that was in the docs; I found it by parsing the exported traces.jsonl and noticing the agent.id was wrong.

One more touch that reads as production, not demo: each agent records both agent.tools_available (the full toolset it was granted) and agent.tools_called (what the model used this run). On this run, Cost Analyst had two tools available (month_to_date_cost and cost_forecast) and used one; Health & Ops had five and used all five. That gap is not a bug to hide, it is real information. "Has two, used one" is exactly the kind of thing you want visible when you are deciding whether an agent is over-provisioned.

The unique part: catching silent waste

Here is my disclaimer up front. I saw versions of all three of these in real runs while building the crew, then engineered them in src/waste_demo.py to trigger reliably so you can watch them on demand instead of waiting for a bad run. That reproduction is on purpose. LLM output is non-deterministic, so in production the same patterns show up on their own, just not on a schedule you can demo. And critically: detection here is delta-vs-baseline, not magic absolute thresholds. That is how real regression detection works. You capture a known-good run, then flag runs that deviate. Every number below is real Nova Pro token usage from a representative run. Your numbers will vary; the ratios are what hold.

First, the clean baseline. This is the "known good" I compare everything against.

CLEAN baseline (one exported run): crew total ~ $0.0083
  health_ops: 2,274 input / 199 output / 3 cycles

Scenario 1: The runaway loop

The agent gets stuck re-reasoning and re-reading the same things.

RUNAWAY LOOP: ~1.2x baseline ($0.0100 vs $0.0083)
  health_ops ran 4 cycles (baseline: 3)
  re-read cpu_utilization twice (baseline: once)
  health_ops input tokens ~1.6x (3,557 vs 2,274)

Same final answer. The APM span is 200 OK. What catches it: agent.cycle_count and tool.call_count. The agent looped more than its baseline and called the same tool repeatedly. No single number is "wrong." The delta is wrong.

Scenario 2: Redundant tool calls

Milder, sneakier. The agent calls a tool it already has the answer for.

REDUNDANT TOOL CALLS: ~1.03x baseline ($0.0085 vs $0.0083)
  running_instances called 3x (baseline: 1x)

A few percent on one run is the kind of thing you never notice. Multiply it across thousands of daily runs and it is a line item. The signal: tool.call_count for running_instances jumped from 1 to 3. Only visible per-tool, per-agent, and easy to miss precisely because the dollar delta is so small on a single run.

Scenario 3: Context bloat (the expensive one)

The agent drags too much context into its prompts. Every extra token in gets paid for, and it cascades.

CONTEXT BLOAT: ~1.4x baseline ($0.0119 vs $0.0083)
  health_ops input tokens elevated, output nearly 4x (803 vs 199)
  supervisor synthesis cost rises too, bloat cascades

This is the meanest one because it compounds. The sub-agent's bloat feeds a bigger blob to the supervisor, whose own synthesis cost then rises too (on this run the supervisor jumped from $0.0032 to $0.0044). The signal: llm.usage.prompt_tokens and llm.cost.usd per agent. You watch prompt tokens creep up where the work did not.

Same answer, different bill

src/compare.py puts CLEAN next to BLOAT side by side.

	CLEAN	CONTEXT BLOAT
Final answer	Correct	Correct
Crew total cost	$0.0083	$0.0119
Delta	-	+43% (~1.4x)
APM status	200 OK	200 OK

Two runs. Both return the right answer. Both are green in any latency-and-errors dashboard. The only place the extra 43% shows up is in the trace, on the per-agent cost attribute you stamped yourself. In the Traccia dashboard this is the moment the tool earns its place: two runs sit side by side, both "successful," and the per-agent cost column is where the bloated one gives itself away. That is the whole argument for agent-native observability in one table.

The detection logic is not clever. It is a delta check: for each agent, compare this run against the baseline and flag three things. More cycles than baseline means a possible runaway loop. Prompt tokens more than 1.25x baseline means possible context bloat. Any tool called more times than baseline means a possible redundant call. That is the whole detector, about fifteen lines in src/compare.py.

The intelligence is in having the baseline and the per-agent attributes to compare against. The trace is what makes those attributes exist.

This detector broke once during the build, and it is a good example of how instrumentation and detection are coupled. When I switched tools to emit one live span per call (the 0ms fix above), the redundant-call check stopped working, because it had been reading a call_count attribute off a single reconstructed span that no longer existed. I had to change it to count span occurrences per tool name instead. The lesson that stuck: change how you record, and you can silently break how you detect. The baseline caught it, which is the whole point.

Watching it happen: the live control panel

This is the panel you saw in the video above. Traces are the source of truth, but a wall of span JSON is not how you show a crew to a teammate. So the repo ships a small live control panel: a single-page UI that runs the real crew and animates the investigation as it happens.

You type a prompt into a command console, hit Investigate, and the view scrolls down to a graph of the crew. The supervisor sits at the top and the three specialists fan out below it, connected by wires. As the run streams, each agent lights up like a traffic signal: idle, then running (with a live activity line, "Reading Cost Explorer", "Scanning security groups"), then done, and the report reveals at the bottom. Every agent card shows the AWS services it touches as small chips, so a viewer can see at a glance that Cost Analyst reads Cost Explorer and the forecast, Health & Ops reads EC2/EBS/Lambda/S3/CloudWatch, and Security Auditor checks security groups, IAM, S3, and GuardDuty.

The panel has two modes. Live runs the real crew: real Nova Pro calls, real read-only AWS reads, real dollars on the trace, about thirteen seconds. Replay animates a saved run from a committed trace file, deterministically and for free, so you can rehearse the visual as many times as you want without spending a token. Both drive the exact same UI from the same event stream; the only difference is whether the events come from a fresh Bedrock run or a recorded one.

The backend is a small FastAPI app that streams the crew's lifecycle as Server-Sent Events. The important part is that the UI is a thin viewer over the same telemetry the trace records; it is not a second, hand-maintained source of truth. What the graph shows is what the crew did.

Build your own, at zero cost and read only

You do not need a paid plan or a live AWS bill to try this. The whole thing runs locally with the file exporter and read-only AWS credentials.

The permission surface is deliberately small: every action is a Get, List, or Describe, across Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, and GuardDuty. There is no create, no modify, no delete anywhere in the toolset. The full policy JSON is in the repo README; if you would rather not hand-roll it, AWS's managed SecurityAudit and ViewOnlyAccess policies cover the same set. Attach it, invoke Nova Pro through Strands, and you have a crew that can look but never touch. To send traces to the hosted dashboard, set TRACCIA_API_KEY; leave it unset and everything writes to a local file. Same spans either way.

The read-only shape is the same for every tool: wrap the real boto3 describe/get/list call in a live span so its duration in the trace is the true AWS read time, return the fields you need, touch nothing. src/tools.py in the repo has all seven; they are all this shape.

An honest take on Traccia

I shipped a real crew against this SDK and read its source to understand the behavior, so here is the assessment grounded in that, not in the marketing page.

What is genuinely good:

It is OpenTelemetry-native. Spans, processors, and resource attributes are standard OTel underneath, so the data model is not proprietary and you are not locked in.
It runs at $0 and offline by default. With no API key it writes to a local file exporter; set TRACCIA_API_KEY and the same spans push to the hosted dashboard. Same spans either way, which made local development and CI painless.
It ships more than a tracer. There is a real cost engine with a bundled pricing snapshot (covering Nova and Claude, among others), a staleness warning when that snapshot ages, and auto-instrumentation for LangChain, CrewAI, OpenAI Agents, and the raw OpenAI/Anthropic/Gemini clients. If you are on one of those stacks, a lot of what I wired by hand would have come for free.
The span-level agent identity model is the best part. A span-level agent.id and agent.name override the process default, which is precisely what let a single-process crew render as a four-agent fleet with real per-agent cost. That is a thoughtful design decision, not an accident.

Where it made me work, and where it could be better:

No Strands integration yet, and it does not hook raw Bedrock. For this stack the cost bridge was manual. That is fine and it gives you control, but a Strands integration would remove the single biggest chunk of setup for AWS-native builders.
The cost processor fails silently (at the time of writing). It skips a span with no error if llm.model is missing or span.type is not LLM. That forgiving behavior is defensible, but the silence cost me an afternoon of a zeroed dashboard. A debug log on skip would fix it outright, and it is the kind of small papercut an early-stage tool usually closes fast.
A couple of sharp edges are only discoverable in the source (at the time of writing). span_scope(parent=None) still inherits the current context (so separate agent traces silently merge unless you detach first), and span_scope is not a context manager (you call .end() yourself). Neither is obvious from the docs today.
Documentation is the real gap. I learned the identity precedence, the llm.model requirement, and the context-detach behavior by reading the SDK, not the docs. For a bootstrapped, early-version product that is understandable, and the SDK itself is readable enough that this was possible. But better docs would turn a day of spelunking into an hour.

Net: for supported frameworks you get a lot for free, and even off the beaten path the OTel foundation and the cost/identity model are solid. The capability is there; the polish that is missing is mostly documentation and a few developer-experience papercuts, which is exactly what you would expect from a product at this stage.

Honest caveats

I want to be straight about the limits, because that is the whole point of this article.

I saw versions of the three waste scenarios in real runs first; the src/waste_demo.py versions just make them fire on cue. LLM output is non-deterministic. In real life these patterns appear on their own, just not on a schedule you can demo.
Detection is delta-vs-baseline, not fixed magic thresholds. You need a known-good run to compare against, same as any regression system.
Every dollar is real Nova Pro token usage against verified us-east-1 pricing, but the exact numbers shift run to run. Do not treat any single figure as a constant. Treat the relationship (roughly 1.4x on the worst pattern I measured) as the lesson, not the exact decimals.
Traccia does not auto-instrument Strands or Bedrock. The cost bridge is about 40 lines you write and own. That is a feature: you control exactly what goes on the span.
I model each agent as its own trace, grouped by session.id. That is a deliberate choice to match how a real fleet is owned and operated. If you prefer one nested trace per run, keep the supervisor as the parent instead of detaching the context. Both are valid; pick the one that matches how your team reasons about the system.
The agent is read-only by IAM policy, not by hope.

The takeaway is not "buy an observability tool." It is that a correct-looking answer tells you nothing about whether the run was efficient, and the only place the truth lives is in the trace, on attributes you have to put there on purpose.

FAQ

What is AI agent observability, and how is it different from LLM monitoring?
LLM monitoring usually watches one model call: latency, errors, maybe token count. Agent observability watches a whole reasoning session: how many cycles an agent took, which tools it called, the cost of each step, and, in a multi-agent crew, which agent did what. Agent failures show up across a multi-step chain, not on a single call, so you need the full trace to see them.

How do I track per-agent cost on Amazon Bedrock?
Bedrock returns token usage after each call. You multiply input and output tokens by the model's per-1K price (for Nova Pro in us-east-1, $0.0008 in and $0.0032 out) and attach that dollar figure to the trace span for the agent that made the call. That is the stamp_llm_cost function in this article. AWS's own tag-based cost allocation in Cost Explorer works at the account and tag level; per-agent, per-reasoning-cycle attribution is what the trace adds on top.

Can AWS Cost Explorer show per-agent cost by itself?
Not on its own. Per AWS's Well-Architected Agentic AI Lens, the default state is that costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. Tag-based allocation plus AgentCore Observability improves this, but per-agent and per-reasoning-phase attribution comes from instrumenting the trace, which is what this build does.

Why does my multi-agent app cost more than I expected even when it works?
Because a correct answer is not a cheap answer. Agents can loop an extra reasoning cycle, re-call a tool they already have the answer for, or drag bloated context from step to step. None of that returns an error; it just adds tokens. The overspend shows up on the bill, not in a latency-and-errors dashboard, which is the "silent waste" this article is about.

Do I need a paid tool or an AWS account to try this?
No. The whole build runs at $0 locally: the Traccia SDK writes traces to a local file with no API key, and the AWS reads use read-only credentials (or the committed replay run, which needs no AWS access at all). You only need a Bedrock model grant if you want to run the live crew against your own account.

Is Traccia open source?
The SDK (traccia-py) is open source under Apache-2.0 and built on OpenTelemetry, so the spans are standard OTel and you are not locked in. The hosted dashboard at traccia.ai is the optional commercial part; you only reach for it when you want the visualization.

Does Traccia support AWS Strands Agents out of the box?
Not at the time of writing. It auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, but not Strands or raw Bedrock converse, so on this stack you stamp cost onto the span yourself (about 40 lines). On a supported framework, most of that is automatic.

Try it

The full code (crew, tools, waste demos, the live control panel, the compare view, and the double-count probe) is on GitHub: ai-agent-observability-aws. There is also a live replay of a run you can click through in the browser: https://simplynadaf.github.io/ai-agent-observability-aws/. Clone it, run python -m src.waste_demo with local export, and watch a perfect answer cost you ~1.4x. Then go instrument your own agents before your bill does the teaching for you.

To put Traccia under your own agents, the on-ramp is deliberately short and free:

Install the SDK (open source, Apache-2.0): pip install traccia. With no API key it writes traces to a local file, so you can see spans at $0 before you sign up for anything. Source and docs: github.com/traccia-ai/traccia-py.
Stamp cost onto your spans using the ~40-line stamp_llm_cost pattern above (or get it for free if you are on LangChain, CrewAI, or the OpenAI/Anthropic/Gemini clients, which Traccia auto-instruments).
See it in the dashboard when you want the visual per-agent cost and the side-by-side run compare: set TRACCIA_API_KEY and the same spans push to traccia.ai. Same spans either way, so nothing about your instrumentation changes.

If you build something with it, tell me what silent waste you found. That is the interesting part.

I Built an AI Agent That Audits AWS (And It Can't Touch Anything)

Sarvar Nadaf — Fri, 18 Sep 2026 15:12:34 +0000

An AI agent just read my AWS account and told me a bucket was open to the internet, SSH was exposed to 0.0.0.0/0, GuardDuty was off, and I was burning $3.65 a month on an Elastic IP attached to nothing.

It did all of that in about two minutes. And here is the part that counts: it physically could not have changed anything even if it tried.

That last sentence is the whole point of this build. Most "give the AI access to my cloud" ideas die on one fear: what if it deletes something, or a bad prompt tricks it into running a destructive command? We remove that fear at the permission layer, not with a polite instruction. The agent runs on a read-only IAM identity. Every write call it could imagine gets rejected by AWS before it happens.

This is a walkthrough of building that agent from scratch. It is one JSON file and one Markdown checklist. By the end you will have a working AWS auditor you can point at your own account, and you will understand every field that makes it work.

The full code is on GitHub: github.com/simplynadaf/aws-auditor-agent.

Who this is for

You use Kiro Crew or the Amazon Q Developer CLI. You know your way around AWS enough to have an account with a few things running. You have heard "AI agent" a hundred times and you want to see what one is built from, without a framework, without a vector database, without 400 lines of Python.

If you can edit a JSON file and write a checklist in Markdown, you can build this.

What an agent is made of (the 6 pieces)
The safety foundation: read-only IAM
Writing the agent config
The skill: your audit checklist
Wiring the AWS tools with MCP
Running it, and what it produced
Making it yours
Why build this when Prowler and Trusted Advisor exist?
What the docs do not tell you

1. What an agent is made of

Strip away the hype and an agent is one JSON file. The filename minus .json is the agent's name. The file describes a chat session: which model to use, what tools it can call, what it is allowed to do without asking you, what extra powers it plugs in, and what knowledge it carries.

Six pieces. That is the entire mental model.

Piece	Field	Plain meaning
Identity	`name`, `description`	What it is called
The brain	`model`	Which LLM answers
Instructions	`prompt`	Its personality and rules
What it can do	`tools`	The toolbox
What runs without asking	`allowedTools`	Pre-signed permission slips
Extra powers	`mcpServers`	Plug in tool servers
Its knowledge	`resources`	Attach skills and files

The one distinction that trips up every beginner is tools versus allowedTools.

tools answers "what CAN this agent use?" If a tool is not listed, it does not exist for the agent.

allowedTools answers "what runs WITHOUT stopping to ask me?" A tool that is in tools but not in allowedTools still works, it just prompts you for approval each time it fires.

Toolbox versus permission slips. Keep that image and the rest is easy.

2. The safety foundation: read-only IAM

Before any config, we build the guardrail. This step is not optional and it is the reason the whole thing is trustworthy.

We attach two AWS-managed policies to the identity the agent uses:

arn:aws:iam::aws:policy/SecurityAudit
arn:aws:iam::aws:policy/job-function/ViewOnlyAccess

SecurityAudit is the policy AWS designed for exactly this job: reading security-relevant configuration across services. ViewOnlyAccess fills the cost gaps, so the agent can see Elastic IPs, volumes, snapshots, and load balancers.

Underneath, both policies are Get*, List*, and Describe* only. There is no Create, no Delete, no Put, no Modify anywhere in them.

aws iam create-user --user-name aws-auditor

aws iam attach-user-policy --user-name aws-auditor \
  --policy-arn arn:aws:iam::aws:policy/SecurityAudit

aws iam attach-user-policy --user-name aws-auditor \
  --policy-arn arn:aws:iam::aws:policy/job-function/ViewOnlyAccess

aws iam create-access-key --user-name aws-auditor
# then paste the keys into: aws configure --profile aws-auditor

Why go through this instead of just telling the model "please do not change anything"?

Because a prompt is a suggestion and IAM is a wall. If the model hallucinates a fix, IAM blocks it. If someone slips a "now delete that bucket" instruction into a file the agent reads, IAM blocks it. The blast radius is zero by construction. You are separating the act of detecting problems from the act of fixing them, which is a security best practice on its own.

The agent has read-only glasses, not a wrench.

3. Writing the agent config

Here is the complete file. Save it as ~/.kiro/agents/aws-auditor.json.

{
  "$schema": "https://raw.githubusercontent.com/aws/amazon-q-developer-cli/refs/heads/main/schemas/agent-v1.json",
  "name": "aws-auditor",
  "description": "Read-only agent that audits an AWS account for security risks and wasted spend, and reports prioritized, cited findings. Never changes anything.",
  "model": "auto",
  "mcpServers": {
    "security": {
      "command": "uvx",
      "args": ["awslabs.well-architected-security-mcp-server@latest"],
      "env": { "AWS_PROFILE": "default", "AWS_REGION": "us-east-1", "FASTMCP_LOG_LEVEL": "ERROR" }
    },
    "cloudtrail": {
      "command": "uvx",
      "args": ["awslabs.cloudtrail-mcp-server@latest"],
      "env": { "AWS_PROFILE": "default", "AWS_REGION": "us-east-1", "FASTMCP_LOG_LEVEL": "ERROR" }
    },
    "pricing": {
      "command": "uvx",
      "args": ["awslabs.aws-pricing-mcp-server@latest"],
      "env": { "AWS_PROFILE": "default", "AWS_REGION": "us-east-1", "FASTMCP_LOG_LEVEL": "ERROR" }
    },
    "awsdocs": {
      "command": "uvx",
      "args": ["awslabs.aws-documentation-mcp-server@latest"],
      "env": { "FASTMCP_LOG_LEVEL": "ERROR" }
    }
  },
  "tools": ["fs_read", "use_aws", "@security", "@cloudtrail", "@pricing", "@awsdocs"],
  "allowedTools": ["fs_read", "use_aws", "@security", "@cloudtrail", "@pricing", "@awsdocs"],
  "toolsSettings": {
    "use_aws": {
      "allowedServices": ["iam", "s3", "s3api", "ec2", "rds", "guardduty", "accessanalyzer", "securityhub", "elbv2", "elasticloadbalancing", "cloudwatch", "sts"]
    }
  },
  "resources": ["skill://~/.kiro/skills/aws-audit/SKILL.md"],
  "prompt": "You are AWS Auditor, a read-only cloud security and cost reviewer. You have ONLY read permissions (SecurityAudit + ViewOnlyAccess). You cannot and must not attempt to change anything, and you must not recommend that you run a fix yourself. Run the checks defined in your aws-audit skill using the available tools. Cite ONLY real resources you actually observed in tool output (real IDs, real names). NEVER invent a finding, a resource, or a number. If a check could not run, put it in the Gaps section as 'not verified' rather than implying it passed. Produce the report exactly in the format your skill defines. For every cost finding include an estimated monthly dollar impact computed from real pricing data, and state the assumption you used. Recommend fixes; never run them.",
  "welcomeMessage": "AWS Auditor here (read-only). Point me at a region and I will report what is risky and what is wasteful, with real evidence. I cannot change anything."
}

Walk through the fields that carry weight.

model is set to auto, which lets the CLI pick the model. You can pin one (run /model in a session to see valid IDs) but auto is the sensible default.

prompt is where the honesty rules live. Read it closely. "Cite ONLY real resources you actually observed." "NEVER invent a finding, a resource, or a number." "If a check could not run, put it in the Gaps section as not verified rather than implying it passed." Those three lines are what separate a useful audit from a confident-sounding hallucination.

toolsSettings.use_aws.allowedServices is a second wall on top of IAM. Even though IAM already blocks writes, this restricts the use_aws tool to a specific list of services it can even attempt to call. Two independent limits: the tool can only reach these services, and IAM only permits reads inside them. Defense in depth.

Notice tools and allowedTools are identical here. That means the agent runs the audit end to end without stopping to ask permission for each read. That is a deliberate trade-off: convenience for a demo, and it is safe precisely because every one of those tools is read-only. If you were doing anything with write access, you would keep the risky tools out of allowedTools so they prompt you.

4. The skill: your audit checklist

The agent config is the wiring. The skill is the brain of the audit. It is a plain Markdown file at ~/.kiro/skills/aws-audit/SKILL.md, attached through the resources field, and the agent reads it on every run.

This is the file you will edit most. It holds the checks, the severity model, the framework mapping, and the exact output format.

The security checks, each backed by a specific read-only API:

Check	Read-only API	Severity
Public S3 bucket	`s3:GetPublicAccessBlock`, `s3:GetBucketPolicyStatus`	CRITICAL
Root account MFA off	`iam:GetAccountSummary`	CRITICAL
Security group open to 0.0.0.0/0 on 22 / 3389 / DB	`ec2:DescribeSecurityGroups`	HIGH
Unencrypted EBS or RDS	`ec2:DescribeVolumes`, `rds:DescribeDBInstances`	HIGH
IAM users without MFA	`iam:ListUsers`, `iam:ListMFADevices`	HIGH
GuardDuty disabled	`guardduty:ListDetectors`	HIGH
Access Analyzer disabled	`accessanalyzer:ListAnalyzers`	MEDIUM
Security Hub standards incomplete	`securityhub:GetEnabledStandards`	MEDIUM

The cost checks, each of which must carry a real dollar figure:

Check	Read-only API
Unattached EBS volume	`ec2:DescribeVolumes` (state = available)
Unassociated Elastic IP	`ec2:DescribeAddresses`
Old or orphaned snapshot	`ec2:DescribeSnapshots`
gp2 volume that should be gp3	`ec2:DescribeVolumes` (VolumeType = gp2)
Idle load balancer	`elbv2:DescribeLoadBalancers` plus target health

The skill tells the agent how to think about severity (a traffic-light model), which frameworks to cite (Well-Architected Security Pillar for structure, CIS AWS Foundations Benchmark for authority, Trusted Advisor for the cost category), and the precise report shape: executive summary, a Top 3 list, findings grouped by Security and Cost, passing checks, and an honest gaps section.

One design choice worth calling out. The skill ships with a small "demo scope" block that tells the agent to only report resources tagged demo-auditor=true. That keeps a demo repeatable and stops it from surfacing anything real. To audit your whole account, you delete that one block. That is the entire difference between "show me the demo" and "audit everything."

5. Wiring the AWS tools with MCP

The agent needs eyes. MCP (Model Context Protocol) servers are how it sees AWS. Each server in the mcpServers block is a small program that exposes a set of tools, and you reference all of a server's tools with an @ prefix, like @pricing.

This build uses four:

@security (Well-Architected security MCP): checks GuardDuty, Security Hub, Access Analyzer.
@cloudtrail: queries account activity.
@pricing: pulls live rates from the AWS Price List API, which is how cost findings get real numbers instead of guesses.
@awsdocs: reads AWS documentation when the agent needs to confirm a detail.

They run through uvx, so you need uv installed (pip install uv). The first time you launch the agent, uvx fetches each server. No manual install step, no Docker.

The general-purpose use_aws tool covers everything else with direct read-only API calls. Between use_aws and the four MCP servers, the agent can reach every check in the skill.

6. Running it, and what it produced

Install the two files, then start the agent:

mkdir -p ~/.kiro/skills/aws-audit ~/.kiro/agents
cp skill/SKILL.md ~/.kiro/skills/aws-audit/SKILL.md
cp agent/aws-auditor.json ~/.kiro/agents/aws-auditor.json

kirocrew chat --agent aws-auditor

On the Amazon Q Developer CLI the command is q chat --agent aws-auditor, and you put the agent in ~/.aws/amazonq/cli-agents/ with the resources entry changed from skill:// to a file:// path.

Then ask it plainly:

Audit us-east-1. Actually call the tools and produce the report.

I ran this against five resources I created on purpose to be broken (the repo has a script for that, plus a teardown script that removes 100% of them). Here is the real output, trimmed. Nothing here is edited for effect.

# AWS Audit - us-east-1, 2026-09-02

## Executive summary
Scoped to the 5 resources tagged demo-auditor=true, this audit found 8 findings:
1 CRITICAL, 3 HIGH, 2 MEDIUM, and 3 cost items. The single most urgent problem is a
publicly readable S3 bucket (demo-auditor-public-5403) that lets anyone on the
internet download its objects.

## Top 3 - do these now
1. Public S3 bucket demo-auditor-public-5403 - bucket policy grants s3:GetObject to
   everyone (Principal: *). Enable Public Access Block and remove the public policy.
2. Security group sg-055250cbcc6f3b37b - SSH port 22 is open to 0.0.0.0/0. Restrict
   to a known admin IP or use SSM Session Manager.
3. GuardDuty is disabled in us-east-1 - no threat detection is running.

The cost section is where the pricing MCP earns its place:

### [COST] Unassociated Elastic IP - ~$3.65/month
- Resource: eipalloc-000992c9956cfaaa5 (public IP 35.173.72.149, no association)
- Estimated impact: $0.005/hr (USE1-PublicIPv4:IdleAddress, us-east-1) x 730 hrs
  = $3.65/mo. Assumption: idle for a full month, on-demand.
- Fix: Release the Elastic IP if not needed, or associate it with a running resource.

Every resource ID is real. Every rate came from the live Price List API. The agent computed the numbers, it did not make them up.

The most convincing part was not a finding, though. It was the honesty. Three moments stood out:

The agent found a snapshot created that same day. The checklist looks for "old" snapshots. Instead of forcing it into the finding, the agent flagged it for completeness and refused to call a fresh snapshot old.

Root MFA was on. It reported that as a PASS. A tool that only ever finds problems just confirms its own bias. Reporting passes is how you know it looked.

And the gaps section listed what it did not check and why: IAM per-user MFA, RDS encryption, and idle load balancers were out of scope for the demo, so it said "not verified" rather than implying those passed. "Not verified" is not "passed." That line in the prompt did real work.

The whole run cost a few cents in demo resources and about two minutes of wall time.

7. Making it yours

The SKILL.md file is yours to own. It is a checklist, not code.

Add checks your team cares about. Each row names the read-only API that backs it, so extending it is a matter of adding a row and a line of guidance. Change the severities to match your risk appetite. Rewrite the report format if your manager wants it a certain way. Delete the demo-scope block to audit the entire region.

A few natural next steps once it works:

Point it at more regions. The demo is us-east-1 only. Loop the region in the prompt or run it per region.

Schedule it. A read-only agent that runs every morning and mails you a diff of new findings is a genuinely useful thing, and it cannot break anything overnight because it cannot write.

Widen the checklist toward a framework you report against, like the full CIS benchmark, one row at a time.

Why build this when Prowler and Trusted Advisor exist?

Fair question. There are mature tools in this space, and you should know when to reach for them instead.

Prowler is the heavyweight: 600+ checks, mapped to CIS, PCI, HIPAA, and more. If you need exhaustive compliance coverage for an audit, use Prowler. The trade-off is that 600 findings with no narrative is a wall of text. It tells you everything and prioritizes nothing.

ScoutSuite is read-only like our agent and produces a nice HTML report. It is excellent for a point-in-time config review. It does no cost analysis and it is static, not conversational. You cannot ask it a follow-up.

Trusted Advisor has the best native cost checks (idle load balancers, unassociated Elastic IPs, underutilized EBS). The catch: the full cost category requires a Business or Enterprise Support plan (AWS docs). On a Basic plan you do not get them, which is exactly why an agent computing the same things from raw Describe calls is useful to a beginner.

Security Hub is the central dashboard, but it must be configured. Our demo run caught the trap live: the standards were subscribed but reported NO_AVAILABLE_CONFIGURATION_RECORDER. Without an AWS Config recorder, most controls cannot evaluate (AWS docs). The dashboard was on, the checks were off. A human skims a green dashboard and moves on. The agent read the actual status and flagged it.

So here is the decision framework:

Reach for	When
Prowler	You need exhaustive, framework-mapped compliance evidence for an audit
ScoutSuite	You want a thorough static config snapshot, security only
Trusted Advisor	You are on Business/Enterprise Support and want native cost checks
This agent	You want security AND cost in one plain-language, prioritized report you can converse with, on any support plan, provably read-only

The edge of the custom agent is not raw coverage. It is clarity, ruthless prioritization (a Top 3, not 600 rows), security and cost in one voice, detecting the "configured but inert" trap, and a provable read-only guarantee you can hand to a nervous manager.

When NOT to use it: if you need certified compliance evidence, if you want continuous automated remediation (this agent only reads), or if your org already runs Prowler in CI and just needs the raw findings. This is a fast, human-friendly first look, not a compliance system of record.

What the docs do not tell you

Four things I hit building this that are not in any single doc:

The use_aws service allowlist is a real second wall, and it is easy to forget s3api. S3 read calls split between s3 and s3api depending on the operation. Leave s3api out of allowedServices and the public-bucket check silently cannot run. Both belong in the list.

"Not verified" has to be forced in the prompt or the model will paper over gaps. Without the explicit "put it in the Gaps section as not verified" instruction, models tend to imply a skipped check passed. That single sentence changed the behavior in testing.

Idle Elastic IP pricing hides behind a specific usage type. The rate is not under a generic "EIP" filter. It is USE1-PublicIPv4:IdleAddress in us-east-1 (since the Feb 2024 public IPv4 charge). If your cost math comes back empty, you are querying the wrong usage type.

Clean up your demo resources. If you use the demo scripts, the teardown deletes by recorded ID and then sweeps by tag, in this order: snapshot, Elastic IP, volume, security group, bucket. Run it right after, or you keep paying the few cents a month the audit just flagged. The irony writes itself.

The takeaway

An agent is not a mystery. It is a JSON file that names a model, lists some tools, plugs in a few MCP servers, and points at a Markdown checklist. The engineering that makes it trustworthy is not in the model at all. It is in the IAM boundary that makes destructive action impossible, and in a prompt that forbids inventing anything.

Build the guardrail first. Then the agent can be as capable as you like, because the worst it can do is tell you the truth about your account.

Grab the two files, attach the two read-only policies, and run it against your own account: github.com/simplynadaf/aws-auditor-agent.

What would you add to the checklist first, security or cost? I am curious which one bites people more in practice.

AI Agent vs Agentic AI: The Distinction That Changes Your Architecture

Sarvar Nadaf — Fri, 11 Sep 2026 12:59:53 +0000

I was on a call last month where a VP said "we're deploying agentic AI" and what they actually had was a single chatbot connected to a database. That's an agent. A good one, maybe. But calling it "agentic AI" is like calling a single REST endpoint "a microservices architecture." That confusion cost them three months and a budget overrun before anyone caught it.

An AI agent is a thing you build. Agentic AI is how you wire many of those things together. One is a worker. The other is the factory floor.

If you missed Part 1 (AI Assistance vs AI Agents), go read that first. It sets the foundation for what we're covering today.

Why this confusion is dangerous
AI agents: the specialist worker
Agentic AI: the system architecture
The quick comparison
When agentic AI goes wrong
The governance gap nobody talks about
Real-world examples: AWS and beyond
When to use each
The bottom line

Why this confusion is dangerous

You buy an agent when you need a system. A vendor shows a slick demo: one agent handling one task beautifully. You deploy it expecting end-to-end workflow automation. Three months later, you're wondering why it can't handle anything beyond that one task.

You over-engineer when you just need an agent. I've seen teams spend months building multi-agent orchestration for what was essentially a ticket-routing problem. A single well-configured agent would've shipped in a week.

You skip governance entirely. Deploy a fleet of agents with no coordination layer, no monitoring, no human oversight. I've watched that go sideways at 2 AM.

The VP on that call? Their "agentic AI" couldn't even hand off context between conversations. It reset every session. That's a stateless function with a chat interface.

AI agents: the specialist worker

An AI agent is a piece of software that does a specific job on its own. It perceives, decides, uses tools, and takes action within clear boundaries for a defined task. One contractor, one job. They show up, do the thing, leave.

What separates an agent from a fancy API call:

Decides how to approach the task (not just following a script)
Can call tools: APIs, databases, search, code execution
Works autonomously until it hits a "done" state
Has guardrails but makes choices within them

You: "Summarize the latest security findings for my AWS account."

AI Agent: *connects to AWS Security Hub*
          *retrieves findings*
          *categorizes by severity*
          *generates summary*

"Here are your findings:
- 3 Critical: Unencrypted S3 buckets (bucket-a, bucket-b, bucket-c)
- 7 High: Security groups with open ports
- 12 Medium: IAM users without MFA

Would you like me to generate remediation steps?"

One agent. One job. Done well.

Agentic AI: the system architecture

Agentic AI isn't a single agent. It's the design approach for building systems where multiple agents work together with planning, coordination, evaluation, and governance layered on top.

If an AI agent is one contractor, agentic AI is the general contractor running the entire construction site: scheduling the specialists, checking their work, handling exceptions, making sure the building doesn't fall down.

What makes it agentic (not just "multiple agents"):

A planning layer that breaks goals into steps
An orchestrator that sequences and delegates
Evaluators that check whether output is actually good
Memory that persists across runs
Policy enforcement and human-in-the-loop checkpoints
Self-correction when things go sideways

┌─────────────────────────────────────────────┐
│              AGENTIC AI SYSTEM               │
├─────────────────────────────────────────────┤
│                                             │
│  ┌─────────────┐    ┌──────────────────┐   │
│  │   Planner   │───▶│   Orchestrator   │   │
│  └─────────────┘    └──────────────────┘   │
│                            │                │
│         ┌──────────────────┼────────┐       │
│         ▼                  ▼        ▼       │
│  ┌────────────┐  ┌────────────┐ ┌───────┐  │
│  │  Agent A   │  │  Agent B   │ │Agent C│  │
│  │(Research)  │  │(Analysis)  │ │(Write)│  │
│  └────────────┘  └────────────┘ └───────┘  │
│         │                  │        │       │
│         └──────────────────┼────────┘       │
│                            ▼                │
│                   ┌──────────────┐          │
│                   │  Evaluator   │          │
│                   └──────────────┘          │
│                            │                │
│                   ┌──────────────┐          │
│                   │   Memory &   │          │
│                   │  Governance  │          │
│                   └──────────────┘          │
│                                             │
└─────────────────────────────────────────────┘

Real-world example: Incident Response System

I helped architect this for a financial services client. Before the agentic system, their mean-time-to-resolution for P1 incidents was 47 minutes. After: 11 minutes. The system paid for itself in the first month from reduced downtime costs alone.

You: "Handle the production incident on the payments service."

Agentic AI System:

  Planner: "This needs investigation, diagnosis, action, 
            and communication."

  Agent 1 (Monitor): *checks metrics, spots CPU spike at 2:03 AM*
  Agent 2 (Logs):    *analyzes logs... initially flags DB connection 
                      issue, evaluator finds low confidence*
  Agent 2 (retry):   *digs deeper, finds memory leak in v2.3.1*
  Agent 3 (Git):     *identifies the commit that introduced it*

  Evaluator: "Root cause confirmed: memory leak in commit abc123. 
              Recommending rollback."

  Orchestrator: "Rollback is destructive. Paging human."

  [Human approves]

  Agent 4 (Deploy): *rolls back to v2.3.0*
  Agent 5 (Verify): *confirms service restored*
  Agent 6 (Comms):  *posts to Slack, creates JIRA ticket, 
                     updates status page*

  Memory: *stores pattern for faster detection next time*

Agent 2 got it wrong the first time. The evaluator caught it. The system self-corrected. That's the difference between a collection of agents and an agentic system.

Think of it like Uber. You request a ride to the airport, flight in 90 minutes. Driver cancels 2 minutes later. A single agent just says "finding new driver..." and makes you wait. An agentic system? It widens the search radius, filters for high-completion drivers, calculates you'll still make your flight, texts you proactively, applies a discount for the inconvenience, flags the cancelling driver's pattern, and remembers to prioritize reliable drivers for your airport rides next time. Six agents coordinating. Context flowing between them. That's the gap.

The quick comparison

Dimension	AI Agent	Agentic AI
What it is	A software component	A system design paradigm
Scope	Single task	Multi-step, multi-agent orchestration
Analogy	One specialist employee	The entire organization
Memory	Often resets after each task	Persistent across interactions
Coordination	Works alone	Multiple agents collaborating
Quality control	Limited self-checking	Built-in evaluators and critics
Governance	Basic guardrails	Policy enforcement, audit trails
Cost	Low: single model + tools	Higher: orchestration overhead
Complexity	Low to medium	High
Time to value	Days to weeks	Weeks to months

Most of my clients are somewhere in between. They have 3-4 agents running independently with no coordination layer. Just agents in a room with no manager. That's where most companies are stuck in mid-2026.

The simplest analogy:

AI Agent = A single microservice. Does one thing well.

Agentic AI = A microservices architecture. The service mesh, orchestration, observability, and resilience patterns that make dozens of services work together.

Nobody calls a single Lambda function "a serverless architecture." Same logic.

When agentic AI goes wrong

Agentic systems fail in ways that single agents don't. I've seen all of these in production:

Agent conflict. Two agents fighting each other for 20 minutes. One scaling up EC2 instances for a traffic spike, the other scaling them down because cost threshold was breached. Back and forth. We burned $600 in compute before someone killed the loop at 2 AM.

Infinite loops. Agent A writes a draft. Agent B reviews it, rejects. Repeat 47 times. $180 for what should have been a $2 task. Nobody set a max iteration limit.

Cascading failures. One agent failed silently, returned partial results. Every agent downstream built on garbage data. Final output looked confident and was completely wrong. Six hours before anyone noticed.

How to prevent this:

Hard iteration limits and timeouts (we use max 5 retries, 120s timeout per agent)
Budget caps per workflow (kill it if it exceeds $X)
Clear priority rules when agents conflict (cost vs availability: which wins? Decide upfront)
Circuit breakers: if an agent fails, stop the pipeline, don't feed garbage downstream
Observability at every handoff (we log every inter-agent message with correlation IDs)

Skip these in a demo. Never in production.

The governance gap nobody talks about

With a single agent, governance is simple: guardrails on access, actions, and data. Configure once.

With an agentic system, governance becomes distributed. And this is where I see teams get burned:

Data boundaries. Agent A sees customer PII for ticket resolution. Agent B handles analytics, should never see PII. If the orchestrator passes context without filtering, compliance violation. Saw this at a healthcare client. HIPAA auditor caught it before production. Fix took two weeks of re-architecting the context-passing layer.

Approval chains. The combination of actions might need approval even if individual ones don't. Agent 1 finds vulnerability + Agent 2 auto-patches + Agent 3 deploys to production = nobody approved a production deployment. Each agent followed its own rules. The system violated yours.

Audit trails. Regulators want to know which component made which decision, what data informed it, who approved. Multi-agent systems need per-agent logging with correlation IDs across the entire workflow. Retrofitting this is painful. Ask me how I know.

Cost governance. Agentic systems spawn sub-tasks, retry loops, parallel workflows that compound. I've seen a $2 expected workflow cost $200 because three agents kept spawning sub-agents to "be thorough." You need budget enforcement at the orchestrator level, not per-agent.

In regulated industries (finance, healthcare, government), the governance architecture might take longer to design than the agents themselves. That's normal. That's also why most "agentic AI" demos fall apart when compliance asks questions.

Real-world examples: AWS and beyond

Single Agent: Amazon Bedrock Agent

Configure one agent with a foundation model, action groups, knowledge base, and guardrails. It handles one task: answering questions, processing orders, analyzing documents. Quick to deploy, clear boundaries, predictable costs.

Multi-Agent: Amazon Bedrock Multi-Agent Collaboration

Orchestrate multiple agents with a supervisor that plans and delegates, specialized sub-agents, shared memory, evaluation logic, and human approval workflows via Step Functions.

My framework recommendation (opinionated)

I've used four frameworks across client engagements this year. Here's my honest take:

Amazon Bedrock + AWS AgentCore is my default for enterprise clients. Not because it's the most elegant API (it's not), but because IAM boundaries, VPC isolation, CloudWatch observability, and compliance controls come out of the box. When a CISO asks "where does my data go?" I have an answer. With other frameworks, I'm building that answer from scratch.

For prototyping and proving a concept fast, LangGraph gets you there quickest if your team already knows LangChain. But I've debugged graph state issues at 2 AM twice now. Production-hardening it takes real effort.

The framework matters less than the architecture. Planning, evaluation, governance, memory, and human-in-the-loop: get those right and you can swap the underlying framework later. Get those wrong and no framework saves you.

When to use each

Single AI agent when: Task is bounded. One model plus tools handles it start to finish. You want it deployed this week. Think: chatbot, code reviewer, data extractor, alert responder.

Agentic AI when: Workflow crosses domains. Multiple specialists must coordinate. You need planning, self-evaluation, and adaptation. Governance matters. Think: incident response, claims processing, research pipelines, end-to-end DevOps automation.

The honest answer for 80% of teams I talk to: Start with a single agent. A well-built agent delivering value today beats a half-built agentic system delivering nothing for six months.

The maturity spectrum

Stage 1: Single AI Agent
         → Deploy in days. Immediate ROI on one task.

Stage 2: Multiple Independent Agents  
         → Each agent owns a domain. No coordination between them.
         → (Most companies are HERE in mid-2026)

Stage 3: Coordinated Multi-Agent System
         → Shared context, handoffs, basic orchestration.
         → Where the ROI multiplier kicks in.

Stage 4: Full Agentic AI
         → Planning, evaluation, governance, memory, self-correction.
         → 47-min incident resolution → 11-min. That kind of impact.

The jump from Stage 2 to 3 is where I spend most of my consulting time. It's not a technology problem. It's a "who owns the orchestration layer" problem. That's an org chart conversation, not a code review.

The bottom line

AI Agent = A component. One autonomous piece of software that gets a specific job done.

Agentic AI = An architecture. The system that plans, coordinates, evaluates, and governs multiple agents working together.

They're not competing. They're layers. You build agents; you architect agentic systems. The agents live inside the architecture.

The progression from Part 1:

AI Assistant  →  Tells you what to do (responds)
AI Agent      →  Does it for you (executes a task)
Agentic AI    →  Orchestrates multiple agents to achieve complex goals

The real question isn't "which one should I use?" It's: "Do I need one specialist, or a team of specialists with a manager?"

Not sure? Start with an agent. Prove value. Then evolve when the single agent hits a wall. You'll know because someone will say "can it also do X, Y, and Z while considering W?" That's your signal.

Check Part 3: Build Your First AI Agent in 30 Minutes - CrewAI + AWS Bedrock. Hands-on, full code, deploy in 30 minutes.

What stage is your company at right now? Stage 1, 2, 3, or 4? And what's the blocker keeping you from the next one? Drop it in the comments.

If this helped, a ❤️ or 🦄 helps other devs find it too. Follow me for more on AWS architecture, FinOps, and AI infrastructure.

How I Built an AI Agent That Cut My AWS Bill by 40% (CrewAI + Bedrock)

Sarvar Nadaf — Wed, 09 Sep 2026 13:09:58 +0000

Last week my AWS bill hit $300. I knew there was waste hiding somewhere. Forgotten EBS volumes, idle Elastic IPs, snapshots from six months ago that nobody remembered creating.

I could open Cost Explorer. Click through dashboards. Manually cross-reference resources.

Or I could let three AI agents do it in 60 seconds.

I built a multi-agent system with CrewAI and Amazon Bedrock that scans everything, identifies waste, and writes a prioritized report with exact dollar savings. It found $125/month I was burning for nothing.

Who This Is For

You run workloads on AWS. Your bill has crept up. You suspect there's waste but don't have time to audit every resource manually. You have 10+ resources running and haven't done a proper audit in 3 months. You want something that scans your account and tells you exactly what to delete, release, or downsize, with dollar amounts attached.

When this won't help: If your account has fewer than 5 resources or you're still in free tier, the overhead of setting this up isn't worth it. Just check Cost Explorer manually.

The Architecture
Why Not Just Use Trusted Advisor?
Why CrewAI + Bedrock
The Custom Tool
Running It
What It Found
Gotchas You'll Hit
Web UI (Bonus)
Key Lessons
Cleanup

The Architecture

Three agents, one pipeline:

Scanner → Optimizer → Report Writer
   ↓           ↓             ↓
AWS APIs    Reasoning    Executive Report

Each agent has a single job.

Agent 1: Cost Intelligence Analyst scans your account using boto3. EC2 instances, EBS volumes, Elastic IPs, snapshots, S3 buckets, and Cost Explorer data. Raw facts with exact numbers.

Agent 2: Optimization Strategist takes the scan results and identifies savings. Orphaned volumes that can be deleted. Unattached IPs burning $3.60/month each. gp2 volumes that should be gp3. Reserved Instance candidates running 24/7 on On-Demand.

Agent 3: Executive Report Writer produces a markdown report with prioritized actions, dollar amounts, and risk levels. The kind of thing you can hand to a CTO.

The pipeline runs sequentially. Each agent passes context to the next through CrewAI's task handoff.

Why Not Just Use Trusted Advisor?

Fair question. AWS already has cost tools. Here's why they weren't enough for me:

Tool	Limitation
Trusted Advisor (free tier)	Only 7 checks. Misses orphaned volumes, old snapshots, gp2→gp3 opportunities
Compute Optimizer	EC2 and Lambda only. No EBS, no EIPs, no S3 lifecycle gaps
Cost Explorer	Shows WHAT you spent, not WHAT TO DO about it
Infracost	Terraform-only. Useless if you clicked things in the console

The multi-agent approach covers all of these in one pass AND produces an actionable report with prioritized fixes. You get "delete vol-0abc123 to save $20/month" instead of a dashboard you have to interpret yourself.

Plus it runs on YOUR schedule. Cron it weekly. Get a fresh report every Monday morning.

Why CrewAI + Bedrock

I tried a single-prompt approach first. One giant prompt asking the LLM to scan AND analyze AND write a report. The results were mediocre. The model tried to do everything and did nothing well. It hallucinated resource IDs that didn't exist because it was juggling too many concerns at once.

Multi-agent fixes this. Each agent has a focused role, a specific backstory, and constrained output expectations. The scanner doesn't try to optimize. The optimizer doesn't write pretty reports.

Amazon Bedrock Nova Pro handles the reasoning. On EC2, the IAM role handles auth automatically. No API keys to manage:

from crewai import LLM

llm = LLM(
    model="bedrock/amazon.nova-pro-v1:0",
    temperature=0.2,
)

Cost per run: about $0.01. The savings it finds will be 100-1000x that.

The Custom Tool: Scanning AWS

CrewAI agents need tools to interact with the world. I built one custom tool that wraps boto3:

from crewai.tools import BaseTool

class AWSCostScannerTool(BaseTool):
    name: str = "AWS Cost Scanner"
    description: str = (
        "Scans your AWS account for running resources and costs."
    )

    def _run(self, scan_type: str = "all") -> str:
        results = {}
        results["ec2_instances"] = self._scan_ec2()
        results["ebs_volumes"] = self._scan_ebs()
        results["elastic_ips"] = self._scan_eips()
        results["ebs_snapshots"] = self._scan_snapshots()
        results["s3_buckets"] = self._scan_s3()
        results["cost_data"] = self._scan_costs()
        return json.dumps(results, indent=2, default=str)

Each _scan_* method is a simple boto3 call. For example, _scan_ebs() runs ec2.describe_volumes() and flags any volume where Attachments is empty. That's an orphan. The scanner doesn't decide what to do about it. It just reports: "vol-0abc123, 20GB gp2, unattached, $2.00/month."

The optimizer doesn't need tools. It's pure reasoning. Takes the scan output and applies FinOps logic: "This volume has no attachments, it's orphaned, that's $20/month wasted."

Running It

git clone https://github.com/SimplyNadaf/crewai-aws-cost-optimizer-ai-agent.git
cd crewai-aws-cost-optimizer-ai-agent
pip3.11 install -r requirements.txt
export AWS_DEFAULT_REGION=us-east-1
python3.11 main.py

You'll see each agent working in sequence. The scanner calls AWS APIs, the optimizer reasons about the results, the report writer formats the final output.

Total time: under 60 seconds for a typical account.

What It Found on My Account

Three orphaned EBS volumes: $33/month. Two unattached Elastic IPs: $7.20/month. A running t3.medium that should be reserved: $82/month in potential savings. Two snapshots from January that nobody needed anymore.

Here's a trimmed version of the actual report output:

╭────────────────── 📊 Cost Optimization Report ──────────────╮
│                                                              │
│  Executive Summary                                           │
│  Current spend: $300/mo → Savings: $125/mo (41.7%)          │
│                                                              │
│  Top Savings Opportunities                                   │
│                                                              │
│   Priority  Resource             Action           Savings    │
│   ─────────────────────────────────────────────────────────  │
│   1         Orphaned EBS (×3)    Delete volumes   $33.00     │
│   2         Elastic IPs (×2)     Release          $7.20      │
│   3         EC2 i-0f7b...        Reserved (1yr)   $82.00     │
│   4         EBS vol-02b...       gp2 → gp3        $3.00      │
│                                                              │
│  Quick Wins (zero risk)                                      │
│   • Delete 3 orphaned EBS volumes → save $33/month           │
│   • Release 2 unused Elastic IPs → save $7.20/month         │
│                                                              │
╰──────────────────────────────────────────────────────────────╯

The quick wins (deleting orphaned volumes, releasing unused IPs) took five minutes to act on. $40/month saved before lunch.

Gotchas You'll Hit

Saved you the debugging time:

Python version matters. CrewAI requires 3.11+. If you're on Amazon Linux 2023, use python3.11 and pip3.11 explicitly. The default python3 is 3.9 and will throw ModuleNotFoundError.

Enable Nova Pro in Bedrock Console first. Go to Bedrock > Model access > Request access for Amazon Nova Pro. Takes 1-2 minutes to approve. Without this you'll get AccessDeniedException: You don't have access to the model.

Cost Explorer needs 24 hours. If you've never used Cost Explorer before, AWS needs ~24h to start collecting data. First run might return empty cost breakdowns. The agents still find orphaned resources, just no historical spend data.

Set the region as an env var, not just config. CrewAI reads AWS_DEFAULT_REGION from the environment, not from ~/.aws/config. Always export it explicitly or the boto3 calls will fail with NoRegionError.

The Web UI (Bonus)

I also built a Streamlit dashboard that wraps the same agents. One-click scan, visual findings, and remediation buttons that delete the orphaned resources for you.

But the CLI version is the core. No web server needed. SSH into any EC2 instance, clone, run, done.

Key Lessons

Separate scanning from reasoning. Agents with tools should gather data. Agents without tools should think. My first version had one agent doing both. It mixed up resource IDs, merged findings from different services, and produced a report with numbers that didn't add up.

Temperature 0.2 for cost analysis. I tried 0.7 first. The optimizer started "suggesting" resources that might exist and estimating savings based on vibes. At 0.2 it sticks to the facts the scanner reported. Nothing invented.

IAM roles over API keys. On EC2, there's zero credential management. The boto3 client picks up the instance role automatically. One less thing to configure, one less secret to leak. I've seen three repos on GitHub with AWS keys in their .env files pushed by accident. Don't be that person.

CrewAI's sequential process fits pipelines. When agents need each other's output, sequential beats hierarchical. Each task's output becomes the next task's context. Parallel would make sense if the agents were independent, but ours depend on each other's findings.

Cleanup

If you created demo resources for testing, remove them:

# Delete orphaned volumes the agents found
aws ec2 delete-volume --volume-id vol-xxx --region us-east-1

# Release unused Elastic IPs
aws ec2 release-address --allocation-id eipalloc-xxx --region us-east-1

# Or use the included cleanup script
bash scripts/cleanup-demo-resources.sh

⚠️ The demo script (scripts/create-demo-resources.sh) creates ~$40/month in dummy resources for testing. Always run cleanup after.

Try It

simplynadaf / crewai-aws-cost-optimizer-ai-agent

3 AI agents analyze your AWS account and find cost savings — powered by CrewAI + Amazon Bedrock

💰 AWS Cost Optimizer Crew

3 AI agents scan your AWS account, find waste, and produce an executive savings report - in under 60 seconds.

⭐ If this helped you, give it a star! It helps others find it.

Video Tutorial • Getting Started • How It Works • Demo • Contributing

🎬 Video Tutorial

Watch the full demo - from zero to finding $125/month in AWS waste:

▶️ Watch on YouTube: I Built an AI That Finds Hidden AWS Costs (It Found $125 in 60 Seconds)

In the video you'll see:

Browsing the source code and architecture on GitHub
Live deployment on an EC2 instance (Amazon Linux 2023)
All 3 agents working in real-time (Scanner → Optimizer → Report Writer)
Streamlit web dashboard with one-click remediation

🤔 The Problem

Your AWS bill keeps climbing. Resources get created and forgotten - orphaned EBS volumes, unattached Elastic IPs, old snapshots nobody…

View on GitHub

Two minutes to set up. A penny to run. Might save you hundreds.

What's eating YOUR AWS budget? Have you tried multi-agent approaches for infrastructure tasks? Drop your experience below.

ReachAloud: I built a multilingual voice tool that reads emergency alerts aloud for people who can't read the screen

Sarvar Nadaf — Mon, 07 Sep 2026 05:37:57 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

ReachAloud is a multilingual text-to-speech accessibility tool for nonprofits and communities. It turns a written emergency alert into clear, natural, spoken audio in any language, for the people that text-only alerts leave behind: those who can't read, can't see the screen, or don't speak the local language.

I want to be honest about the scope. ReachAloud is not an early-warning system. It does not detect floods, send alerts, or replace official channels. It is the last-mile comprehension layer. The warning already exists as an SMS, a banner, a siren. ReachAloud makes that warning understandable to everyone it reaches.

That distinction came from thinking about one event. In August 2026, more than 1,200 people were lost in the Nepal-Tibet floods, a wall of water that arrived with almost no usable warning. When a warning does go out, it usually goes out as text. But a written alert only helps the people who can read it, on a screen they can see, in a language they know. In a real crowd (an elderly villager, someone who is blind, a trekker who doesn't speak Nepali) the alert that could save a life arrives in a form they cannot use. ReachAloud is dedicated to those victims, and the site carries that dedication too. It exists so the next warning reaches everyone, in a voice they understand.

Here is what it does:

Speaks any alert in any language. One ElevenLabs multilingual model reads whatever text you type, in whatever language it is written.
Emergency broadcast mode. A full-screen, high-contrast takeover plays the alert aloud and cycles a giant caption through every language. Voice for people who can't read the screen. Huge text for people who can't hear it.
Play in every language back-to-back, for mixed crowds where locals and visitors need the same warning in turn.
Works offline. Once the page has loaded, a service worker keeps the alerts playing even when the network is down. That is exactly when a flood kills connectivity.
Download and share each alert as an MP3, for a WhatsApp group, a loudspeaker, or a printed QR poster at a trailhead.

Demo

Live: simplynadaf.github.io/reachaloud

No sign-up needed. Six pre-generated flood alerts (English, Nepali, Marathi, Hindi, Arabic, Chinese) play instantly with zero API calls. To hear your own text spoken, click Hear it now and paste a free ElevenLabs key.

Code

simplynadaf / reachaloud

Multilingual text-to-speech accessibility tool that turns emergency alerts into natural voice in any language. Built with ElevenLabs.

🔊 ReachAloud

Emergency alerts, spoken aloud in every language

The last-mile comprehension layer for emergency alerts. ReachAloud turns a written warning into clear, natural, offline-capable spoken audio in any language, for the people that text-only alerts leave behind: low-literacy, low-vision, and non-native readers.

▶️ Watch the 2-minute demo

Why this exists

When a flash flood hits, the warning goes out as text: an SMS, a banner, a push notification. But a written alert only helps the people who can read it, on a screen they can see, in a language they know. In a mixed, high-stress crowd (elderly villagers, someone who is blind, a trekker who does not speak the local language) the alert that could save a life arrives in a form they cannot use.

ReachAloud is dedicated to the more than 1,200 people lost in the 2026 Nepal-Tibet floods It does not detect disasters or send warnings.…

View on GitHub

The whole app is a single index.html (Tailwind + GSAP, no build step) plus a small serverless function and the pre-generated audio. Everything is MIT licensed.

How I Built It

The design goal was a demo that is impossible to break in front of a judge. It had to work with no API key and no network, while still proving live ElevenLabs voice on demand. That led to a two-mode architecture.

ElevenLabs is the load-bearing core

The whole product is the voice. For someone who can't read the screen, the audio is the deliverable, so the TTS can't be a nice-to-have bolted on the side. I used the eleven_multilingual_v2 model for one specific reason: it auto-detects the language of the text you give it. So the same code path speaks English, Nepali, Marathi, Hindi, Arabic, and Chinese with no per-language branching. You type the alert, ElevenLabs figures out the language, and the same reassuring voice reads it.

I picked a calm, default premade voice (Sarah) on purpose. In an emergency, panic in the delivery makes things worse. A steady voice is part of the accessibility.

Mode 1: pre-generated static clips (judge-proof)

I wrote a small Python script (scripts/generate_demo.py) that renders one flood-evacuation alert into all six languages once, then saves them as static MP3s plus a manifest.json the frontend reads. The whole six-language set costs about 670 characters of the monthly free quota. Generated once, then served forever with zero API calls.

MODEL_ID = "eleven_multilingual_v2"   # auto-detects the language of the text
VOICE_ID = "EXAVITQu4vr4xnSDxMaL"     # Sarah: reassuring, free-tier friendly

def tts(text):
    payload = {
        "text": text,
        "model_id": MODEL_ID,
        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75},
    }
    req = urllib.request.Request(
        f"{BASE}/text-to-speech/{VOICE_ID}",
        data=json.dumps(payload).encode(),
        headers={"xi-api-key": API_KEY, "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return resp.read()

This is what GitHub Pages serves. It works offline, needs no account, and never touches a quota.

Mode 2: live TTS with no backend at all

I wanted people to hear their own alert live, but GitHub Pages has no server to hide a key behind. So the Hear it now flow calls api.elevenlabs.io directly from the browser using a key the user pastes. The key lives only in localStorage (opt-in checkbox) and is sent only to ElevenLabs, never to me, because there is no "me" server in this path. The modal shows the free-tier facts (10,000 characters a month, no card) and links straight to the key page, so anyone can try it in under a minute.

There is also an optional Vercel proxy (api/speak.js) for anyone who would rather keep a shared key server-side. Both paths hit the same model.

The accessibility work is the interesting part

The emergency broadcast was the piece I cared most about getting right, because it has to serve two opposite disabilities at once:

People who can't see the screen get the alert as loud spoken voice, looping through every language.
People who can't hear get a giant, high-contrast caption that cycles in sync with the audio.

Getting it safe took work. It never auto-plays, because an unexpected siren is its own hazard. It reuses a single Web Audio context for the attention chime, since creating one per click hits the browser's context limit and throws. The loop is capped at three cycles so it always ends on its own. And it is dismissable three ways: a Stop button, the Escape key, or a click on the backdrop. Everything respects prefers-reduced-motion, focus is trapped and restored, and the dialogs use proper role="dialog" and aria-live semantics.

Offline, because that's when it matters

A flood is exactly when connectivity dies. A service worker pre-caches the app shell and all six clips, then serves them cache-first, so after one visit the alerts still play with no internet. One honest limitation: the CDN-loaded fonts and CSS do not cache, so styling degrades offline. The core alert audio, the part that saves someone, still plays.

Prize Categories

Best Use of ElevenLabs.

The voice is not a feature of ReachAloud. It is the whole product. The eleven_multilingual_v2 model is what lets one alert reach a low-literacy elder, a blind neighbor, and a foreign trekker in their own language, from the same box of text. I used it three ways: pre-rendered static clips for a bulletproof offline demo, direct browser-to-ElevenLabs calls for live bring-your-own-key TTS, and an optional server-side proxy. All of it on the free tier.

In memory of the victims of the August 2026 Nepal-Tibet floods.

My Dev.to CLI Got Its First Community PR. Image Uploads From Terminal.

Sarvar Nadaf — Thu, 03 Sep 2026 11:36:02 +0000

Three weeks after launching devpub, I got a notification I wasn't expecting: a pull request from someone I'd never talked to.

Harish / @harishteens had forked the repo, read the issues, picked one that I'd been putting off for weeks, and built a complete solution. Tests included. Design decisions documented. Edge cases handled.

The feature? devpub upload. The one command that should have existed from day one but couldn't, because the Forem API literally doesn't support it.

pip install devpub==0.3.0
devpub upload cover.png

That one command now gives you this:

  Uploaded: cover.png

              Uploaded Images
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ File         ┃ URL                                       ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ cover.png    │ https://hello.doclang.workers.dev-uploads.s3.amazonaws.com/… │
└──────────────┴──────────────────────────────────────────┘

No more opening dev.to/new in the browser just to drag an image and copy the URL.

The problem
How devpub upload works
The authentication problem
What Harish built
Try it
What's next

The problem: no image endpoint in the API

When I launched devpub in v0.1, the goal was to replace the Dev.to web editor entirely. Write locally, push with one command, track analytics. Done.

But there was a gap. Every time I wrote an article with diagrams or a cover image, I had to:

Open dev.to/new in the browser
Click the image upload button
Select the file
Wait for upload
Copy the URL
Paste it into my local markdown
Close the tab

Seven steps for something that should be devpub upload figure.png.

The reason I hadn't built this: the Forem API V1 has no image upload endpoint. It doesn't exist. You can set cover_image in article frontmatter, but only to a URL that already exists somewhere. The API cannot create that URL.

I filed issue #11 with a note saying "this requires reverse-engineering the web editor's upload mechanism" and moved on to other features.

Harish didn't move on. He figured it out.

How devpub upload works

The Dev.to web editor uploads images to POST /image_uploads. It's a multipart form submission, same as any file upload. But it's not authenticated with your API key. It uses your browser session.

# Upload a single image
devpub upload cover.png

# Upload multiple images
devpub upload *.png

# Get ready-to-paste Markdown
devpub upload architecture.png --markdown
# Output: ![architecture](https://hello.doclang.workers.dev-uploads.s3.amazonaws.com/uploads/...)

The --markdown flag is the one I use most. Write your article with ![](./figures/diagram.png), then run devpub upload figures/*.png --markdown and paste the output directly over your local references.

The authentication problem

Here's why this feature sat undone for three weeks. The /image_uploads endpoint requires:

A _Devto_Forem_Session cookie (your login session)
A CSRF token (anti-forgery protection)
An Origin header matching https://hello.doclang.workers.dev

Your API key? Useless. This endpoint doesn't accept it.

So devpub needs two extra credentials beyond your API key. They live in .devpub/.env (which devpub init gitignores by default):

DEVPUB_SESSION_COOKIE=_Devto_Forem_Session=abc123...
DEVPUB_CSRF_TOKEN=a1b2c3d4...

If you run devpub upload without setting these, it doesn't just fail with a cryptic error. It shows you exactly where to find them:

╭─ Session Credentials Required ─────────────────────────────╮
│                                                             │
│  devpub upload needs your browser session (not API key).    │
│                                                             │
│  1. Open https://hello.doclang.workers.dev/new in Chrome                       │
│  2. Press F12 → Application → Cookies → dev.to             │
│  3. Copy _Devto_Forem_Session value                         │
│  4. View page source → find <meta name="csrf-token">       │
│  5. Add both to .devpub/.env                                │
│                                                             │
╰─────────────────────────────────────────────────────────────╯

One deliberate design choice: these are login credentials, not a scoped token. The .env file is gitignored, and when they expire (you'll get a 401 or 403), you just re-copy them. devpub tells you that upfront rather than leaving you to guess why uploads suddenly stopped working.

What Harish built

Harish's PR wasn't a quick hack. It was a proper module with clear boundaries:

File	Role
`src/devpub/api/uploads.py`	`ImageUploader` class with all HTTP logic
`src/devpub/cli/images.py`	Click command + Rich table output
`tests/test_uploads.py`	29 tests, all HTTP-mocked with respx

Three things stood out in his implementation:

1. Fail before the network. File existence, extension validation, and size check (25 MB limit) all happen before any HTTP request fires. A typo in a filename costs zero network round-trips.

2. Flexible response parsing. Dev.to's upload response isn't documented, so the _extract_url method handles every response shape that's been observed: links.url, image.url, images[0], a raw string. If none match, it raises with the full response body included. A wrong URL silently landing in your article would be worse than a loud error.

3. Cookie flexibility. You can paste the full cookie header (a=1; b=2; _Devto_Forem_Session=xyz) or just the session value. Both work. Because copying one value out of DevTools is easy to get wrong.

The PR description was thorough. Design rationale for keeping ImageUploader separate from DevtoClient (different auth models shouldn't share a class). Explicit call-out that tests pin the request shape, not the live response. A suggestion to smoke-test against a real session before release.

This is what good open source contributions look like. Not just code that works, but code that explains why it works that way.

Other changes in v0.3.0

Beyond the upload feature:

Fixed hardcoded User-Agent: Was stuck at devpub/0.1.0. Now reads the actual version from importlib.metadata.
Fixed empty API key edge case: DevtoClient(api_key='') used to fall through to environment variables silently. Now it raises.
Repo cleanup: Removed personal draft articles from tracking, updated .gitignore for research docs and recordings.

Try it yourself

pip install devpub==0.3.0

For uploads, you need the session credentials (one-time setup):

# Add to .devpub/.env
DEVPUB_SESSION_COOKIE=your_session_cookie_here
DEVPUB_CSRF_TOKEN=your_csrf_token_here

Then:

devpub upload cover.png              # Upload and get URL
devpub upload *.png --markdown       # Ready-to-paste Markdown tags
devpub upload diagram.png --dry-run  # Validate without uploading

Full source: github.com/simplynadaf/devpub

What's next

Two follow-ups that Harish explicitly scoped out of his PR (smart -- ship the primitive first, then build on it):

Auto-rewrite during push: devpub push could detect local image paths like ![](./figures/arch.png), upload them automatically, and rewrite the URLs in-place before publishing.
Cover image shortcut: devpub push -f article.md --cover photo.png to upload the image and set cover_image in frontmatter in one step.

Both become trivial now that the upload primitive exists.

Shoutout

Big thanks to Harish @harishteens for the first external contribution to devpub. The PR was clean, well-tested, and properly documented. If you're looking for an open-source project to contribute to, devpub has open issues ranging from beginner-friendly to architecturally interesting.

What's your image workflow for Dev.to articles? Drag-and-drop in the browser? Hosted on GitHub? Imgur? I'd like to know if devpub upload fills a gap people feel, or if everyone's already solved this differently.