DEV Community: José Catalá

What 628,000 Flights Taught Me About Cancellations (And Why They're Harder to Track Than You Think)

José Catalá — Sat, 18 Apr 2026 19:26:45 +0000

Flight cancellations look simple from the outside: a flight was scheduled, it didn't happen, it shows "Cancelled." Done.

After processing 628,000+ flights across 1,300 airports, I can tell you it's not that simple. Cancellations are one of the messiest problems in airport data — and understanding them properly forced me to build detection infrastructure I didn't know I needed.

Here's what I learned.

The three types of cancellation (that most people treat as one)

Not all cancellations are the same, and the differences matter for both the traveller and the data pipeline.

Early cancellation: The airline cancels the flight hours or days before departure and updates the schedule. The flight board never shows it. From a scraping perspective, this flight simply doesn't appear. You can't detect it by watching the board — you'd need to compare against the original schedule, which you may not have.

Late cancellation: The flight is on the board with status "Scheduled" or "On time" and then, sometimes within 30 minutes of departure, flips to "Cancelled." This is the one passengers hate most, and it's the one your data pipeline needs to actively detect.

Ghost cancellation: The flight disappears from the board entirely without ever showing "Cancelled." The row was there in the previous scrape, it's gone now, and there's no explicit status change. This is the hardest one.

MyAirports encounters all three, and each needs a different detection strategy.

Ghost flights: the cancellation nobody tells you about

This was the most surprising thing I learned. A non-trivial number of airport websites simply remove a cancelled flight from their board rather than marking it as cancelled. It just vanishes.

If your scraper only looks at the current board state, you'll never see these cancellations. They happened, but the evidence is gone.

The fix is snapshot comparison:

// cancellation-detector.js
async function detectGhostCancellations(iata, direction) {
  const now = new Date();
  const windowMinutes = 90;

  // Get what we saw in the last scrape
  const prevScrape = await db.flight.findMany({
    where: {
      airportIata: iata,
      direction,
      lastSeen: { lt: new Date(now - 15 * 60 * 1000) }, // older than 15 min
      scheduledTime: {
        gte: now,
        lt: new Date(now.getTime() + windowMinutes * 60 * 1000)
      },
      status: { not: 'arrived' }
    }
  });

  // Get what's on the board now
  const currentBoard = await scrapeAirport(iata, direction);
  const currentFlightNumbers = new Set(currentBoard.map(f => f.flightNumber));

  // Anything that was there and is now gone
  const disappeared = prevScrape.filter(f => 
    !currentFlightNumbers.has(f.flightNumber) &&
    f.status !== 'cancelled' // not already marked
  );

  return disappeared.map(f => ({
    ...f,
    status: 'cancelled',
    cancellationReason: 'ghost', // disappeared without notice
    detectedAt: now
  }));
}

The window matters. A flight that departed won't show on the board either — you only want to flag flights that disappeared before their scheduled time. The 15-minute "last seen" buffer prevents false positives from temporary scrape failures.

Normalizing "Cancelled" across 30+ languages

Article 3 in this series covered status normalization in general. Cancellations specifically deserve attention because the variation is extreme.

A cancelled flight might show as:

Language	Status string
English	`Cancelled`, `Canceled`, `CNCL`, `CNX`, `CXL`
Spanish	`Cancelado`, `Anulado`, `CANCELADO`
German	`Gestrichen`, `Annulliert`, `Abgesagt`
French	`Annulé`, `Supprimé`
Italian	`Cancellato`, `Soppresso`
Portuguese	`Cancelado`, `Suprimido`
Russian	`Отменён`, `Отменен`
Arabic	`ملغي`, `ملغاة`
Turkish	`İptal`, `İptal Edildi`
Norwegian	`Kansellert`, `Innstilt`

And then there are airline-specific codes with no natural language at all: CNX, CXL, CNCL, XLD, CX. Some airports emit only the code without any label.

The normalizer handles this in layers:

// status-normalizer.js

const CANCELLED_EXACT = new Set([
  // English
  'cancelled', 'canceled', 'cancellation', 'cncl', 'cnx', 'cxl', 'xld', 'cx',
  // Spanish
  'cancelado', 'anulado',
  // German
  'gestrichen', 'annulliert', 'abgesagt',
  // French
  'annulé', 'annule', 'supprimé', 'supprime',
  // Italian
  'cancellato', 'soppresso',
  // Portuguese
  'cancelado', 'suprimido',
  // Dutch
  'geannuleerd', 'geannuleerd vlucht',
  // Norwegian/Swedish/Danish
  'kansellert', 'innstilt', 'inställd', 'aflyst',
  // Finnish
  'peruutettu',
  // Russian
  'отменён', 'отменен', 'отменен рейс',
  // Turkish
  'iptal', 'iptal edildi',
  // Arabic (normalized, RTL stripped)
  'ملغي', 'ملغاة',
  // Chinese
  '取消', '已取消',
]);

const CANCELLED_CONTAINS = [
  'cancel', 'annul', 'gestr', 'suprim', 'peruut', 'innstil',
];

function normalizeToCancelled(raw) {
  const s = raw.toLowerCase().trim().replace(/[^\p{L}\p{N}\s]/gu, '');

  if (CANCELLED_EXACT.has(s)) return 'cancelled';
  if (CANCELLED_CONTAINS.some(pat => s.includes(pat))) return 'cancelled';

  return null; // not a cancellation
}

The replace(/[^\p{L}\p{N}\s]/gu, '') strips punctuation and special characters using Unicode property escapes — critical for Arabic and Chinese status strings that sometimes arrive with RTL markers or extra punctuation.

Tracking cancellations over time: the `flights` table

Every scraped flight goes into the flights table. When a cancellation is detected — either from an explicit status change or a ghost detection — the flight record gets updated:

// persistence.js
async function saveFlights(iata, direction, flights) {
  for (const flight of flights) {
    await prisma.flight.upsert({
      where: { flightNumber_scheduledTime_airportIata: {
        flightNumber: flight.flightNumber,
        scheduledTime: flight.scheduledTime,
        airportIata: iata,
      }},
      update: {
        status: flight.status,
        lastSeen: new Date(),
        ...(flight.status === 'cancelled' && {
          cancelledAt: new Date(),
          cancellationReason: flight.cancellationReason || 'explicit',
        }),
      },
      create: {
        ...flight,
        firstSeen: new Date(),
        lastSeen: new Date(),
      }
    });
  }
}

The cancelledAt timestamp and cancellationReason fields are what make the historical analysis possible later.

Mass cancellation detection: weather, strikes, and cascades

Individual cancellations are noise. Mass cancellations are a signal worth surfacing.

When an airport has 10+ cancellations in a short window, something structural is happening: weather, strike action, airport closure, ground stop, or an airline operational meltdown. The data pattern is distinctive:

// anomaly-detector.js
async function detectMassCancellations(iata, windowHours = 2) {
  const since = new Date(Date.now() - windowHours * 60 * 60 * 1000);

  const recentCancellations = await prisma.flight.count({
    where: {
      airportIata: iata,
      status: 'cancelled',
      cancelledAt: { gte: since }
    }
  });

  const baseline = await getBaselineCancellationRate(iata, windowHours);

  if (recentCancellations > baseline * 3) {
    return {
      type: 'mass_cancellation',
      airport: iata,
      count: recentCancellations,
      baseline,
      deviationFactor: recentCancellations / baseline,
      detectedAt: new Date()
    };
  }

  return null;
}

async function getBaselineCancellationRate(iata, windowHours) {
  const dayOfWeek = new Date().getDay();

  const result = await prisma.$queryRaw`
    SELECT AVG(daily_count) as avg_count
    FROM (
      SELECT DATE(cancelled_at) as day, COUNT(*) as daily_count
      FROM flights
      WHERE airport_iata = ${iata}
        AND status = 'cancelled'
        AND EXTRACT(DOW FROM cancelled_at) = ${dayOfWeek}
        AND cancelled_at > NOW() - INTERVAL '30 days'
      GROUP BY DATE(cancelled_at)
    ) daily
  `;

  return result[0]?.avg_count ?? 1;
}

The 3x deviation threshold is empirical — below that and you get false positives from normal day-to-day variation. Above 3x is consistently meaningful.

What the data actually shows

After accumulating 628,000+ flights, the cancellation patterns are clear:

Cancellation rate by time of day — Early morning flights (05:00–07:00) cancel at roughly 1.2x the rate of afternoon flights. Not because airlines plan worse in the morning, but because cascades: a late-arriving aircraft the night before causes the early morning departure to cancel, and the delay propagates forward through the day.

Cancellation rate by day of week — Mondays and Fridays have the highest cancellation rates, roughly 1.4x midweek. This matches the business travel calendar: high demand creates less buffer for aircraft rotations.

The "cancellation cascade" pattern — When we detect a cancellation on a route (e.g., MAD→LHR at 06:30), the probability of a cancellation on the return (LHR→MAD, any time that day) jumps by roughly 40%. Same aircraft type, same crew. The cascade is real and detectable from the data alone.

Ghost cancellations by region — Ghost cancellations (flights that vanish without notice) are most common in South American and Southeast Asian airports — roughly 3x the rate of European and North American airports. The likely cause is older CMS software that just deletes cancelled entries rather than updating their status.

Surfacing this in the API

The delay_stats table in the insights engine stores cancelled percentage per route per time bucket:

// insights/analyzers/delay-stats.js (relevant section)
const cancelledPct = bucket.totalFlights > 0
  ? (bucket.cancelledCount / bucket.totalFlights) * 100
  : 0;

await prisma.delayStat.upsert({
  where: { /* composite key */ },
  update: { cancelledPct, /* other fields */ },
  create: { cancelledPct, /* other fields */ },
});

This feeds directly into the delayRisk field on every flight in the API response:

{
  "flightNumber": "IB3163",
  "scheduledTime": "2026-04-18T06:30:00Z",
  "status": "scheduled",
  "insights": {
    "delayRisk": {
      "onTimePct": 68,
      "avgDelayMinutes": 22,
      "cancelledPct": 4.7,
      "sample": 143,
      "context": "IB to LHR, Fri morning"
    }
  }
}

A cancelledPct of 4.7% on a Friday morning Iberia flight to London isn't alarming in isolation — but if you're booking travel insurance or a tight connection, it's exactly the number you want.

The EU261 connection

Under EU Regulation 261/2004, a cancellation notified fewer than 14 days before departure entitles the passenger to compensation: €250–€600 depending on route distance. The detection pipeline feeds directly into this.

When a flight cancels in our system with cancelledAt set to less than 14 days before scheduledTime, the EU261 eligibility flag gets set:

// eu261.js
function checkEu261Eligibility(flight) {
  if (flight.status !== 'cancelled') return null;

  const noticeHours = (flight.scheduledTime - flight.cancelledAt) / (1000 * 60 * 60);
  const noticeDays = noticeHours / 24;

  if (noticeDays >= 14) return { eligible: false, reason: 'advance_notice' };

  const distanceKm = getRouteDistance(flight.originIata, flight.destIata);

  return {
    eligible: true,
    compensationEur: distanceKm < 1500 ? 250 : distanceKm < 3500 ? 400 : 600,
    noticeDays: Math.round(noticeDays),
    distanceKm
  };
}

The catch: cancelledAt is when we detected the cancellation, not necessarily when the airline officially notified passengers. For ghost cancellations — the ones that disappeared without notice — we mark cancellationReason: 'ghost' and flag the EU261 check as needing manual verification.

What I'd do differently

Track the full status timeline. Right now, if a flight goes Scheduled → Delayed → Cancelled, I only have the final state and the timestamps. I should log every status transition. A flight that was delayed 4 times before finally cancelling tells a different story than one that cancelled cleanly.

Capture the board timestamp. Each scrape runs against live airport data, but the data is often cached on the airport side. I should record the "data as of" timestamp from the airport's API response, not just my scrape time.

Watch for reinstatements. Cancelled flights occasionally get reinstated — a replacement aircraft is found, the weather clears faster than expected. Right now, a reinstatement would just update the status from "cancelled" to "scheduled," but I don't surface this event explicitly.

Flight cancellations are a solved problem from the traveller's perspective — the airline's app tells you. But from the data perspective, they're a distributed, multilingual, inconsistent mess. The ghost cancellations alone took me three months to discover were happening.

The full cancellation API is available at myairports.online/developers. Free tier: 100 requests/day, no card required.

How I Automated Onboarding 1,300 Airport Websites Without Looking at a Single One

José Catalá — Thu, 16 Apr 2026 20:38:55 +0000

The original version of MyAirports required a developer (me) to open DevTools, find the network request carrying flight data, copy the URL, figure out the headers, and paste everything into a config file. One airport at a time.

That's fine for ten airports. It doesn't scale to a thousand.

So I built a pipeline that does it automatically: give it an IATA code, and it finds the airport's website, discovers the flight data API, detects the scraping strategy, and writes the config code. No DevTools. No manual inspection. Here's how it works.

The problem with airport websites

Every airport runs its own website. Some are operated by independent airport authorities. Some are run by national operators — Spain's AENA runs 46 airports on the same platform; Finland's Finavia runs 20. Some outsource to the same flight display software vendor.

This creates a few patterns worth detecting:

Operator groups: airports that share the same backend API. Find one, and you've implicitly found all of them.
Standalone airports: unique websites with unique APIs.
Vendor software: airports using products like AirportInfo or similar. Same HTML structure, same endpoints, different domain.
DOM-only: no discoverable API. Flight data is rendered server-side, so you scrape HTML.

The discovery pipeline needs to detect which pattern applies and generate the right config for each.

Step 1: Find the right URL

The first obstacle is simple: given an IATA code like ATH (Athens), what's the official airport website URL?

This sounds trivial but isn't. Search for "Athens airport" and you get aggregators — Skyscanner, TripAdvisor, Wikipedia — before you get the official site. The flight data lives on the official site, not on aggregators.

The solution is a purpose-built search query:

// searcher.js
async function findAirportUrl(iata, airportName, country) {
  const query = `"${airportName}" official airport website flight departures arrivals -skyscanner -tripadvisor -rome2rio`;

  const results = await serperSearch(query);
  return filterAggregators(results);
}

function filterAggregators(results) {
  const blocklist = [
    'skyscanner', 'tripadvisor', 'rome2rio', 'kayak', 'expedia',
    'flightaware', 'flightradar24', 'wikipedia', 'wikitravel',
    'airportia', 'flightsfrom', 'airports-worldwide',
    // 30+ more
  ];

  return results.filter(r => 
    !blocklist.some(blocked => r.url.includes(blocked))
  );
}

The blocklist has grown to 30+ domains over time — whenever a false positive slips through and breaks a discovery, the domain gets added. The negative search terms in the query help, but they're not reliable enough on their own.

For airports where Serper finds nothing useful, the pipeline falls back to in-browser DuckDuckGo search with the same filtering logic.

Step 2: Intercept network traffic

Once the URL is found, the pipeline launches a stealth Playwright browser and loads the page. Rather than looking at the HTML, it watches the network:

// interceptor.js
async function discoverApi(page) {
  const candidates = [];

  page.on('response', async (response) => {
    const url = response.url();
    const contentType = response.headers()['content-type'] || '';

    if (!contentType.includes('application/json')) return;

    try {
      const body = await response.json();
      const score = scoreResponse(body, url);
      if (score > 0) {
        candidates.push({ url, headers: response.request().headers(), score });
      }
    } catch (_) {}
  });

  await page.goto(targetUrl, { waitUntil: 'networkidle' });
  return candidates.sort((a, b) => b.score - a.score);
}

function scoreResponse(body, url) {
  let score = 0;
  const str = JSON.stringify(body).toLowerCase();

  if (str.includes('flight') || str.includes('flightnumber')) score += 3;
  if (str.includes('departure') || str.includes('arrival')) score += 3;
  if (str.includes('iata') || str.includes('icao')) score += 2;
  if (str.includes('status') || str.includes('gate')) score += 2;
  if (str.includes('airline') || str.includes('carrier')) score += 2;
  if (Array.isArray(body) && body.length > 0) score += 1;
  if (body.flights || body.departures || body.arrivals) score += 3;

  // URL indicators  
  if (url.includes('flight') || url.includes('fids')) score += 2;
  if (url.includes('departure') || url.includes('arrival')) score += 2;

  return score;
}

The highest-scoring response is the flight data API. The request headers that triggered it — including any session cookies, CSRF tokens, or API keys — get captured at the same time. One browser session is all it takes; from then on, the system calls the API directly over HTTP without a browser.

This is the core idea behind MyAirports: use a browser once to discover, then never again.

Step 3: WAF detection and FlareSolverr fallback

Before the browser even hits the target page, the pipeline checks for WAF protection:

// waf-detector.js
async function detectWaf(page) {
  const title = await page.title();
  const content = await page.content();

  if (title.includes('Just a moment') || content.includes('cf-browser-verification')) {
    return 'cloudflare';
  }
  if (content.includes('incapsula') || content.includes('visid_incap')) {
    return 'incapsula';
  }

  return null;
}

If Cloudflare is detected, the pipeline hands off to FlareSolverr — a Docker sidecar that runs a real browser, solves the JS challenge, and returns the cookies:

// flaresolverr.js
async function solveCloudflare(url) {
  const response = await fetch('http://localhost:8191/v1', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      cmd: 'request.get',
      url,
      maxTimeout: 60000
    })
  });

  const data = await response.json();
  return {
    cookies: data.solution.cookies,
    userAgent: data.solution.userAgent
  };
}

The solved cookies get stored in a Session table and reused until they expire, so the challenge-solving cost is paid once per airport, not on every scrape.

Step 4: Detect operator groups

Some airports share infrastructure. Spain's AENA airports all use the same API at aena.es/es/flight-search. Once you've discovered one AENA airport, you've got the pattern for all 46.

The group detector looks for these shared patterns:

// group-detector.js
const KNOWN_OPERATORS = [
  { name: 'AENA', urlPattern: /aena\.es/, airports: ['MAD', 'BCN', 'VLC', /* ... */] },
  { name: 'Finavia', urlPattern: /finavia\.fi/, airports: ['HEL', 'OUL', /* ... */] },
  { name: 'ADP', urlPattern: /parisaeroport\.fr/, airports: ['CDG', 'ORY'] },
];

function detectOperatorGroup(discoveredUrl) {
  for (const op of KNOWN_OPERATORS) {
    if (op.urlPattern.test(discoveredUrl)) return op;
  }
  return null;
}

When a group match is found, the code generator writes configs for all airports in the group from a single template.

Step 5: AI pattern learning for DOM-only airports

Some airports don't have a JSON API. Flight data is rendered in HTML tables or card layouts, with no network request to intercept.

The AI learner handles these:

// ai-learn.js
async function learnPattern(url) {
  const page = await launchStealthBrowser();
  await page.goto(url, { waitUntil: 'networkidle', timeout: 18000 });

  const html = await page.content();
  const trimmed = trimHtml(html, 15000); // fits in context window

  const prompt = `
    This is an airport flight board HTML page. 
    Identify CSS selectors for each flight row, flight number, 
    destination/origin, scheduled time, gate, and status.
    Also list any status text values and what they mean.
    Return JSON: { rowSelector, columns: { flightNumber, destination, time, gate, status }, statusMappings }
  `;

  const result = await deepseekChat(prompt, trimmed);
  return JSON.parse(result);
}

Successfully AI-learned airports so far:

Airport	Row selector	Layout
GYD (Baku)	`a.flt_row`	Card with `.fl_col.*` columns
KTW (Katowice)	`div.timetable__row.flight-board__row`	Table-like rows
TLL (Tallinn)	`li.flights-list__item`	List card
DUS (Düsseldorf)	(table class detection)	Standard table

Each of these would have taken 20–30 minutes of manual DevTools work. The AI learner does it in about 30 seconds.

Step 6: Auto-generate config code

Once the pattern is known, the code generator writes the actual config:

// code-generator.js
function generateAirportEntry(iata, name, url, strategy) {
  if (strategy.type === 'api') {
    return `
  ${iata}: {
    name: '${name.replace(/'/g, "\\'")}',
    url: '${url}',
    apiEndpoint: '${strategy.endpoint}',
    headers: ${JSON.stringify(strategy.headers, null, 6)},
  },`;
  }
}

For operator groups it generates a shared template and references:

function generateOperatorEntry(operator, airports) {
  return `
const ${operator.name.toLowerCase()}Config = {
  apiBase: '${operator.apiBase}',
  headers: ${JSON.stringify(operator.headers, null, 2)},
};

${airports.map(a => `  ${a}: { ...${operator.name.toLowerCase()}Config, iata: '${a}' },`).join('\n')}
`;
}

Running at scale: 1,300 airports

npm run auto-discover                    # process all airports
npm run auto-discover -- --country DE   # just one country
npm run auto-discover -- --skip 400     # resume from checkpoint

The tool reads 6,000+ entries from the OpenFlights world airports CSV, processes in batches of 10 with a 60-second timeout per airport, and writes configs directly to disk.

Results after one full run across 1,300 airports:

276 new airports discovered — APIs found, configs generated, immediately usable
~600 DOM-only airports — HTML scraping with multilingual status normalisation
~400 failures — heavy WAF, no official website, or page timeouts
Operator groups detected: AENA (46), Finavia (20), ADP (3), and several smaller groups

Total processing time: ~4 hours on a single VPS. Compared to 20–30 minutes per airport manually, that's roughly 400 hours of DevTools work replaced by a shell command.

What still breaks

The pipeline isn't perfect. Common failure modes:

Heavy bot protection: Akamai Bot Manager and Imperva with device fingerprinting go beyond what FlareSolverr can handle. These remain manual.

Dynamic API tokens: Some airports generate short-lived tokens that expire between scrapes. Discovery captures them, but they're invalid an hour later. These need re-discovery on every request, which is expensive.

iFrame-embedded boards: Some airports embed a third-party flight board in an iframe from a different domain. Network interception doesn't always capture those requests correctly.

No-English flight data: A few airports only publish in their local language (Cyrillic, Arabic, CJK). The status normaliser handles most of it, but field mapping still fails occasionally.

The auto-discovery pipeline turned a manual, one-by-one chore into a batch job. The 276 airports it found in one run would have taken months to add by hand. The failures are the interesting edge cases — and those are where the AI pattern learner earns its keep.

The API is free to try at myairports.online/developers. Free tier: 100 requests/day, no card required.

Two Auth Problems: How MyAirports Handles Identity on Both Sides of the API

José Catalá — Tue, 14 Apr 2026 12:42:50 +0000

Every API has an auth problem. Usually it's one: how do you verify that the person calling your API is allowed to?

Building MyAirports — a real-time flight data API covering 1,000+ airports — gave me two auth problems. The first is the standard one: verifying developers who use my API. The second is messier: authenticating against 1,000+ different airport APIs, each of which expects something different.

Here's how I solved both.

The outward-facing problem: clean developer auth

When a developer hits the MyAirports API, I want the experience to be familiar. You sign up, you get a key, you put it in a header, it works.

API key format

Keys follow this pattern:

ma_live_<32 random hex characters>

The ma_live_ prefix makes keys identifiable at a glance — in log files, in config dumps, in error messages. If you accidentally paste your key into a public repo, a secret scanning tool can spot it. This is the same pattern Stripe, Resend, and most modern API providers use, and for good reason.

Storage: never store the plaintext key

The key is generated once, shown to the user once, and then discarded. What gets stored in the database is the SHA-256 hash of the key:

// generation
const rawKey = `ma_live_${crypto.randomBytes(16).toString('hex')}`;
const hashedKey = crypto.createHash('sha256').update(rawKey).digest('hex');

// store only hashedKey in ApiKey table
// return rawKey to the user (once)

// lookup
const incomingHash = crypto.createHash('sha256').update(incomingKey).digest('hex');
const record = await prisma.apiKey.findFirst({ where: { hashedKey: incomingHash } });

This means a database breach doesn't expose live keys. The attacker gets hashes that are useless without the original plaintext.

Auth middleware: two paths into the system

The middleware checks in priority order:

// api/src/auth/middleware.js
export async function authMiddleware(req, res, next) {
  // Path 1: API key (developer access)
  const apiKey = req.headers['x-api-key'];
  if (apiKey) {
    const hashed = sha256(apiKey);
    const record = await prisma.apiKey.findFirst({ where: { hashedKey: hashed } });
    if (record) {
      req.user = await prisma.user.findUnique({ where: { id: record.userId } });
      return next();
    }
  }

  // Path 2: JWT cookie (dashboard/browser access)
  const token = req.cookies.accessToken;
  if (token) {
    try {
      const payload = jwt.verify(token, process.env.JWT_SECRET);
      req.user = payload;
      return next();
    } catch (_) {}
  }

  // Unauthenticated — public endpoints proceed, protected endpoints return 401
  next();
}

API keys handle programmatic access. JWTs handle the browser dashboard. The same middleware serves both, and there's no ambiguity about which wins — API key always takes precedence.

The JWT setup is standard: 1-hour access tokens, 30-day refresh tokens, both in httpOnly cookies. HS256, signed with JWT_SECRET. The short access token TTL limits blast radius if a token leaks; the refresh token means users don't re-authenticate constantly.

Rate limiting: in-memory counters, not DB hits

Rate limiting on every request is a performance concern. Hitting PostgreSQL on every API call to check a counter would add latency to every response. Instead:

// In-memory store — fast lookup
const counters = new Map(); // userId -> { count, date }

export function checkRateLimit(userId, plan) {
  const today = new Date().toISOString().split('T')[0];
  const entry = counters.get(userId) || { count: 0, date: today };

  if (entry.date !== today) {
    entry.count = 0;
    entry.date = today;
  }

  const limit = plan === 'pro' ? 1000 : 100;
  if (entry.count >= limit) {
    return { allowed: false, remaining: 0, limit };
  }

  entry.count++;
  counters.set(userId, entry);
  return { allowed: true, remaining: limit - entry.count, limit };
}

The in-memory counters are flushed to the ApiUsage PostgreSQL table every hour. This gives you persistence across restarts and a usage history for the dashboard, without a DB round-trip on every request.

Rate limit state is communicated back to callers via standard headers:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 47
X-RateLimit-Reset: 1712275200

And when the limit is exceeded:

HTTP 429 Too Many Requests
{
  "error": "Rate limit exceeded",
  "limit": 100,
  "remaining": 0,
  "resetAt": "2026-04-15T00:00:00Z"
}

The inward-facing problem: 1,000 different auth schemes

Now for the harder problem.

MyAirports scrapes flight data from airport websites to serve its API. Each airport website is different. And "different" doesn't just mean different data formats — it means different auth requirements too.

Here's a sample of what the scraper encounters:

No auth required — most airports. The flight data API is unauthenticated or only requires session cookies the browser already has.
Cloudflare JS challenge — the page won't load without executing a JavaScript fingerprinting challenge first. No challenge, no data.
Incapsula WAF — similar JS challenge, different vendor.
CSRF tokens — some airport APIs require a valid CSRF token extracted from the page before accepting data requests.
Leaked third-party API keys — during browser discovery, the scraper sometimes intercepts network requests that include API keys in headers. Airports that license flight data software sometimes include the vendor's API key in client-side requests.

The last one deserves its own Prisma model:

model LeakedKey {
  id          Int      @id @default(autoincrement())
  airportIata String
  keyName     String   // e.g. "X-API-Key", "Authorization"
  keyValue    String   // the actual leaked key value
  source      String   // URL it was found in
  tested      Boolean  @default(false)
  working     Boolean  @default(false)
  createdAt   DateTime @default(now())
}

When the interceptor finds an API key embedded in a request header during discovery, it logs it here. Some of these keys work for direct calls to the upstream flight data vendor — an interesting optimization, though using them raises its own questions.

WAF session persistence

Solving a Cloudflare or Incapsula challenge is expensive: it requires a full browser session, human-like behavior simulation, and sometimes a FlareSolverr sidecar to handle the cryptographic JS challenge:

docker run -d \
  --name flaresolverr \
  -p 8191:8191 \
  ghcr.io/flaresolverr/flaresolverr:latest

Paying that cost on every scrape is wasteful. Instead, solved cookies are stored in the Session table and reused:

model Session {
  id          Int      @id @default(autoincrement())
  airportIata String   @unique
  cookies     Json     // solved WAF cookies
  userAgent   String   // must match the browser that solved the challenge
  expiresAt   DateTime
  createdAt   DateTime @default(now())
}

The session store checks for a valid unexpired session before launching a browser. A session that's still valid skips the challenge entirely — the scraper just presents the saved cookies.

The anti-bot evasion stack

Beyond WAF cookies, the browser sessions themselves need to look human. The stealth layer manages six distinct browser profiles — each with a coherent set of navigator properties, WebGL fingerprints, HTTP headers, and behavioral patterns.

The profiles are designed to look like real returning users rather than fresh bot fingerprints. Persistent Chromium user-data directories reinforce this — the browser accumulates cookies and browsing history across scrapes, which is a signal many WAFs check.

Stale session eviction

Sessions expire. WAF cookies typically last 24-48 hours. The system evicts sessions as they expire and re-solves challenges when needed.

There's a parallel mechanism for discovered API endpoints: the ApiCache table tracks success and failure counts per endpoint. An endpoint that returns zero flights three times in a row gets flagged as stale:

model ApiCache {
  id           Int      @id @default(autoincrement())
  airportIata  String   @unique
  endpoint     String
  headers      Json
  successCount Int      @default(0)
  failCount    Int      @default(0)
  lastUsed     DateTime
  stale        Boolean  @default(false)
}

A stale endpoint triggers re-discovery — the browser fires up again, re-intercepts the network traffic, and hopefully finds the new API URL.

The architecture of two auth systems

Stepping back: the MyAirports API is a thin, clean auth layer sitting on top of a chaotic, unpredictable auth environment.

Outward-facing: developers see a consistent, well-documented interface. API keys in headers, rate limits in response headers, 429s when limits are hit. Predictable.

Inward-facing: the scraper deals with a different kind of problem every hour. No auth, WAF challenges, leaked keys, CSRF tokens, expired sessions. Unpredictable by design.

The key insight is that these two problems need to be separated architecturally. The developer-facing API layer doesn't know or care about WAF sessions — that complexity is fully contained in the scraping layer. The scraping layer doesn't know anything about ma_live_* keys — that's the API layer's concern.

The split keeps each layer simple. The outward-facing auth is straightforward standard API security. The inward-facing auth is a collection of adapters for a messy world.

The API is free to try at myairports.online/developers. Free tier: 100 requests/day, no card required.

How Airport Websites Try to Stop You Scraping Them (And What I Did About It)

José Catalá — Mon, 13 Apr 2026 09:48:31 +0000

I scrape 200+ airport departure boards every few minutes. Here's every defence I've had to break through — and the ones that still slow me down.

Why Airport Websites Are a Scraper's Nightmare

Airport websites are an unusual target. They're not e-commerce sites trying to protect pricing data, and they're not news sites worried about plagiarism. Most of them just want to display departure boards to travellers. So you'd think they'd be easy to scrape.

They're not.

The problem is traffic patterns. An airport website gets hammered by bots — flight trackers, travel aggregators, airline apps, insurance tools, and journalists — all hitting the same handful of endpoints constantly. To survive, airports have layered in the same enterprise anti-bot stacks that protect banks and ticketing sites. They often don't even know exactly what's protecting them; it came bundled with their CDN contract.

Building MyAirports — a real-time flight data API covering 200+ airports — meant going through every layer. Here's what I found, roughly in order of how annoying each one is.

Layer 1: IP Rate Limiting

The most basic defence. Too many requests from one IP in a short window → 429 Too Many Requests or a silent block.

What it looks like: Your first few requests succeed. Then you start getting 429s, or worse, 200 responses with no flight data (the site is serving a decoy empty page rather than tipping you off with an error code).

What I did: Distribute requests across time. Each airport is on a polling interval calibrated to how often its data actually changes — a regional airport with 12 daily departures doesn't need to be hit every 30 seconds. Spreading the intervals out naturally reduces per-IP request rates.

For airports where even polite rates get blocked, I route through a residential proxy pool. It's not free, but for the ~15% of airports where it's necessary, there's no other option.

The nuance: Silent 200s with empty data are worse than 429s. I added a validation step that checks response plausibility — if a departure board suddenly returns zero flights for a major hub at 09:00 on a Tuesday, that's not real data. The scraper logs it as a suspected block rather than storing empty results.

Layer 2: User-Agent and Header Fingerprinting

Servers look at your request headers. A bare curl or Node fetch has obvious tells:

User-Agent: node-fetch/1.0
Accept: */*
Accept-Language: (missing)
Accept-Encoding: gzip, deflate

Real browsers send 15–20 headers in a specific order, with specific values that match the stated browser version.

What it looks like: Immediate 403, or redirect to a CAPTCHA page that your headless browser can't easily solve.

What I did: Use a real browser engine (Playwright with Chromium) rather than raw HTTP requests. This gives you the genuine browser fingerprint for free — correct header order, correct values, correct navigator properties, correct TLS handshake cipher suite order.

The cipher suite thing matters more than people realise. Node.js and browsers negotiate TLS with different cipher preference orders. Some DDoS protection vendors fingerprint the TLS handshake before the HTTP request even arrives. If your cipher order looks like OpenSSL rather than Chrome, you're flagged at the network layer before you've sent a single byte of HTTP.

// Using Playwright with a realistic browser context
const browser = await chromium.launch({
  headless: true,
  args: ['--no-sandbox', '--disable-blink-features=AutomationControlled']
});

const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...',
  locale: 'en-GB',
  timezoneId: 'Europe/London',
  viewport: { width: 1366, height: 768 }
});

The --disable-blink-features=AutomationControlled flag is important — without it, navigator.webdriver returns true in the page context, which is a trivial bot signal that JavaScript on the page can read.

Layer 3: Cloudflare Bot Management

This is where things get genuinely hard. Cloudflare's bot management suite runs a JavaScript challenge in the browser that collects dozens of signals:

Canvas fingerprint (GPU-specific rendering of a test image)
Audio context fingerprint
Font enumeration
WebGL renderer string
Screen dimensions vs. window dimensions
Mouse movement entropy (does the mouse move like a human?)
Timing of JavaScript execution
Whether window.chrome exists and has the right shape
Dozens more

The challenge runs silently in the background. Pass it and you get a cf_clearance cookie that's valid for a few hours. Fail and you get a 403 or an interactive CAPTCHA.

What it looks like: Page loads fine, but all API calls to the flight data endpoint return 403. Or: you get the page, the page loads some JS, and then the JS makes a secondary XHR that gets blocked.

What I did: Playwright with the playwright-extra stealth plugin, which patches the most common automation detection vectors:

import { chromium } from 'playwright-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';

chromium.use(StealthPlugin());

The stealth plugin patches things like:

navigator.webdriver → undefined
navigator.plugins → realistic plugin list
window.chrome → full Chrome object shape
Canvas fingerprint → randomised slightly per session
permissions.query for notifications → returns denied (real browsers do; headless ones used to throw)

This handles maybe 70% of Cloudflare deployments. The other 30% use more aggressive challenges that require the cf_clearance cookie to be renewed more frequently, or require solving an actual interactive challenge. For those, I use a CAPTCHA solving service — 2captcha or AntiCaptcha — which has human solvers available for the rare cases that get through the stealth layer.

Layer 4: Akamai Bot Manager

Akamai is different from Cloudflare. Instead of a one-time challenge, Akamai's bot manager injects a sensor script into every page that runs continuously and phones home with behavioural data. It's looking for things like:

Are mouse movements suspiciously perfect? (Bots often move in straight lines)
Are keystrokes happening at humanly possible intervals?
Is the page being scrolled at all?
Does the _abck cookie (Akamai's session cookie) contain a valid HMAC signature computed from real browser data?

The _abck cookie is the key. It's a JWT-like token containing sensor data, and it's validated server-side on each request. If the cookie is missing or invalid, API requests are rejected — even if you've got a valid session.

What it looks like: You can load the page fine. But when the page makes XHR requests to /api/flights or wherever the data lives, they come back 403 with no body. The failure is invisible to users (the JS handles the retry), but your scraper just gets nothing.

What I did: Two approaches depending on the airport.

For airports where the data is available in the initial HTML (the departure board is server-rendered), I just parse the HTML directly. Akamai only blocks the XHR calls; the first page load often goes through fine and contains all the data I need.

For airports where the data is loaded client-side via XHR, I use Playwright and let the full page JavaScript run. Playwright with stealth generates valid-enough sensor data that the _abck cookie gets accepted. I then extract the cookie and replay it on subsequent requests — though it expires and needs refreshing every 30–60 minutes.

// Wait for the XHR to complete rather than trying to replicate it
await page.waitForResponse(
  response => response.url().includes('/api/flights') && response.status() === 200
);
const data = await page.evaluate(() => window.__FLIGHT_DATA__);

Layer 5: Dynamic Endpoint Obfuscation

Some airport sites don't have a stable API endpoint. The URL for the flight data changes with each deployment — it might be hashed, versioned, or entirely randomised. Hardcoding /api/v1/flights breaks the moment they redeploy.

What it looks like: Your scraper works for weeks, then silently breaks when the airport updates their site. The URL 404s and you don't find out until someone notices the data hasn't updated.

What I did: Intercept network requests rather than targeting a specific URL. Playwright's page.on('response', ...) captures every response the page receives. I filter for responses that look like flight data — JSON with an array of objects containing recognisable fields like flight numbers, times, or status strings.

page.on('response', async (response) => {
  const url = response.url();
  if (!url.includes('flight') && !url.includes('departure') && !url.includes('arrival')) {
    return;
  }

  try {
    const body = await response.json();
    if (isFlightData(body)) {
      await processFlights(body);
    }
  } catch {
    // Not JSON, skip
  }
});

function isFlightData(data) {
  const arr = Array.isArray(data) ? data : data.flights || data.data || data.departures;
  if (!Array.isArray(arr) || arr.length === 0) return false;
  const sample = arr[0];
  return sample.flightNumber || sample.flight_no || sample.ident || sample.designator;
}

This is more resilient to site updates because I'm not coupled to a specific URL structure — I'm coupled to the shape of the data, which changes far less often.

Layer 6: Session Fixation and IP-Bound Cookies

This is the one that took me longest to diagnose. Some airports use session cookies that are bound to the IP address of the first request. The cookie is issued by the server and contains an HMAC of the client IP. If you grab the cookie from one IP (say, your local machine while debugging) and try to use it from another IP (your VPS), the server rejects it.

What it looks like: Everything works locally. On the VPS, you get 403s or empty data even though you're sending the right cookies. The difference is invisible in the request — you're sending a valid-looking cookie, it's just bound to the wrong IP.

What I did: I wrote about this in detail in a separate post, but the short version: I built SkipRestriction — a self-hosted Shadowsocks proxy running on the same VPS as the scraper. When I need to grab cookies for an airport with IP-bound sessions, I browse through the proxy so the cookie is issued to the VPS's IP. The scraper, also running on the VPS, then uses those cookies from the same IP.

Layer 7: JavaScript-Rendered Data with Anti-Automation Timing

Some sites render flight data using JavaScript that deliberately runs on a slight delay, or loads data in chunks, or only renders the full table after a scroll event. These are heuristics designed to detect headless browsers, which typically don't scroll and don't trigger layout-based events.

What it looks like: Playwright loads the page, you query the DOM for flight rows, and you get back 3 rows when there should be 45. The rest haven't rendered yet.

What I did: Inject realistic user behaviour before extracting data:

// Simulate a human reading the page before scraping
await page.waitForLoadState('networkidle');
await page.evaluate(() => window.scrollBy(0, 300));
await page.waitForTimeout(800 + Math.random() * 400);
await page.evaluate(() => window.scrollBy(0, 500));
await page.waitForTimeout(600 + Math.random() * 300);

// Now query for data
const flights = await page.$$eval('.flight-row', rows =>
  rows.map(row => ({
    flight: row.querySelector('.flight-number')?.textContent.trim(),
    status: row.querySelector('.status')?.textContent.trim(),
    time: row.querySelector('.scheduled-time')?.textContent.trim(),
  }))
);

The random delays matter — consistent 800ms waits are a bot signal. Human reading speed has natural variance.

The Layer I Haven't Beaten: Real CAPTCHA with Proof-of-Work

A handful of airports (3–4 in my coverage set) use a challenge mechanism that requires solving a proof-of-work puzzle — essentially, computing a hash with certain properties before the session is established. It's not a visual CAPTCHA; it's a compute challenge that's expensive for bots to complete at scale and trivially fast for a single human browser session.

I currently handle these with a hybrid approach: a real browser session is established manually once (or triggered via a webhook), the resulting cookies are stored, and the scraper uses those cookies until they expire. For airports with long cookie TTLs (24h+), this is workable. For airports that require fresh challenges every hour, it's not scalable and those airports currently have lower update frequency in MyAirports.

If you've cracked this one cleanly, I'd genuinely like to hear how.

What Works, What Doesn't, and What Changes Constantly

The honest summary: Playwright + stealth gets you through 80% of sites. IP management (residential proxies or VPS-matched cookies) handles another 10%. The remaining 10% requires per-site custom solutions, ongoing maintenance, or accepting reduced update frequency.

What changes most often: Cloudflare updates its challenge algorithms roughly every 6–8 weeks. When it does, stealth plugins need updating. I follow the puppeteer-extra-plugin-stealth repo closely and have a monitoring alert that fires when a Cloudflare-protected site's data goes stale — that's usually the signal that a new challenge version is live.

What's surprisingly stable: The actual data extraction logic. Airport websites update their visual design sometimes, but the underlying data structure in the HTML or JSON is remarkably consistent over time. The bot protection changes; the data format rarely does.

The Scraper Stack (For Reference)

Playwright (Chromium) for JS-heavy sites
playwright-extra + stealth plugin for Cloudflare
got (Node HTTP client) for simple HTML-only sites
Residential proxy pool for IP-rate-limited airports
Shadowsocks self-hosted proxy for IP-bound cookie issues
2captcha for the small number of visual CAPTCHAs
PostgreSQL for storage, with a scrape_log table tracking per-airport success/failure rates
Node-cron for scheduling, with dynamic intervals per airport based on data freshness requirements

The full architecture, including the self-learning caching layer, is covered in a previous post.

The flight data is live at myairports.online — if you want to use it rather than scrape airport sites yourself, the API handles all of this so you don't have to.

Have a layer I haven't covered? Drop it in the comments — I'm genuinely curious what else is out there.

Your Flight Was Cancelled. Here's How I Built a Tool That Finds Every EU261 Claim at Any Airport

José Catalá — Fri, 10 Apr 2026 07:59:21 +0000

Most passengers never claim the compensation they're legally owed. I built a feature that changes that — here's how it works under the hood.

The €250 Nobody Asks For

When Lufthansa cancelled flight LH954 from Frankfurt to Birmingham this morning, 25 flights at that airport alone became eligible for compensation under EU261 in the last 7 days.

Most of those passengers won't claim a cent.

Not because they're ineligible — under EU Regulation 261/2004, a cancelled flight on a route under 1,500 km entitles each passenger to €250 in cash compensation, regardless of whether the airline refunds the ticket. It's not a voucher. It's not store credit. It's money.

The problem isn't eligibility. It's awareness. Most travellers don't know the regulation exists, and those who do often assume claiming is too complicated to bother with.

I built MyAirports — a real-time airport data API — to solve the scraping and data normalisation problem. But once I had clean, reliable flight status data for hundreds of airports, a second problem became obvious: all this data about delays and cancellations was sitting there unused. I was already tracking which flights were cancelled, their distances, their routes. The EU261 calculation is basically a lookup table. So I built the Claims tab.

Here's how it works.

EU261 in Two Minutes

EU Regulation 261/2004 applies when:

The flight departs from an EU airport, or arrives at an EU airport on an EU/UK carrier
The flight is cancelled or delayed by more than 3 hours at arrival
The disruption is not caused by "extraordinary circumstances" (bad weather, air traffic control strikes, etc.)

Compensation tiers by route distance:

Distance	Compensation
Under 1,500 km	€250
1,500–3,500 km	€400
Over 3,500 km	€600

UK passengers still have equivalent rights post-Brexit under UK261.

That's it. The airline cannot refuse if the criteria are met. They can offer a voucher — but you're entitled to cash.

The Data Problem I Already Had Solved

MyAirports scrapes live flight data from airport websites — the actual departure boards, not third-party aggregators. Every flight gets:

IATA flight code
Origin and destination airport (with IATA codes)
Scheduled and actual departure/arrival times
Real-time status: on_time, delayed, cancelled, diverted, landed, etc.
Airline name

The status normalisation work was significant — airport websites display cancellations in seven different languages with wildly inconsistent terminology. I wrote about that in a previous post. By the time a flight hits the database, it has a clean, normalised status field.

For the Claims feature, cancelled is the key trigger. But not all cancellations qualify — the regulation only covers flights that departed from or arrived at an EU/UK airport. Since I'm already storing IATA codes, I can check country eligibility instantly against a static lookup of EU member state airport codes.

Calculating Eligibility Automatically

The core logic is straightforward once the data is clean:

function getEU261Compensation(flight) {
  // Must be cancelled or delayed 3+ hours
  if (flight.status !== 'cancelled' && flight.delayMinutes < 180) {
    return null;
  }

  // Route must touch EU/UK
  const fromEU = isEUAirport(flight.origin);
  const toEU = isEUAirport(flight.destination);
  const isEUCarrier = isEUAirline(flight.iataCode.slice(0, 2));

  if (!fromEU && !(toEU && isEUCarrier)) {
    return null;
  }

  // Distance-based compensation
  const distance = getGreatCircleDistance(flight.origin, flight.destination);

  if (distance < 1500) return 250;
  if (distance < 3500) return 400;
  return 600;
}

getGreatCircleDistance uses stored lat/lon for each IATA airport code — a ~7,000-entry lookup table I pre-compiled from public data. The great-circle calculation is just the haversine formula:

function haversine(lat1, lon1, lat2, lon2) {
  const R = 6371; // Earth radius in km
  const dLat = toRad(lat2 - lat1);
  const dLon = toRad(lon2 - lon1);
  const a =
    Math.sin(dLat / 2) ** 2 +
    Math.cos(toRad(lat1)) * Math.cos(toRad(lat2)) * Math.sin(dLon / 2) ** 2;
  return R * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
}

This runs at query time — no pre-computation needed. The distance between any two airports is deterministic, so it could be cached, but it's cheap enough that it doesn't matter.

What the Claims Tab Shows

When you open any airport page on myairports.online and click Claims, you see every flight in the last 7 days that passes the eligibility check — ranked by date, with:

Flight number and airline
Route (origin → destination)
Status (Cancelled / Delayed)
Distance (so you can verify the tier yourself)
Compensation amount (€250 / €400 / €600)
Regulation (EU261 or UK261)
Claim action — either a direct claim link to the airline's own portal, or a one-click claim via an integrated claims service

Birmingham Airport (BHX) had 25 flights eligible for compensation in the last 7 days when I last checked. That's not unusual. At a mid-sized European hub, 20–30 eligible flights per week is typical in normal operating conditions. During bad weather events or industrial action, that number can spike into the hundreds.

Two Claim Paths

Not every airline makes it easy to claim directly. Some (like British Airways and Lufthansa) have functional online claim forms — they comply, slowly. Others have forms that are buried, broken, or deliberately tedious.

The Claims tab surfaces both options:

Direct airline claim — for carriers with reliable self-service portals (EasyJet, Air France, Ryanair). We link directly to their claim page with the flight details pre-filled where possible.

Assisted claim for €2.99 — for cases where direct claiming is harder, or where passengers just don't want to deal with it. A claims management service handles the entire process: paperwork, follow-up, escalation to alternative dispute resolution if the airline refuses. They charge a flat €2.99 rather than the typical 25–35% cut that most "no win no fee" services take.

The decision of which path to offer is driven by a per-airline lookup table I maintain — essentially a reliability score for each carrier's own claims process. If the airline consistently pays out promptly on direct claims, we show the direct link. If they're known to stall, the assisted option is surfaced first.

The Unexpected Use Case

I built the Claims tab primarily as a user-facing feature. But it turns out developers using the API have found it useful too.

The /airports/{iata}/claims endpoint returns the same structured data:

{
  "airport": "BHX",
  "period_days": 7,
  "eligible_flights": [
    {
      "flight": "LH954",
      "airline": "Lufthansa",
      "date": "2026-04-10",
      "origin": "FRA",
      "destination": "BHX",
      "status": "cancelled",
      "distance_km": 766,
      "compensation_eur": 250,
      "regulation": "EU261",
      "claim_url": "https://www.lufthansa.com/gb/en/flight-irregularity-claim"
    }
  ],
  "total_eligible": 25
}

A few travel agencies and corporate travel tools have integrated this to automatically flag claims for their customers after disrupted trips. The value proposition is simple: most business travellers don't track their own delays. If a tool can surface "your employee on LH954 is owed €250" after the fact, that's real money recovered with zero effort.

What I Learned

Three things stood out building this:

Cancellation data is noisy. Airport websites sometimes mark a flight as cancelled and then re-list it under a new flight number (a "rescue flight"). If you don't de-duplicate carefully, you'll show a claim for a flight that technically departed. I added a reconciliation step that checks if the same route, same date, and similar departure time appears with a landed status anywhere in the dataset.

Extraordinary circumstances are unverifiable at query time. The regulation exempts cancellations caused by things outside the airline's control — storms, ATC strikes, bird strikes. I don't attempt to classify this automatically. The Claims tab shows all cancellations that pass the route/distance check and notes that eligibility is subject to circumstances. The claims process itself resolves extraordinary-circumstance disputes.

Distance matters more than people think. A Dublin→Birmingham cancellation (322 km) yields €250. A Paris CDG→Birmingham cancellation (488 km) also yields €250. But a longer intra-EU route like Lisbon→Helsinki (2,800 km) would yield €400. Passengers often assume cancelled = €250 flat, then feel cheated when their long-haul claim is assessed differently. Making the distance explicit in the UI has reduced those support queries significantly.

Try It

Every airport on myairports.online has a Claims tab. If you flew into or out of a European airport in the last 7 days and your flight was cancelled, check the Claims tab for your airport — there's a reasonable chance your flight is listed.

If you're a developer, the /airports/{iata}/claims endpoint is part of the MyAirports API. The free tier gives you 100 requests/day; the paid tier removes the cap and adds historical claims data going back 90 days.

EU261 was designed to make airlines accountable. The data to enforce it has always existed — it just needed surfacing.

MyAirports is a real-time airport data API covering 200+ airports across Europe, the Americas, and Asia-Pacific. The Claims feature is live at myairports.online.

How I Solved the IP-Bound Cookie Problem in My Web Scraper With a Self-Hosted Proxy

José Catalá — Tue, 07 Apr 2026 11:22:19 +0000

There's a category of airport website that breaks my scraper in a specific, maddening way: they issue session cookies that are pinned to the IP address of the request that created them.

The symptom looks like this: you manually grab a working cookie from your browser, paste it into your server-side scraper, and it immediately returns a 401 or redirects to a login page. The cookie is valid — you just created it. But it was created from your laptop's IP, and your VPS has a different one. The airport's backend checks both the cookie and the origin IP, and when they don't match, it rejects the session.

For MyAirports — a real-time flight data API covering 1,000+ airports — this is a real operational problem. Here's how I solved it.

Why IP-Bound Cookies Exist

Session fixation attacks are the reason. A server that binds cookies to IPs prevents an attacker who steals your session cookie from replaying it from a different location. It's a legitimate security measure, and it's common in systems that handle sensitive operational data — which airport backend systems often do.

For a scraper, it creates a workflow problem. You can:

Grab a fresh cookie from a real browser session
Replay that cookie from your server

But only if both steps happen from the same IP.

The Manual Cookie Workflow

For airports that require authenticated sessions (particularly those protected by HCaptcha or Incapsula), my architecture uses a grab-cookies.js tool. It launches a real browser, navigates to the airport site, solves challenges if needed, and exports the resulting session cookies.

Those cookies then get passed to the scraper as headers:

// Simplified example from the scraper pipeline
const cookies = await loadCookies('MAD'); // Madrid Barajas
const response = await fetch(endpoint, {
  headers: {
    'Cookie': cookies.join('; '),
    'Referer': 'https://www.aena.es'
  }
});

When grab-cookies.js runs on the VPS itself, everything works — the cookie was issued to the VPS IP, and the scraper makes requests from the same IP. But during development and debugging, running a headless browser on the server just to grab a cookie is slow and inconvenient. You want to do it from your local machine.

That's where the IP mismatch breaks things.

The Solution: Route Your Browser Through the VPS

The fix is simple in principle: make your local browser appear to come from the VPS IP when visiting airport websites. Any traffic that comes from your browser through the proxy will receive cookies bound to the VPS IP — the same IP your production scraper uses.

I used SkipRestriction.uk to set up a self-hosted Shadowsocks proxy on the same VPS where MyAirports runs. The setup is a single Docker command:

docker run -d \
  --name shadowsocks-server \
  --restart always \
  -p 8388:8388/tcp \
  -p 8388:8388/udp \
  shadowsocks/shadowsocks-libev:latest \
  ss-server -s 0.0.0.0 -p 8388 -k YOUR_PASSWORD -m aes-256-gcm

Point your local Shadowsocks client at YOUR_VPS_IP:8388, configure your browser to use it as a SOCKS5 proxy, and now your browser traffic exits from the VPS. Airport websites see your VPS IP. Cookies issued to that session are tied to the VPS IP. When the production scraper replays those cookies from the same VPS, the IP check passes.

Why Shadowsocks and Not a VPN

A traditional VPN would work too, but Shadowsocks has a few practical advantages for this use case:

It's lightweight. Running WireGuard or OpenVPN on the same VPS as a production scraper means managing certificates, key exchange, and routing tables. The Shadowsocks Docker container adds almost nothing to the host.

Traffic looks like HTTPS. Shadowsocks was designed to be indistinguishable from normal encrypted web traffic. Some airport networks filter VPN protocols at the packet level. Shadowsocks is harder to detect and block.

Per-application routing. With a VPN, all traffic from your machine goes through the VPS — including your Slack messages and YouTube tabs. With a SOCKS5 proxy, you configure specific browsers or applications to route through it. You can keep two browser profiles: one going through the VPS for cookie grabbing, one going directly for everything else.

No kernel modules. The Docker container runs entirely in user space. No system-level network reconfiguration needed.

The Full Cookie Workflow

With the proxy in place, the workflow for IP-bound cookies looks like this:

Local browser (via Shadowsocks proxy)
  → VPS IP
  → Airport website
  → Cookie issued to VPS IP

grab-cookies.js export
  → cookies stored server-side

Production scraper (on same VPS)
  → Same VPS IP
  → Airport website
  → Cookie accepted ✓

The roundtrip adds maybe 80-120ms of latency compared to a direct connection, which is irrelevant for a one-time cookie grab operation.

Setting It Up

The SkipRestriction.uk guides cover the full setup for both VPS (cloud server) and Raspberry Pi (local self-hosting). The site also walks through client configuration for every major platform — macOS, Windows, iOS, Android, and Linux.

If you're running a scraper on a VPS and hitting IP-bound session issues, the proxy approach is the cleanest fix. The alternative — running your entire debugging workflow on the server via SSH — is technically viable but slow and error-prone.

Two commands on the server side, one config file on the client side, and the IP mismatch problem disappears.

The proxy runs on port 8388. The MyAirports API runs on port 443. They coexist on the same $5/month VPS without any conflict.

MyAirports is a free real-time flight data API covering 1,000+ airports. Try it at myairports.online/developers.

The Shadowsocks setup guide used for this project is at skiprestriction.uk — open source, self-hosted, no accounts required.

628,000 Flights Later: Building a Flight Analytics Engine on Accumulated Scrape Data

José Catalá — Mon, 06 Apr 2026 16:40:24 +0000

Every time MyAirports scrapes a flight board, the flights are stored in PostgreSQL. What started as a caching mechanism turned into something more interesting: a dataset of real flight behaviour that can answer questions no static API can answer.

After accumulating 628,000+ flight records across 1,000+ airports, the data supports four categories of analytics:

Airline reliability rankings per airport (on-time rate, average delay, cancellation rate)
Delay pattern breakdowns by airline, destination, day of week, and time of day
Airport busyness heatmaps (7×24 grid: average flight volume by hour and day)
Route reliability stats with per-airline breakdown (LHR→JFK: which airline is most reliable?)

Here's how the analytics layer works.

The raw data

Each Flight record in PostgreSQL has:

flightNumber, airline (IATA), type (arrival/departure)
airport (IATA), origin, destination
scheduled, estimated, actual (timestamps, local timezone)
status (scheduled/boarding/departed/arrived/delayed/cancelled/unknown)
confidence, strategy, scrapedAt

The actual timestamp and status field are what make analytics possible. When a flight's status is arrived and actual is set, we know when it actually landed. The delay is simply actual - scheduled in minutes.

Two compute strategies: incremental and nightly

Running full analytics queries over 628,000 rows on every API request would be too slow. The system uses a two-track approach.

Incremental counters — updated after each scrape:

When a new batch of flights arrives, the system immediately updates lightweight counters: per-airline flight counts at that airport, per-terminal/gate flight counts. These are cheap to maintain and power fast lookups.

Nightly full recompute — runs at 2 AM UTC:

Once per day, the system runs a full pass over all historical data to recompute the richer analytics tables. This is where the delay stats, airline scorecards, and busyness grids are built. It's more expensive, but the data doesn't need to be real-time — yesterday's delay patterns are as useful as today's.

Airline reliability rankings

The query is straightforward in concept, but requires minimum thresholds to be useful:

SELECT
  airline_iata,
  COUNT(*) as total_flights,
  COUNT(*) FILTER (WHERE delay_minutes <= 15) * 100.0 / COUNT(*) as on_time_pct,
  AVG(delay_minutes) FILTER (WHERE delay_minutes > 0) as avg_delay_minutes,
  COUNT(*) FILTER (WHERE status = 'cancelled') * 100.0 / COUNT(*) as cancelled_pct
FROM flights
WHERE airport_iata = 'LHR'
  AND actual IS NOT NULL
  AND status IN ('arrived', 'departed')
GROUP BY airline_iata
HAVING COUNT(*) >= 10
ORDER BY on_time_pct DESC;

The HAVING COUNT(*) >= 10 is the important part. Without a minimum flight threshold, airlines that had one flight and it happened to be on time appear at the top with 100% on-time performance. Ten flights is a low bar, but it filters out pure noise.

The result for LHR looks like this:

{
  "rank": 1,
  "airlineIata": "LX",
  "airlineName": "Swiss International Air Lines",
  "onTimePct": 88.2,
  "avgDelayMinutes": 7.4,
  "cancelledPct": 0.3,
  "totalFlights": 2340
}

Delay patterns: finding when and why delays happen

The delay breakdown goes a level deeper than overall stats. It slices delay data by:

Day of week — Monday departures are often worse than Wednesday ones
Time bucket — morning (06-12), afternoon (12-18), evening (18-24), night (00-06)
Destination — some routes are systematically late
Airline — some carriers at a specific airport are consistently worse than their global average

The time bucket breakdown is particularly useful. At large hub airports, delays compound through the day: a slot disruption in the morning propagates through afternoon flights. The heatmap data makes this visible.

Airport busyness heatmap

The busyness grid answers "when is this airport busy?" — useful for scheduling, understanding delay risk by time, and just general situational awareness:

SELECT
  EXTRACT(DOW FROM scheduled) as day_of_week,  -- 0=Sun, 6=Sat
  EXTRACT(HOUR FROM scheduled) as hour,
  AVG(flight_count) as avg_flights
FROM (
  SELECT
    DATE_TRUNC('hour', scheduled) as bucket,
    EXTRACT(DOW FROM scheduled) as day_of_week,
    EXTRACT(HOUR FROM scheduled) as hour,
    COUNT(*) as flight_count
  FROM flights
  WHERE airport_iata = $1
  GROUP BY 1, 2, 3
) hourly
GROUP BY day_of_week, hour
ORDER BY day_of_week, hour;

The result is a 7×24 matrix of average hourly flight volumes, suitable for heatmap rendering in any charting library.

Gate and terminal predictions

The most interesting use of accumulated data is predicting which gate or terminal a flight will use before the airport announces it. Airlines tend to use the same gates consistently at their hub airports.

After each scrape, the system updates per-airline terminal and gate counters:

// After normalizing a batch of flights
for (const flight of flights) {
  if (flight.terminal) {
    incrementCounter(`${airport}:${airline}:terminal:${flight.terminal}`);
  }
  if (flight.gate) {
    incrementCounter(`${airport}:${airline}:gate:${flight.gate}`);
  }
}

When serving a flight that doesn't have a gate assigned yet, the API enriches it with the most common gate for that airline at that airport:

if (!flight.gate && insights?.terminal?.predicted) {
  flight.insights = {
    terminal: { predicted: insights.terminal.value, confidence: insights.terminal.confidence }
  };
}

The confidence value is the fraction of historical flights that used the predicted terminal. For BA at LHR, it's nearly always Terminal 5 — confidence approaches 1.0. For a code-share flight operated by a third party, confidence might be 0.3.

The route reliability endpoint

One API endpoint that required no special query design: route-level stats. Given an origin and destination, return reliability data for each airline that operates the route:

GET /api/routes/LHR/JFK/insights

{
  "origin": "LHR",
  "dest": "JFK",
  "route": {
    "totalFlights": 1240,
    "onTimePct": 78.3,
    "avgDelayMinutes": 16.5,
    "cancelledPct": 1.8,
    "topAirline": "BA",
    "topAirlineOnTimePct": 84.2
  },
  "airlines": [
    { "airlineIata": "BA", "onTimePct": 84.2, "totalFlights": 890 },
    { "airlineIata": "VS", "onTimePct": 71.4, "totalFlights": 350 }
  ]
}

This is data that's genuinely hard to find anywhere else. The big flight data providers have it, but it's locked behind enterprise pricing. The MyAirports version is built from first-party airport data, accumulated one scrape at a time.

What the data still can't do

A few honest caveats about 628,000 flights:

It's not uniform. Airports with many active users have more scrape history than airports discovered recently. LHR has thousands of records; a newly added regional airport might have dozens.

It's biased toward requested airports. The system scrapes when someone requests data. Popular airports are over-represented relative to their actual traffic volumes.

Delay calculations require actual timestamps. Many airport APIs only return scheduled times, not actual arrival/departure times. Those flights contribute to volume counts but not delay calculations.

The data is directionally useful — the patterns it reveals are real — but it's not a controlled dataset. It's accumulated opportunistically.

All three endpoints are available in the MyAirports API:

GET /api/airports/:iata/insights/airlines
GET /api/airports/:iata/insights/delays
GET /api/airports/:iata/insights/busyness
GET /api/routes/:origin/:dest/insights

Free tier at myairports.online/developers. No credit card required.

Normalizing Flight Statuses Across 7 Languages: What I Learned Building a Global Airport API

José Catalá — Mon, 06 Apr 2026 16:39:30 +0000

When you scrape flight data from airport websites across 85+ countries, you quickly discover that nobody agrees on anything. Not the data format, not the field names, not the status strings.

"Departed" is SAL, FLY, CER, AIR, DEP, DEPARTED, Departed, Отправлен, Väljunud, DESPEGADO, gestartet, or partito, depending on which airport website you're looking at.

Building MyAirports meant writing a status normalizer that handles all of these — and maps them to a single, predictable enum. Here's how it works and what surprised me along the way.

The target schema

Every flight the API returns has a status field from this set:

scheduled | boarding | departed | arrived | delayed | cancelled | unknown

Seven values. Simple, predictable, usable in any frontend without additional mapping. The hard part is getting there from the real world.

Layer 1: Exact string matches

The first layer is the most straightforward — a lookup table of known status strings to their canonical values:

const EXACT = {
  // English
  'scheduled': 'scheduled',
  'on time': 'scheduled',
  'departed': 'departed',
  'in flight': 'departed',
  'arrived': 'arrived',
  'landed': 'arrived',
  'delayed': 'delayed',
  'cancelled': 'cancelled',
  'boarding': 'boarding',
  'gate open': 'boarding',

  // Spanish (AENA codes)
  'SAL': 'departed',  // salida
  'AIR': 'departed',  // en aire
  'FLY': 'departed',
  'CER': 'departed',  // cerrado
  'LND': 'arrived',   // aterrizó
  'FNL': 'arrived',   // final
  'IBK': 'arrived',   // in-block
  'BOR': 'boarding',  // embarcando
  'EMB': 'boarding',
  'ULL': 'boarding',
  'OPE': 'boarding',
  'EST': 'scheduled', // estimado
  'PRG': 'scheduled', // programado
  'RET': 'delayed',   // retraso
  'CNL': 'cancelled',

  // Russian
  'Отправлен': 'departed',
  'Прилетел': 'arrived',
  'Посадка': 'boarding',
  'Задержан': 'delayed',
  'Отменён': 'cancelled',

  // Estonian
  'Väljunud': 'departed',
  'Saabunud': 'arrived',
  'Pardal': 'boarding',

  // Finnish (Finavia)
  'Lähtenyt': 'departed',
  'Saapunut': 'arrived',
  'Nousussa': 'boarding',
};

This handles probably 60-70% of real-world cases. The remainder need more work.

Layer 2: Substring patterns

Some status strings embed dynamic information — actual times, gate numbers, gate closures. You can't match them exactly, but you can match them by pattern:

const SUBSTRING_PATTERNS = [
  // "Delayed 00:45"
  [/delay/i, 'delayed'],
  // "Cancel" in any language starting with 'cancel'
  [/cancel/i, 'cancelled'],
  // AENA-style: "CERRADO" (closed), "PROGRAMADO" (scheduled)
  [/^cerr/i, 'departed'],
  [/^progr/i, 'scheduled'],
  // "Check-in open" → scheduled (not boarding yet)
  [/check.?in/i, 'scheduled'],
];

Layer 3: Regex patterns for dynamic content

A handful of airports embed actual timestamps or qualifiers directly in the status string. Phoenix Sky Harbor is a good example — their status field for delayed departures looks like "Now 14:22" (the actual departure time). That's not a status, that's a timestamp, and it means the flight has departed.

const REGEX_PATTERNS = [
  // Phoenix: "Now 12:21 PM" → departed
  [/^now\s+\d/i, 'departed'],
  // Finavia: "Expected 14:35" → still scheduled
  [/^expected\s+\d/i, 'scheduled'],
  // Generic: "14:35" alone means actual time → departed/arrived
  [/^\d{1,2}:\d{2}$/, 'arrived'],
];

Layer 4: Russian stem matching

Russian is morphologically rich — words inflect heavily based on grammatical case, number, and gender. "Посадка" (boarding) appears as "посадке", "посадкой", "посадку" in different sentence positions. Exact matching misses these.

The solution isn't a full NLP library (overkill for six words) — it's stem prefixes. The first four or five characters of a word are usually enough to identify it:

const RUSSIAN_STEMS = {
  'посад': 'boarding',   // posad- (посадка, посадке...)
  'отпра': 'departed',   // otpra- (отправлен, отправился...)
  'прил':  'arrived',    // pril- (прилетел, прилетела...)
  'задер': 'delayed',    // zader- (задержан, задержка...)
  'отмен': 'cancelled',  // otmen- (отменён, отменённый...)
};

This catches all the common inflected forms without false positives.

The object guard

One thing I didn't expect: some airport APIs return status as an object rather than a string:

{ "status": { "code": "DEP", "description": "Departed", "color": "#ff0000" } }

The normalizer checks for this and extracts code or description before attempting any matching. Without this guard, typeof status === 'object' crashes the string matching silently.

AENA's 30+ status codes

AENA (the Spanish airport operator, covering all 22 Spanish airports) has the most elaborate status system I encountered. They use two to three character codes from their SIGER ground handling system:

FNL, LND, IBK → arrived
BOR, EMB, ULL, OPE, APE → boarding
SAL, AIR, FLY, CER, APA → departed
EST, PRG, INI, SCH, HOR → scheduled
RET, DEM, REM → delayed
CNL, DES, CAN → cancelled

Each code has a specific operational meaning (IBK = in-block, ULL = final boarding call, DEM = ground delay) that the normalizer collapses to the six canonical values. The operational detail is lost, but that's the right tradeoff for a public API.

What "unknown" means in practice

The unknown bucket catches everything the normalizer can't confidently classify. In production, it appears for:

Empty strings or null values (some airports just don't return status for on-time flights)
Airport-specific codes that haven't been mapped yet
Status strings in languages I haven't configured (Arabic, Japanese, Chinese)

Returning unknown is far better than returning a wrong value. Frontends can show "—" for unknown statuses and users understand it means data isn't available.

Running the normalizer

The normalizer runs at the end of every scrape, after the field mapper has identified which JSON field contains the status. It's a pure function:

normalizeStatus(raw) // → 'scheduled' | 'boarding' | 'departed' | 'arrived' | 'delayed' | 'cancelled' | 'unknown'

Input is always a string (after the object guard). Output is always one of the seven values. No exceptions are thrown — unrecognized inputs return 'unknown'.

The broader lesson

I went into this project expecting the hard part to be scraping. The really hard part turned out to be normalization — taking data from wildly inconsistent sources and making it uniformly useful.

Status normalization is one example. The same problem appears with timestamps (airports use their local timezone, not UTC, and the formats vary wildly), location fields (some put "MILAN, MALPENSA" in the IATA field), and airline codes (some use IATA codes, some use ICAO, some use full names).

The principle that worked: design a strict output schema first, then build the normalization layer to enforce it, and treat every edge case as an addition to the normalizer rather than a special case in the consumer.

The API is live and free to try at myairports.online/developers. Arrivals and departures for 1,000+ airports, statuses normalized to the seven values above.

Next in this series: how 628,000 accumulated flight records are turned into airline delay rankings, gate predictions, and airport busyness heatmaps.

How I Built a Web Scraper That Gets Faster and Cheaper With Every Request

José Catalá — Mon, 06 Apr 2026 16:38:33 +0000

Most web scrapers are dumb: they do the same work every time. Open a browser, load a page, parse the DOM, extract data, close the browser. Every single request.

Building MyAirports — a real-time flight data API covering 1,000+ airports — forced me to think about scraping differently. At scale, running a headless browser for every request is expensive and slow. The system I ended up building instead gets faster and cheaper the more it runs, because it learns.

Here's how it works.

The core insight: most airport SPAs load data from internal JSON APIs

Go to any modern airport website. Open DevTools. Click on the Network tab. Filter for XHR/Fetch requests. What you'll see almost immediately is a call to something like:

GET https://api.heathrow.com/pihub/flights/arrivals
GET https://aena.es/sites/Satellite?pagename=AENA_ConsultarVuelos&airport=MAD
GET https://cph.dk/api/FlightInformation/GetFlightInfoTable?direction=A

The webpage is a React or Angular SPA, and the actual flight data comes from an internal API. That API is often unauthenticated, or authenticated only by session cookies the browser already has. If you can capture that API call, you can make it directly — no browser needed for subsequent requests.

This is the foundation of the whole system.

The 5-layer pipeline

Every flight data request flows through five layers:

Layer 1: Operator registry
For ~500 airports I've already reverse-engineered the API, the system goes directly there. No discovery needed. A curated registry maps IATA codes to endpoints, required headers, and a transform function for normalizing the response. This handles roughly half of all requests, takes 1-2 seconds, and costs nothing beyond the API call.

Layer 2: API cache (discovered endpoints)
The system maintains a database of APIs it has discovered during previous browser sessions. Before firing up a browser, it checks: have we already seen a good API for this airport? If yes, call it directly. This is the "memory" layer — it's how the system gets cheaper over time.

Layer 3: Browser discovery
Only if both previous layers miss does the system launch a real browser (Playwright, stealth mode). But even here it's not scraping the DOM — it's intercepting network traffic. The browser loads the airport's flight page, and the interceptor scores every JSON response it sees:

function scoreResponse(url, body) {
  let score = 0;
  if (url.includes('flight') || url.includes('arrival') || url.includes('departure')) score += 3;
  if (Array.isArray(body) && body.length > 5) score += 2;
  if (body[0]?.flightNumber || body[0]?.flight_no) score += 3;
  if (body[0]?.scheduled || body[0]?.scheduledTime) score += 2;
  return score;
}

Any response with a score above 5 is a candidate. The best candidate is saved to the ApiCache table — the discovered endpoint, the required headers, the response shape. Future requests for that airport hit Layer 2 instead.

Layer 4: Normalizer
Regardless of which layer found the data, every response goes through a normalizer that converts it to a unified flight schema. Airport APIs have wildly different shapes — nested objects, flat arrays, GraphQL responses, tables with inconsistent column names. The normalizer handles all of it.

Layer 5: Persistence and stale-while-revalidate cache
Results are stored in PostgreSQL and kept in an in-memory cache with a stale-while-revalidate strategy. A stale response is returned immediately while a fresh scrape runs in the background.

The result: the system pays for its own speed

When the system first encounters a new airport, it's slow: a full browser session takes 10-15 seconds. But the next time that airport is requested, the discovered API is available in the cache — and the call takes 1-2 seconds.

Over time, the API cache fills up. More and more airports become fast. The "discovery" cost is paid exactly once per airport. After that, it's a cheap HTTP call.

There's also an eviction mechanism: if a cached API endpoint starts returning 0 flights three times in a row, it gets flagged as stale and the system re-discovers it. Airport websites do change their APIs occasionally.

The auto-discovery tool

To seed the API cache without waiting for organic traffic, I built a batch auto-discovery CLI that processes airports from the OpenFlights CSV database:

npm run auto-discover -- --country IT
npm run auto-discover -- --min-size large

It runs the full browser discovery pipeline for each airport, then writes the discovered endpoints to the config files automatically. Processing 1,300+ airports this way found 276 new ones — airports whose APIs I hadn't previously known existed.

AI pattern learning as a last resort

For airports where network interception fails (because the page loads data from a server-side render rather than a client-side API call), there's a DeepSeek-powered fallback: send the page's HTML to the AI and ask it to identify CSS selectors for the flight table.

GYD (Baku) — a.flt_row card layout with .fl_col.* columns
KTW (Katowice) — div.timetable__row.flight-board__row with .flight-board__col--*
TLL (Tallinn) — li.flights-list__item card layout

The AI doesn't always get it right on the first try, but for airports with non-standard layouts it's significantly faster than manually inspecting the DOM.

What I learned

The most important architectural decision was treating discovery as a one-time investment rather than an ongoing cost. Every browser session is expensive — it's a browser, running JavaScript, waiting for a page to load. If you pay that cost once and cache the result, you amortize it across hundreds of future requests.

The second insight was scoring rather than parsing: instead of trying to understand an airport's specific API format upfront, intercept everything and look for responses that smell like flight data. The scoring heuristic is simple but effective, and it works across wildly different API shapes.

The third was building eviction in from the start. APIs change. Caching without a sensible eviction strategy means you end up with a database full of dead endpoints.

The result is a system that today handles most requests in 1-2 seconds, covers 1,000+ airports, and gets incrementally better with every new airport it encounters.

The API is free to try at myairports.online/developers.

If you found this interesting, I write about building MyAirports — a real-time flight data API. The next post covers how the system normalizes flight status strings from 7 different languages.

MyAirports: Real-Time Flight Data for 1,000+ Airports — Now with a Free API

José Catalá — Sun, 05 Apr 2026 10:06:42 +0000

Whether you're building a travel app, a flight notification bot, or just want to check if your flight is on time — finding reliable, real-time flight data has always been painful. Most data providers charge hundreds of dollars per month, require lengthy contracts, or give you stale data wrapped in clunky XML.

MyAirports.online is changing that.

What Is MyAirports?

MyAirports (https://myairports.online) is a live flight tracking platform covering 1,000+ airports across 85+ countries. It delivers up-to-the-minute arrivals and departures for airports from London Heathrow to Singapore Changi, Madrid Barajas to New York JFK — all through a simple REST API or a clean web interface.

The API — Simple, Fast, and Free to Start

GET https://myairports.online/api/airports/LHR/arrivals

You get clean JSON back — flight number, airline, origin/destination, times, gate, terminal, status, and code-share numbers.

Free accounts: 100 requests/day. Pro plan ($29/month): 1,000 requests/day.

Beyond Flights: Ground Transport, Parking & Car Rental

Ground connections — metro, buses, trains, shuttles, taxis with pricing and times.
Parking options — short-stay, long-stay, valet, economy, premium with current rates.
Car rental — major brands and local companies at the airport.

Endpoints:

https://myairports.online/connections/MAD — Madrid ground transport
https://myairports.online/parking/MAD — Madrid parking
https://myairports.online/car-rental/MAD — Madrid car rental

Global Coverage: 85+ Countries

Europe: LHR, MAD, BCN, AMS, VIE, CPH, DUB, FCO, MXP, ZRH, PRG, LIS, and more.
Americas: JFK, EWR, LGA, DFW, DEN, MCO, PHX, LAS.
Middle East & Asia: DXB, SIN.
Oceania & Africa: MEL, BNE, PER, NBO, CAI.

Developer Dashboard

API Playground — test endpoints with auto-generated curl/JS/Python snippets.
Usage Monitoring — 14-day chart of API calls and remaining quota.
API Key Management — generate/rotate ma_live_* keys with one click.
Request Logs — filterable log with timestamp, endpoint, and status.

Who Is This For?

Developers building travel apps or flight bots — clean JSON, no enterprise pricing.
Travelers who want a fast, ad-free flight board.
Businesses that need reliable airport data at reasonable cost.

Try It Now

Visit myairports.online — no auth needed for public flight boards.

For API access: myairports.online/signup

Real-time flight data. Clean JSON. No contracts. No headaches.

Originally published on Medium