DEV Community: Pallab Mondal

I Spent 3 Weeks Debugging Rate Limits Before I Realized the Problem Wasn't My Code

Pallab Mondal — Thu, 23 Jul 2026 21:22:36 +0000

Ever chased a bug for days, only to discover the "bug" was actually the platform working exactly as designed? That happened to me building a client reporting pipeline. The lesson stuck.

Here's what nobody tells you about pulling marketing data from multiple ad platforms: the hard part was never the dashboard. It was everything underneath it.

The Setup That Looked Simple on Paper

The brief sounded easy. Pull spend, clicks, and conversions from Google Ads and Meta. Store it. Display it in a chart. A junior dev could knock this out in a sprint, I figured.

Reality disagreed. Google Ads API enforces operation quotas per developer token, and those quotas scale differently depending on account tier. Meanwhile, Meta's Marketing API throttles based on a rolling usage score tied to the ad account itself, not your app.

Two platforms. Two completely different throttling philosophies. Neither documented in a way that made the actual limits obvious until you hit them in production.

Where Things Actually Broke

My first version polled every client account every hour. Fine for three clients. Then we onboarded client number twelve, and Meta started returning 429s intermittently. Not consistently — intermittently. That's the worst kind of bug.

I initially assumed it was a code issue. Retry logic, maybe a race condition in my job scheduler. I spent three weeks going down that path. Eventually, I found the real cause: cumulative API call volume across all client accounts was tripping Meta's app-level rate limit, not the individual account limit.

The fix wasn't more retries. It was a request queue with exponential backoff, plus a priority system so active dashboards refreshed before idle ones. Simple in hindsight. Expensive in dev hours.

The Real Architecture Behind Multi-Platform Reporting

If you're building this yourself, here's what a production-grade pipeline actually needs, based on what broke for me.

A Queue, Not a Cron Job

Don't just fire off API calls on a schedule and hope for the best. Use a proper job queue — something like BullMQ or Sidekiq — with retry policies and backoff built in from day one. This alone would've saved me those three weeks.

Token Refresh as a First-Class Concern

OAuth tokens expire silently. Build monitoring specifically for auth failures, separate from your general error logging. Otherwise, you'll find out a client's Google Ads token expired only when they ask why their dashboard looks empty.

A Normalization Layer Between Raw Data and Storage

Google Ads calls it "cost." Meta calls it "spend." GA4 buries similar metrics under different dimension names entirely. Build one internal schema and map every platform into it, rather than letting frontend code handle platform-specific field names.

javascript // Simplified normalization example function normalizeMetric(platform, rawData) { const mapping = { google_ads: { cost: rawData.cost_micros / 1e6, clicks: rawData.clicks }, meta: { cost: rawData.spend, clicks: rawData.clicks }, }; return mapping[platform] || rawData; }

This function looks trivial. It isn't, once you're handling currency conversions, timezone mismatches, and attribution window differences across five platforms simultaneously.

Should You Actually Build This?

Here's the honest answer, from someone who's now maintained this kind of system for two years. If reporting infrastructure isn't your product's core differentiator, building it yourself is usually the wrong call.

I don't say that lightly. I like building things. However, every hour spent debugging Meta's rate limit quirks was an hour not spent on features that actually made our product better for users.

What Changed My Mind

Eventually, we moved most of our client reporting workflow onto a dedicated platform, RaiseReturn, instead of continuing to maintain our in-house pipeline. It already handled the multi-platform normalization, the token refresh monitoring, and the rate-limit-aware queuing I'd built by hand.

The API integrations existed already, tested against the platforms' quirks by a team that deals with exactly this problem full-time. That's not something you replicate cheaply in a side project or even a dedicated sprint.

Where Custom Code Still Makes Sense

To be clear, this isn't an argument against ever building integrations. If your reporting needs pull from truly proprietary internal systems — a custom CRM, an internal data warehouse — no off-the-shelf tool covers that. You'll build custom connectors regardless.

For standard ad platforms though, the problem is already solved. Well-tested, production-hardened, and maintained by people who get paged when Meta ships a breaking API change at 2 AM. You probably don't want to be that person.

Practical Takeaways for Your Own Stack

If you're currently building or maintaining a custom reporting pipeline, a few things worth checking right now.

First, verify your retry logic actually implements exponential backoff, not fixed-interval retries. Fixed intervals make rate limiting worse, not better, under sustained load.

Second, add dedicated alerting for OAuth token failures. Silent auth failures are the most common cause of "why is this client's data missing" tickets.

Third, honestly evaluate the maintenance cost against a dedicated automated reporting tool. Sometimes the math favors buying. Sometimes it doesn't. Either way, run the numbers before defaulting to "we'll just build it."

Final Thought

Client reporting infrastructure is a genuinely interesting engineering problem. It's also, for most teams, not the problem worth solving from scratch. Know the difference, and your future 2 AM self will thank you.

How We Built an Asynchronous Multi-API Pipeline for Marketing Automation

Pallab Mondal — Wed, 27 May 2026 20:33:23 +0000

When you are building a platform designed to consolidate data from Google Ads, GA4, Meta Ads, and Google Search Console, you quickly run into a massive bottleneck: API latency and rate limiting.

If an agency has 50 clients, and each client needs data pulled from 4 different ad networks, a chronological extraction loop will completely stall your application. Your server will time out, your database connection pools will exhaust, and your users will look at a spinning loading wheel forever.

While the core engine of our platform—RaiseReturn—remains a proprietary, private codebase, I wanted to share the exact architectural logic we used to solve the asynchronous fetching problem in Node.js.

The Bottleneck: Chronological vs. Parallel Processing

Most standard API wrappers encourage you to fetch data chronologically:

Wait for Google Ads to respond.
Take that data, then wait for Meta Ads to respond.
Wait for GA4 to respond.

This is a terrible approach for multi-tenant scalability. If one API experiences a 3-second network delay, your entire pipeline backs up. Instead, you need to execute these network requests concurrently, handle individual failures gracefully without crashing the whole sequence, and normalize the disparate payloads into a unified schema before writing to your database.

A Sanitized Example: The Asynchronous Batch Worker

Below is a lightweight, clean abstraction of how you can structure a data worker to fetch metrics from entirely different API endpoints concurrently using Promise.allSettled.

Using allSettled ensures that if the Meta Ads API randomly throws an authentication error or a 500 timeout, your script still successfully captures and processes the Google Ads and GA4 data for that client.

// A lightweight abstraction of a concurrent data extraction worker
const fetchClientMetrics = async (clientConfig) => {
  const providers = [
    { name: 'googleAds', fetchFn: callGoogleAdsAPI },
    { name: 'metaAds', fetchFn: callMetaAdsAPI },
    { name: 'ga4', fetchFn: callGA4API }
  ];

  // Execute all network requests concurrently
  const fetchPromises = providers.map(provider => 
    provider.fetchFn(clientConfig[provider.name])
      .then(data => ({ provider: provider.name, success: true, data }))
      .catch(error => ({ provider: provider.name, success: false, error: error.message }))
  );

  const results = await Promise.allSettled(fetchPromises);
  
  const payload = {
    clientId: clientConfig.id,
    timestamp: new Date(),
    metrics: {}
  };

  results.forEach((result) => {
    if (result.status === 'fulfilled') {
      const { provider, success, data, error } = result.value;
      if (success) {
        // Normalize your data pipeline here
        payload.metrics[provider] = data;
      } else {
        console.error(`Error pulling from ${provider}:`, error);
        payload.metrics[provider] = { error: 'Extraction failed' };
      }
    }
  });

  return payload;
};

Scaling Up to Production

In a real production environment, you cannot simply let hundreds of these tasks fire randomly inside an Express controller. Furthermore, you have to back this up with a robust message broker or worker layer:

Job Queuing: Push data sync tasks to a queue backend (like AWS SQS or BullMQ).
State Management: Track extraction states dynamically in a database like MongoDB or PostgreSQL to handle retries for failed API responses.
Payload Sanitization: Run the incoming arrays through a dedicated transformer layer to map differing currency codes, timezone variations, and metric naming conventions into a unified dashboard layout.

Building the Future of Reporting

Building out reliable, fault-tolerant infrastructure that constantly listens to fluctuating external APIs takes significant time and architectural planning. Therefore, that is the exact reason we built RaiseReturn—to give agencies enterprise-level reporting infrastructure right out of the box without forcing them to spend months writing custom ETL pipelines.

If you are currently building data pipelines or wrestling with marketing API schemas, let's talk architecture in the comments below!