Throttling

Definition

Throttling is a traffic control mechanism that regulates the rate of API requests by delaying or queuing excess requests instead of outright rejecting them. Unlike rate limiting, which acts as a hard barrier (you hit the limit, you’re blocked), throttling acts more like a traffic light - it slows down the flow to keep everything moving smoothly. It ensures that no single client overwhelms your system while maintaining service availability.

The key difference between throttling and rate limiting is in how they handle excess traffic. Rate limiting says “you’ve used your 100 requests per minute, come back later.” Throttling says “you’re making requests too fast, I’ll process them but slower.” This makes throttling feel less abrupt to users while still protecting your infrastructure.

Throttling is essential for maintaining system stability under variable load. During traffic spikes, it prevents cascading failures by applying backpressure - slowing down upstream clients when downstream services can’t keep up. It’s like having an intelligent buffer that adapts to system capacity in real-time.

Example

Video Streaming Services: YouTube throttles video downloads based on your viewing speed. If you’re watching at normal speed, it downloads ahead but stops buffering when it has enough. This prevents users from downloading entire movies when they’ll only watch 5 minutes, saving bandwidth for others.

Cloud Storage Upload: Dropbox throttles large uploads during peak hours. Instead of rejecting your 50GB folder upload, it queues the files and processes them at a controlled rate. You see progress, just slower than during off-peak times. This keeps the service responsive for everyone.

Payment Processors: Stripe throttles bulk payment operations. If you try to process 10,000 payments at once, they’ll accept the batch but process them at a controlled rate (maybe 100/second) to prevent overloading their fraud detection systems. All payments eventually process, just not all at once.

API Gateway During Traffic Spike: During a viral marketing campaign, your API might receive 100x normal traffic. Instead of crashing, the gateway throttles incoming requests to match your backend capacity (maybe 1000 req/sec). Requests queue up, users see slower responses (2-3 seconds instead of 200ms), but the service stays online.

Database Connection Pool: When your app receives a surge of traffic, the connection pool throttles database queries. Instead of opening 10,000 simultaneous connections (which would crash the DB), it maintains a fixed pool (maybe 100 connections) and queues the rest. Each query waits its turn, preventing system collapse.

Analogy

The Highway Merge Lane: When a highway narrows from 3 lanes to 1, cars don’t crash or get rejected. They throttle - merging one by one in a controlled manner. Traffic slows down, but everyone eventually gets through. Without throttling, you’d have gridlock or accidents.

The Restaurant Kitchen: During a busy dinner rush, the kitchen doesn’t reject reservations. Instead, they throttle service - accepting orders but pacing the cooking to match chef capacity. Customers wait longer for food, but everyone gets served. The alternative (trying to cook everything at once) would result in chaos and burnt meals.

The Checkout Line: A store doesn’t close when a line forms. Customers queue and checkout at the cashier’s natural pace (throttling). The line gets longer during peaks, but everyone waits their turn. If they tried to process everyone simultaneously, it would be chaos.

The Water Pressure Regulator: When too many people turn on taps simultaneously, a pressure regulator throttles water flow to prevent pipe damage. Everyone still gets water, just at a controlled rate. Without throttling, the sudden pressure drop could burst pipes.

Code Example

// Client-side throttling with backoff
class ThrottledAPIClient {
  constructor(maxRequestsPerSecond = 10) {
    this.maxRPS = maxRequestsPerSecond;
    this.requestQueue = [];
    this.processing = false;
  }

  async request(url, options) {
    return new Promise((resolve, reject) => {
      this.requestQueue.push({ url, options, resolve, reject });
      this.processQueue();
    });
  }

  async processQueue() {
    if (this.processing || this.requestQueue.length === 0) return;

    this.processing = true;
    const delayBetweenRequests = 1000 / this.maxRPS;

    while (this.requestQueue.length > 0) {
      const { url, options, resolve, reject } = this.requestQueue.shift();

      try {
        const response = await fetch(url, options);

        // Server indicates throttling
        if (response.status === 429) {
          const retryAfter = response.headers.get('Retry-After') || 5;
          console.log(`Throttled by server. Waiting ${retryAfter}s...`);
          await new Promise(r => setTimeout(r, retryAfter * 1000));

          // Re-queue the request
          this.requestQueue.unshift({ url, options, resolve, reject });
          continue;
        }

        resolve(response);
      } catch (error) {
        reject(error);
      }

      // Throttle: wait before next request
      await new Promise(r => setTimeout(r, delayBetweenRequests));
    }

    this.processing = false;
  }
}

// Usage
const client = new ThrottledAPIClient(10); // 10 requests/second max

// Make 100 requests - they'll be throttled automatically
for (let i = 0; i < 100; i++) {
  client.request('/api/data', { method: 'GET' })
    .then(res => res.json())
    .then(data => console.log(`Request ${i} completed`));
}

Diagram

graph TB
    subgraph Client["Client Application"]
        C1[Request 1]
        C2[Request 2]
        C3[Request 3]
        C4[Request N...]
    end

    subgraph Throttle["Throttling Layer
(API Gateway)"]
        Q[Request Queue]
        R[Rate Controller
100 req/sec]
        M[Monitor Backend Load]
    end

    subgraph Backend["Backend Services"]
        B1[Service Instance 1]
        B2[Service Instance 2]
        DB[(Database)]
    end

    C1 --> Q
    C2 --> Q
    C3 --> Q
    C4 --> Q

    Q --> R
    M -.Check Capacity.-> B1
    M -.Check Capacity.-> B2
    M -.Adjust Rate.-> R

    R -->|Controlled Flow
100 req/sec| B1
    R -->|Controlled Flow
100 req/sec| B2
    B1 --> DB
    B2 --> DB

    style Q fill:#fff4e6
    style R fill:#ffe6e6
    style M fill:#e6f3ff
    style Throttle fill:#f0f0f0

Security Notes

Best Practices

1. Layer Your Defense: Combine throttling (smooths traffic) with rate limiting (hard caps) and circuit breakers (fail fast when downstream is down)
2. Adaptive Throttling: Adjust throttle rate based on real-time backend capacity, not just static configuration
3. Queue Management: Set maximum queue sizes to prevent memory exhaustion. Better to reject than queue forever
4. Transparent Communication: Use Retry-After headers and 429 status codes to tell clients when they can retry
5. Per-Resource Throttling: Different endpoints have different costs. Throttle expensive operations (searches, reports) more aggressively than cheap ones (static data)
6. Prioritize Traffic: Give higher priority to authenticated users over anonymous, or paid tiers over free
7. Graceful Degradation: When throttling, consider returning cached/stale data instead of making clients wait
8. Monitor and Alert: Track throttle rates, queue depths, and rejection rates. Alert when thresholds exceed normal patterns

Common Mistakes

Not Distinguishing Throttling from Rate Limiting: Teams often confuse the two. Rate limiting is a hard stop (403/429), throttling is intentional slowing (queuing, delaying). Use both together for best results.

Client-Side Only Throttling: Trusting clients to throttle themselves is naive. Malicious actors will bypass it. Always enforce server-side.

Infinite Queues: Queuing requests without a maximum queue size leads to memory exhaustion. Set limits and reject when full.

No Retry-After Header: When throttling, always tell clients when to retry. Without guidance, they’ll hammer your API with retries.

Fixed Throttle Rates: Static throttle rates (always 100 req/sec) don’t adapt to changing load. Use dynamic throttling based on backend health.

Throttling Authentication Endpoints: Be careful throttling login/signup. Too aggressive and legitimate users can’t access your service. Too lenient and you’re vulnerable to brute force.

No Differentiation: Throttling all traffic equally treats a DDoS attack the same as a premium customer. Implement tiered throttling.

Logging Every Throttled Request: Throttling generates high volumes of events. Log samples or aggregates, not every single throttled request, or you’ll overwhelm your logging system.

Standards & RFCs

Related Terms