Solving Scheduled ETL Race Conditions in Kubernetes: A Real Estate API Case Study

1. The Challenge: MLS-Grid API and Strict Rate Limits

The core feature of this platform required serving real-time property prices and rich media to users. To achieve this, I built an ETL (Extract, Transform, Load) engine designed to pull raw data from the MLS-grid API, transform it to fit our relational database schema, and apply indexing for optimized query performance on the frontend. However, the upstream API imposed a strict limitation: media and property pictures could only be loaded once per hour per property.

• Data Extraction: Fetch real-time listings from the external MLS-grid API.
• Transformation: Reshape the payload to match our internal SQL schema for fast indexing.
• Constraint: External media rate limits meant any redundant API calls would instantly result in blocked requests (HTTP 429).

2. The Incident: Concurrency Disasters in Production

The ETL engine was designed to trigger at midnight (3:00 AM NY time) via a scheduled cron job inside the Node.js backend. In staging (with one instance), this ran flawlessly. The problem surfaced during our production deployment on Google Cloud Platform (GCP). To handle user traffic, we scaled the web backend to run on three concurrent Kubernetes pods.

• At exactly 3:00 AM, all three pods independently triggered the ETL engine.
• The database was immediately flooded with 3x duplicated records for every property.
• Because all three pods requested the media assets simultaneously, the 1-request-per-hour MLS limit was instantly breached, causing all subsequent image fetch operations to fail entirely.

3. The Approach: Environment-Driven Worker Isolation

To solve this, I needed to decouple the background processing engine from the standard HTTP web servers without overcomplicating the infrastructure. Instead of managing complex distributed locks (like Redis Redlock) or deploying heavy message brokers, I leveraged Kubernetes Deployment manifests and environment variables to orchestrate a dedicated worker pod.

"By strictly separating HTTP traffic handlers from background data processing, we eliminated race conditions and ensured absolute control over external API consumption."

4. Implementation: The Dedicated Worker Pod

I created a secondary Kubernetes deployment using the exact same Docker image, but configured it to run as a single replica. I injected a specific environment variable (`IS_ETL: true`) into this new pod, while explicitly setting `IS_ETL: false` on the three web-facing pods. At the application layer, the ETL scheduler simply checks this flag before executing.

# Web Pods Deployment (Handles User Traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: real-estate-web
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: backend-api
        env:
        - name: IS_ETL
          value: "false"

---
# Worker Pod Deployment (Handles Background Tasks)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: real-estate-worker
spec:
  replicas: 1 # Ensures only ONE instance ever runs the job
  template:
    spec:
      containers:
      - name: backend-worker
        env:
        - name: IS_ETL
          value: "true"

5. Application Guard Logic

Inside the Node.js application codebase, the cron scheduler is wrapped in a simple initialization guard. If the container boots up as a standard web pod, the cron job is skipped entirely.

import { scheduleJob } from 'node-schedule';
import { runMlsETL } from './etl-engine';

export function initializeBackgroundJobs() {
  const isWorkerNode = process.env.IS_ETL === 'true';

  if (isWorkerNode) {
    console.log('[Worker] Initializing MLS-Grid ETL schedule (3:00 AM EST)');
    
    // Runs at 3:00 AM NY time
    scheduleJob('0 3 * * *', async () => {
      await runMlsETL();
    });
  } else {
    console.log('[Web] Skipping background jobs. IS_ETL flag is false.');
  }
}