Distributed Task System — JavaScript Project

1. The Problem

Build a coordinator that runs a batch of tasks through a worker function. Real workers fail intermittently, so each task is retried up to a maximum number of attempts. The coordinator returns the successful results plus a list of any tasks that failed even after all retries.

Why this project? Distributed systems assume failure is normal — networks blip, workers die. Retry-with-a-limit and tracking permanent failures is the reliability pattern behind Celery, Temporal, and every job coordinator. This logic ports fully; only the "where the worker runs" changes.

2. How to Think About This

For each task, try the worker; if it throws, catch and try again, up to maxRetries. If an attempt succeeds, keep that result. If every attempt fails, record the task as permanently failed (result null). The coordinator gathers per-task results and a failed list. In a real system the worker runs on another machine and "failure" is a network timeout — but the retry-and-track logic is exactly this.

3. The Build

coordinator.mjs

export class Coordinator {
  constructor(maxRetries = 3) { this.maxRetries = maxRetries; }

  run(tasks, worker) {
    const results = {};
    const failed = [];
    for (const task of tasks) {
      const result = this._runWithRetries(task, worker);
      results[task] = result;
      if (result === null) failed.push(task);
    }
    return { results, failed };
  }

  // Try the task, retrying on failure up to maxRetries.
  _runWithRetries(task, worker) {
    for (let attempt = 0; attempt < this.maxRetries; attempt++) {
      try {
        return worker(task);
      } catch {
        continue;   // failed; try again
      }
    }
    return null;     // all retries exhausted
  }
}

// Demo: a worker that fails the first time, then succeeds (simulated flakiness).
const attempts = {};
function flakyWorker(task) {
  attempts[task] = (attempts[task] || 0) + 1;
  if (attempts[task] < 2) throw new Error("temporary failure");
  return task * 10;
}

const coordinator = new Coordinator(3);
console.log(coordinator.run([1, 2, 3], flakyWorker));

4. Test & Prove

coordinator.test.mjs

import { Coordinator } from "./coordinator.mjs";

let pass = 0, fail = 0;
function test(desc, fn) {
  try { fn(); console.log("  \u2713", desc); pass++; }
  catch (e) { console.log("  \u2717", desc, "\u2014", e.message); fail++; }
}
function assert(cond, msg) { if (!cond) throw new Error(msg || "assertion failed"); }

test("a reliable worker succeeds on every task", () => {
  const c = new Coordinator(3);
  const out = c.run([1, 2, 3], t => t * 2);
  assert(out.results[1] === 2 && out.results[3] === 6 && out.failed.length === 0);
});
test("a flaky task succeeds within the retry budget", () => {
  const tries = {};
  const worker = t => {
    tries[t] = (tries[t] || 0) + 1;
    if (tries[t] < 2) throw new Error("flaky");
    return t * 10;
  };
  const out = new Coordinator(3).run([5], worker);
  assert(out.results[5] === 50 && out.failed.length === 0);
});
test("a task failing past the limit is marked failed", () => {
  const out = new Coordinator(3).run([1], () => { throw new Error("always"); });
  assert(out.results[1] === null && out.failed.includes(1));
});
test("retries stop at the configured maximum", () => {
  let calls = 0;
  new Coordinator(3).run([1], () => { calls++; throw new Error("x"); });
  assert(calls === 3);
});

console.log(`\n${pass} passed, ${fail} failed`);

5. The Interface

node coordinator.mjs

{ results: { '1': 10, '2': 20, '3': 30 }, failed: [] }

6. Run It

node coordinator.mjs
node coordinator.test.mjs

# In a real system the worker runs on another machine (over HTTP/gRPC) and
# "failure" is a network timeout; the retry-and-track logic is unchanged.

🎯 Try this next

Add exponential backoff — wait longer between each retry
Dispatch tasks to real remote workers over HTTP and await results
Add a dead-letter list that stores failed tasks for later inspection
Run independent tasks concurrently with Promise.all instead of one at a time

What you practised

The retry-with-a-limit reliability pattern, distinguishing a transient failure (retry) from a permanent one (record it), and separating coordination logic (portable) from where the worker actually runs.

Reference: Error Handling · Classes · Control Flow