Task Queue — JavaScript Project

1. The Problem

Build a task queue: producers add jobs (a function plus its arguments) to a shared queue, and a pool of N workers pull jobs off and run them concurrently. A wait() resolves once every queued job has been processed.

Why this project? Background job systems (Celery, Sidekiq, BullMQ) all share this shape: a queue plus a worker pool. Modelling it teaches the producer/consumer pattern and how a fixed number of workers drain a backlog.

2. How to Think About This

JavaScript is single-threaded, so instead of OS threads we use async worker functions and a shared array as the queue — this faithfully models the producer/consumer pattern (real Node job systems do exactly this with the event loop). Each worker loops: take the next job, await it, repeat until the queue is empty. Spin up N workers with Promise.all; when they all finish, every job is done. Because the (possibly slow, awaited) jobs interleave, they run concurrently up to the worker count.

3. The Build

queue.mjs

export class TaskQueue {
  constructor(numWorkers = 2) {
    this.numWorkers = numWorkers;
    this.jobs = [];        // shared queue: { fn, args }
    this.results = [];
  }

  add(fn, ...args) {
    this.jobs.push({ fn, args });
  }

  // One worker: pull and run jobs until the queue is empty.
  async _worker() {
    while (this.jobs.length > 0) {
      const { fn, args } = this.jobs.shift();
      const result = await fn(...args);
      this.results.push(result);
    }
  }

  // Start N workers and resolve once all jobs are processed.
  async run() {
    const workers = Array.from({ length: this.numWorkers }, () => this._worker());
    await Promise.all(workers);
    return this.results;
  }
}

// Demo: a "slow" async job.
const square = (n) => new Promise(r => setTimeout(() => r(n * n), 10));

const q = new TaskQueue(3);
for (let i = 0; i < 5; i++) q.add(square, i);
const results = await q.run();
console.log("Results:", results.sort((a, b) => a - b));   // [0, 1, 4, 9, 16]

4. Test & Prove

queue.test.mjs

import { TaskQueue } from "./queue.mjs";

let pass = 0, fail = 0;
async function test(desc, fn) {
  try { await fn(); console.log("  \u2713", desc); pass++; }
  catch (e) { console.log("  \u2717", desc, "\u2014", e.message); fail++; }
}
function assert(cond, msg) { if (!cond) throw new Error(msg || "assertion failed"); }

const slow = (n) => new Promise(r => setTimeout(() => r(n * n), 5));

await test("processes every job", async () => {
  const q = new TaskQueue(2);
  for (let i = 0; i < 5; i++) q.add(slow, i);
  const out = await q.run();
  assert(out.length === 5);
});
await test("produces correct results", async () => {
  const q = new TaskQueue(3);
  [1, 2, 3, 4].forEach(n => q.add(slow, n));
  const out = (await q.run()).sort((a, b) => a - b);
  assert(JSON.stringify(out) === JSON.stringify([1, 4, 9, 16]));
});
await test("works with a single worker", async () => {
  const q = new TaskQueue(1);
  q.add(slow, 6);
  const out = await q.run();
  assert(out[0] === 36);
});
await test("an empty queue finishes immediately", async () => {
  const out = await new TaskQueue(2).run();
  assert(out.length === 0);
});

console.log(`\n${pass} passed, ${fail} failed`);

5. The Interface

node queue.mjs

Results: [ 0, 1, 4, 9, 16 ]

6. Run It

node queue.mjs
node queue.test.mjs

🎯 Try this next

Add job priorities so urgent jobs jump the queue
Add retries: if a job throws, requeue it up to N times
Return results in submission order instead of completion order
Add a real worker pool with Node worker_threads for CPU-bound jobs

What you practised

The producer/consumer pattern, draining a shared queue with a fixed pool of async workers, coordinating completion with Promise.all, and how concurrency works on a single-threaded event loop.

Reference: Async & Promises · Arrays · Classes