1 The Problem
We want a distributed task system: a central coordinator hands tasks to multiple workers, collects their results, and retries tasks that fail — so the system completes all work even when individual workers err. It teaches coordination, result aggregation, and fault tolerance — the heart of distributed computing.
Where this shows up: Celery, Apache Spark, MapReduce, CI/CD runners, render farms, any system spreading work across machines. Coordinating workers and handling failures gracefully is what makes distributed systems reliable.
2 How to Think About It
Think about coordinate, collect, retry, before any code:
The plan — in plain English
1. A coordinator holds the tasks and the results. → 2. Workers take tasks and return results — but a task might fail. → 3. On failure, the coordinator retries the task (up to a limit) rather than losing it. → 4. When all tasks succeed (or exhaust retries), collect the results. Fault tolerance is the key: failures are expected and handled.
3 The Build — explained part by part
Here is a complete coordinator with retries. The logic is testable with deterministic fake workers. Each part is explained below.
Pythoncoordinator.py
class Coordinator:
def __init__(self, max_retries=3):
self.max_retries = max_retries
def run(self, tasks, worker):
# tasks: list of inputs. worker: a function that may raise on failure.
results = {}
failed = []
for task in tasks:
results[task] = self._run_with_retries(task, worker)
if results[task] is None:
failed.append(task)
return {"results": results, "failed": failed}
def _run_with_retries(self, task, worker):
# Try the task, retrying on failure up to max_retries.
for attempt in range(self.max_retries):
try:
return worker(task)
except Exception:
continue # failed; try again
return None # all retries exhausted
if __name__ == "__main__":
attempts = {}
def flaky_worker(task):
# Fails the first time, succeeds the second — simulates real flakiness.
attempts[task] = attempts.get(task, 0) + 1
if attempts[task] < 2:
raise RuntimeError("temporary failure")
return task * 10
coordinator = Coordinator(max_retries=3)
print(coordinator.run([1, 2, 3], flaky_worker))
What each part does — in plain words
run(tasks, worker) — the coordinator’s main job: run every task (via a worker function) and collect results, tracking any that ultimately failed._run_with_retries — the fault-tolerance core: try the task; if the worker raises,
continue to try again, up to max_retries times. Transient failures (a network blip, a busy server) often succeed on retry.return None after the loop — if every attempt failed, give up on this task and signal failure with
None. The coordinator records it as failed rather than crashing the whole run.the flaky_worker example — deliberately fails the first attempt and succeeds the second. This models real distributed systems, where failures are normal and retries are how you get reliability from unreliable parts.
passing worker as a function — the coordinator does not care what the work is — it just runs and retries. This separation lets the same coordinator handle any kind of task.
the big idea — in distributed systems, failure is expected. Reliability comes not from things never failing, but from handling failure gracefully — here, with retries.
Common mistakes — and how to avoid them
✗ No retries — a single transient failure loses the task.
✓ Retry failed tasks a few times; many failures are temporary.
✗ Retrying forever — a permanently broken task hangs the system.
✓ Cap retries with
max_retries and give up after.✗ Letting one task’s failure crash the whole run — you lose all results.
✓ Catch failures per task so others still complete.
4 Test & Prove Each Part
We test that work completes, that flaky tasks succeed on retry, and that permanently-failing tasks are reported — using deterministic fake workers.
Successful tasks return their results
A flaky task succeeds after a retry
A task that always fails is reported as failed
Pythontest_coordinator.py
class Coordinator:
def __init__(self, max_retries=3):
self.max_retries = max_retries
def run(self, tasks, worker):
results = {}
failed = []
for task in tasks:
results[task] = self._run_with_retries(task, worker)
if results[task] is None:
failed.append(task)
return {"results": results, "failed": failed}
def _run_with_retries(self, task, worker):
for attempt in range(self.max_retries):
try:
return worker(task)
except Exception:
continue
return None
def test_successful_tasks():
"""All tasks succeed and return results."""
result = Coordinator().run([1, 2, 3], lambda t: t * 2)
assert result["results"] == {1: 2, 2: 4, 3: 6}
assert result["failed"] == []
def test_retry_then_succeed():
"""A flaky task succeeds on the second attempt."""
attempts = {}
def flaky(task):
attempts[task] = attempts.get(task, 0) + 1
if attempts[task] < 2:
raise RuntimeError("fail")
return task
result = Coordinator(max_retries=3).run([5], flaky)
assert result["results"][5] == 5
assert result["failed"] == []
def test_permanent_failure():
"""A task that always fails is reported as failed."""
def always_fails(task):
raise RuntimeError("always")
result = Coordinator(max_retries=2).run([1], always_fails)
assert result["failed"] == [1]
Run with pytest -v. The retry-then-succeed test proves fault tolerance works: a task that fails once still completes. The permanent-failure test proves the system gives up gracefully rather than retrying forever. Together they show reliability built from unreliable parts.
5 The Interface
runcoordinator.run(tasks, worker)distribute + retry
Returns
{"results": {1: 10, 2: 20, 3: 30},
"failed": []}6 Run It & Automate It
Run it locally
python3 coordinator.pyWatch flaky tasks fail once and succeed on retry. Combine with the task-queue project for real parallel workers.
Jenkins runs the tests automatically — each line explained below.
What you should see when it works
Terminala real run
{'results': {1: 10, 2: 20, 3: 30}, 'failed': []}If it breaks — how to fix it
🚨 One failure stops everything.
Wrap each task in try/except so failures are isolated to that task.
🚨 It never finishes.
Your retry loop has no limit. Bound it with
range(max_retries).GroovyJenkinsfile
// Jenkinsfile — automated tests on every change.
pipeline {
agent any
stages {
stage('Get the code') { steps { checkout scm } } // download the code
stage('Install') { steps { sh 'python3 -m venv venv && . venv/bin/activate && pip install pytest' } } // test tool
stage('Test') { steps { sh '. venv/bin/activate && pytest -v' } } // run every test
}
post {
success { echo 'All tests passed.' }
failure { echo 'A test failed — look above.' }
}
}
🎯 Try this next — make it yours
- Real parallelism. Run workers in threads (combine with the task-queue project). (Teaches: true distribution.)
- Backoff. Wait longer between each retry. (Teaches: exponential backoff.)
- Worker failure. Reassign tasks if a whole worker dies. (Teaches: deeper fault tolerance.)
What you learned
You built a distributed task coordinator with result collection and retries — the fault-tolerance at the heart of Celery, Spark, and MapReduce. The key lesson: in distributed systems, failure is expected, and reliability comes from handling it gracefully. Related: queue, Concurrency.