← thecodex.expert · The Codex Family of Knowledge
Tier 4 · Advanced · Python Project

Key-Value Store

Build a persistent key-value database with an append-only log, like a tiny Redis. Learn durable storage and crash recovery.

🧠 Teaches how to think spoonfed, every age Last verified:

1 The Problem

We want a key-value store: set, get, and delete keys, with data that survives restarts and crashes. The trick is an append-only log: every change is appended to a file, and the current state is rebuilt by replaying it. It teaches durable storage — the foundation of real databases.

Where this shows up: Redis, databases, event sourcing, write-ahead logs, message queues. The append-only log is one of the most important ideas in systems — it is how databases stay durable and recover from crashes.

2 How to Think About It

Think about surviving a crash, before any code:

The plan — in plain English
1. Keep the data in memory for fast access. → 2. Every change (set or delete) is also appended to a log file — never overwriting, just adding a line. → 3. On startup, replay the log from the beginning to rebuild the current state. → Because we only ever append, a crash mid-write loses at most the last entry — everything before is safe.

set

get

delete

Start: replay log file

Rebuild in-memory state

Operation?

Update memory + append to log

Read from memory

Remove from memory + append to log

Data is durable

3 The Build — explained part by part

Here is a complete key-value store with an append-only log. Each part is explained below.

Pythonkvstore.py
import json

class KVStore:
    def __init__(self, log_file="store.log"):
        self.log_file = log_file
        self.data = {}
        self._replay()

    def _replay(self):
        # Rebuild state by replaying every logged operation.
        try:
            with open(self.log_file) as f:
                for line in f:
                    op = json.loads(line)
                    if op["type"] == "set":
                        self.data[op["key"]] = op["value"]
                    elif op["type"] == "delete":
                        self.data.pop(op["key"], None)
        except FileNotFoundError:
            pass      # no log yet; start empty

    def _append(self, entry):
        with open(self.log_file, "a") as f:
            f.write(json.dumps(entry) + "\n")

    def set(self, key, value):
        self.data[key] = value
        self._append({"type": "set", "key": key, "value": value})

    def get(self, key):
        return self.data.get(key)

    def delete(self, key):
        self.data.pop(key, None)
        self._append({"type": "delete", "key": key})

if __name__ == "__main__":
    store = KVStore()
    store.set("name", "Alice")
    store.set("city", "Mumbai")
    print(store.get("name"))     # Alice — survives restarts
What each part does — in plain words
self.data = {} — the in-memory copy, for fast reads. The log file is the durable record.

_replay() — on startup, read the log line by line and re-apply every operation, rebuilding the exact current state. A set stores a value; a delete removes it. This is how the store survives restarts.

_append(entry) — the key idea: open the file in append mode ("a") and add one JSON line. We never rewrite the file, only add to it. Appending is fast and crash-safe.

set / delete — each updates memory and appends to the log, so the change is durable immediately.

get — just reads from memory; no disk access needed, so reads are instant.

why append-only? — if the program crashes mid-write, only the last line might be corrupt; everything before is intact. Overwriting a whole file risks losing everything.
Common mistakes — and how to avoid them
✗ Overwriting the whole file on each change — a crash mid-write can lose everything.
✓ Append one line per change; never rewrite the file.
✗ Not replaying the log on startup — data appears lost after a restart.
✓ Rebuild state by replaying the log in __init__.
✗ Reading from disk on every get — slow.
✓ Keep an in-memory copy for fast reads; the log is only for durability.

4 Test & Prove Each Part

We test set/get/delete and — crucially — that data survives a simulated restart by reloading from the log.

A set value can be read back
A deleted key returns nothing
Data survives a restart (reloaded from the log)
Pythontest_kvstore.py
import json
import os
import tempfile


class KVStore:
    def __init__(self, log_file):
        self.log_file = log_file
        self.data = {}
        self._replay()

    def _replay(self):
        try:
            with open(self.log_file) as f:
                for line in f:
                    op = json.loads(line)
                    if op["type"] == "set":
                        self.data[op["key"]] = op["value"]
                    elif op["type"] == "delete":
                        self.data.pop(op["key"], None)
        except FileNotFoundError:
            pass

    def _append(self, entry):
        with open(self.log_file, "a") as f:
            f.write(json.dumps(entry) + "\n")

    def set(self, key, value):
        self.data[key] = value
        self._append({"type": "set", "key": key, "value": value})

    def get(self, key):
        return self.data.get(key)

    def delete(self, key):
        self.data.pop(key, None)
        self._append({"type": "delete", "key": key})


def temp_log():
    return os.path.join(tempfile.mkdtemp(), "test.log")


def test_set_get():
    """A set value reads back."""
    store = KVStore(temp_log())
    store.set("a", "1")
    assert store.get("a") == "1"


def test_delete():
    """A deleted key returns None."""
    store = KVStore(temp_log())
    store.set("a", "1")
    store.delete("a")
    assert store.get("a") is None


def test_survives_restart():
    """Data persists across a reload from the log."""
    log = temp_log()
    store = KVStore(log)
    store.set("name", "Alice")
    reloaded = KVStore(log)      # simulate a restart
    assert reloaded.get("name") == "Alice"

Run with pytest -v. The test_survives_restart test is the important one: it creates a store, then creates a second store from the same log — simulating a restart — and proves the data is still there. That is durability, tested.

5 The Interface

setstore.set(key, value)store durably
getstore.get(key)read (from memory)
deletestore.delete(key)remove durably
Durability
Every set/delete is appended to the log,
so data survives restarts and crashes.

6 Run It & Automate It

Run it locally
python3 kvstore.py
Set some keys, stop, run again — they are still there, rebuilt from store.log.

Jenkins runs the tests automatically — each line explained below.

What you should see when it works
Terminala real run
Alice
If it breaks — how to fix it
🚨 Data is gone after restart.
You are not replaying the log, or not appending on set. Check both.
🚨 JSONDecodeError on replay.
A log line is corrupt (perhaps a crash mid-write). Real stores skip or truncate the bad final line.
GroovyJenkinsfile
// Jenkinsfile — automated tests on every change.
pipeline {
    agent any
    stages {
        stage('Get the code') { steps { checkout scm } }       // download the code
        stage('Install') { steps { sh 'python3 -m venv venv && . venv/bin/activate && pip install pytest' } }  // test tool
        stage('Test') { steps { sh '. venv/bin/activate && pytest -v' } }   // run every test
    }
    post {
        success { echo 'All tests passed.' }
        failure { echo 'A test failed — look above.' }
    }
}
🎯 Try this next — make it yours
  1. Compaction. Periodically rewrite the log to drop overwritten/deleted keys. (Teaches: log compaction.)
  2. Expiry. Add time-to-live on keys, like Redis. (Teaches: timestamps.)
  3. Network access. Wrap it in the socket server so clients connect. (Teaches: making it a real service.)
What you learned
You built a durable key-value store with an append-only log — the crash-safe storage technique behind Redis, databases, and event sourcing. Replaying a log to rebuild state is one of the most powerful ideas in systems. Related: json, File I/O.