thecodex.expert · The Codex Family of Knowledge
SQL

Advanced SQL

CTEs for readable queries, indexes for fast queries, transactions for safe writes — SQL beyond the basics.

ISO/IEC 9075 PostgreSQL 16 Last verified:
Canonical Definition

Advanced SQL covers CTEs (named subqueries for readability), recursive CTEs (tree/hierarchy traversal), DDL (CREATE TABLE with constraints, indexes, views), transactions implementing ACID guarantees, and query analysis with EXPLAIN ANALYZE.

One sentence

CTEs, transactions, indexes, and schema design are where SQL stops being just "query language" and becomes a complete data engineering toolkit.

CTEs — Common Table Expressions

SQLctes.sql
-- CTE: a named subquery defined before the main query
-- Syntax: WITH name AS (subquery) SELECT ... FROM name

-- Simple CTE — improves readability
WITH high_earners AS (
    SELECT id, name, salary, department
    FROM   employees
    WHERE  salary > 80000
)
SELECT department, COUNT(*) AS count, AVG(salary) AS avg
FROM   high_earners
GROUP  BY department;

-- Multiple CTEs — chain them together
WITH
    dept_stats AS (
        SELECT   department_id,
                 AVG(salary) AS avg_sal,
                 COUNT(*)    AS headcount
        FROM     employees
        GROUP BY department_id
    ),
    large_depts AS (
        SELECT * FROM dept_stats WHERE headcount > 10
    )
SELECT d.name, l.avg_sal, l.headcount
FROM   large_depts l
JOIN   departments d ON d.id = l.department_id
ORDER  BY l.avg_sal DESC;

-- Recursive CTE — traverse hierarchies (org charts, trees)
WITH RECURSIVE org_chart AS (
    -- Base case: the root (CEO has no manager)
    SELECT id, name, manager_id, 0 AS depth
    FROM   employees
    WHERE  manager_id IS NULL

    UNION ALL

    -- Recursive case: employees whose manager is already in the CTE
    SELECT e.id, e.name, e.manager_id, oc.depth + 1
    FROM   employees e
    JOIN   org_chart oc ON e.manager_id = oc.id
)
SELECT REPEAT('  ', depth) || name AS hierarchy, depth
FROM   org_chart
ORDER  BY depth, name;

DDL — CREATE, ALTER, constraints

SQLddl.sql
-- CREATE TABLE with constraints
CREATE TABLE orders (
    id          BIGSERIAL PRIMARY KEY,         -- auto-incrementing PK
    customer_id INTEGER   NOT NULL
                REFERENCES customers(id)       -- foreign key
                ON DELETE CASCADE,             -- delete orders if customer deleted
    status      VARCHAR(20) NOT NULL
                DEFAULT 'pending'
                CHECK (status IN ('pending','confirmed','shipped','cancelled')),
    total       NUMERIC(12, 2) CHECK (total >= 0),
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Indexes — dramatically speed up queries on large tables
CREATE INDEX idx_orders_customer ON orders (customer_id);
CREATE INDEX idx_orders_status   ON orders (status);
-- Composite index for queries filtering on both
CREATE INDEX idx_orders_cust_status ON orders (customer_id, status);
-- Partial index — only index a subset of rows
CREATE INDEX idx_pending_orders ON orders (created_at)
    WHERE status = 'pending';

-- Altering a table
ALTER TABLE orders ADD COLUMN notes TEXT;
ALTER TABLE orders ALTER COLUMN total SET NOT NULL;
ALTER TABLE orders DROP COLUMN notes;
ALTER TABLE orders ADD CONSTRAINT check_total CHECK (total >= 0);

-- Views — saved queries that look like tables
CREATE VIEW active_orders AS
    SELECT o.*, c.name AS customer_name
    FROM   orders o JOIN customers c ON o.customer_id = c.id
    WHERE  o.status NOT IN ('cancelled');

Transactions and ACID

SQLtransactions.sql
-- Transaction: a sequence of operations that succeeds or fails atomically
BEGIN;

UPDATE accounts SET balance = balance - 500 WHERE id = 1;
UPDATE accounts SET balance = balance + 500 WHERE id = 2;

-- If anything fails, ROLLBACK undoes everything
-- If everything succeeds, COMMIT makes changes permanent
COMMIT;

-- ROLLBACK example
BEGIN;
DELETE FROM orders WHERE customer_id = 42;
-- Oh wait, we did not want to delete all of them
ROLLBACK;  -- nothing was deleted

-- Savepoints — partial rollback within a transaction
BEGIN;
INSERT INTO orders (customer_id, total) VALUES (1, 500);
SAVEPOINT order_inserted;
INSERT INTO order_items (order_id, product_id) VALUES (1, 99);  -- fails
ROLLBACK TO SAVEPOINT order_inserted;  -- undo only the items insert
INSERT INTO order_items (order_id, product_id) VALUES (1, 100); -- try again
COMMIT;

-- ACID properties:
-- Atomicity:   all or nothing — partial failures roll back completely
-- Consistency: data satisfies all constraints before and after
-- Isolation:   concurrent transactions do not interfere
-- Durability:  committed transactions survive system crashes (WAL)

EXPLAIN ANALYZE — understanding query performance

SQLexplain.sql
-- EXPLAIN — show the query plan (estimated costs)
EXPLAIN SELECT * FROM orders WHERE customer_id = 42;

-- EXPLAIN ANALYZE — run the query and show actual vs estimated
EXPLAIN ANALYZE
SELECT o.id, o.total, c.name
FROM   orders    o
JOIN   customers c ON o.customer_id = c.id
WHERE  o.status = 'pending'
ORDER  BY o.created_at;

-- Key things to look for in the output:
-- Seq Scan: full table scan (may need an index)
-- Index Scan: uses an index (good)
-- Rows: estimated vs actual rows (large difference = stale statistics)
-- Cost: startup cost..total cost (in arbitrary units)
-- Loops: how many times this node executed

-- Run ANALYZE to update statistics when rows change significantly
ANALYZE orders;

-- Index types
CREATE INDEX USING btree ON orders (created_at);  -- default, range queries
CREATE INDEX USING hash  ON orders (customer_id); -- equality only, faster
CREATE INDEX USING gin   ON products (tags);      -- arrays, JSONB, full-text
CREATE INDEX USING gist  ON locations (coords);   -- geometric data

JSON and JSONB in PostgreSQL

SQLjsonb.sql
-- PostgreSQL has first-class JSON support
CREATE TABLE products (
    id      SERIAL PRIMARY KEY,
    name    TEXT,
    details JSONB   -- JSONB: binary JSON, indexed, faster than JSON
);

INSERT INTO products (name, details) VALUES
    ('Laptop', '{"brand":"Dell","specs":{"ram":16,"ssd":512},"tags":["work","gaming"]}');

-- Access JSON fields
SELECT details->'brand' AS brand,           -- returns JSON: "Dell"
       details->>'brand' AS brand_text,      -- returns text:  Dell
       details->'specs'->>'ram' AS ram       -- nested: 16
FROM   products;

-- Filter on JSON field
SELECT * FROM products WHERE details->>'brand' = 'Dell';
SELECT * FROM products WHERE (details->'specs'->'ram')::int > 8;

-- JSON array contains
SELECT * FROM products WHERE details->'tags' ? 'gaming';

-- Index JSONB for fast queries
CREATE INDEX idx_products_brand ON products ((details->>'brand'));
CREATE INDEX idx_products_gin   ON products USING gin (details);  -- all keys
Commonly confused
CTEs are not always faster than subqueries. In PostgreSQL versions before 12, CTEs were "optimization fences" — the query planner could not push WHERE conditions into a CTE. Since PostgreSQL 12, CTEs are inlined by default (the planner can optimize through them). CTEs are primarily a readability and reuse tool; do not assume they are a performance optimization.
An index does not always get used. The query planner estimates costs and may choose a sequential scan over an index if it estimates that more than ~10-15% of rows will be returned (a table scan is faster for large result sets because it avoids random I/O). Use EXPLAIN ANALYZE to see what actually happened.
TRUNCATE is not the same as DELETE without WHERE. DELETE FROM table (without WHERE) removes all rows one by one, fires triggers, and can be rolled back. TRUNCATE removes all rows instantly without scanning them, does not fire row-level triggers, and in PostgreSQL can be rolled back within a transaction. TRUNCATE resets sequences; DELETE does not.
How this connects
Requires first

Transaction isolation levels and MVCC

SQL defines four transaction isolation levels (ISO/IEC 9075 §4.32): READ UNCOMMITTED (can see uncommitted changes — "dirty reads"), READ COMMITTED (default in PostgreSQL — only sees committed data), REPEATABLE READ (same query returns same results within a transaction), SERIALIZABLE (equivalent to serial execution). PostgreSQL implements isolation via MVCC (Multi-Version Concurrency Control) — each row has a creation transaction ID (xmin) and a deletion transaction ID (xmax). A read sees a "snapshot" of row versions committed before the transaction began, without blocking writers. This is why PostgreSQL readers never block writers and writers never block readers.

Write-Ahead Log (WAL) and durability

PostgreSQL's durability guarantee is implemented by the Write-Ahead Log (WAL). Before any data page is modified, the change is written to the WAL (a sequential append-only log on disk). If the database crashes, PostgreSQL replays the WAL to recover to a consistent state. This means COMMIT returns to the client only after the WAL record is flushed to disk (fsync). WAL is also the mechanism for streaming replication — standby servers replay the primary's WAL in real time. The wal_level, archive_mode, and max_wal_senders settings control replication behaviour.

Sources

1
PostgreSQL 16 — WITH Queries (CTEs). postgresql.org/docs/current/queries-with.html.
2
PostgreSQL 16 — Indexes. postgresql.org/docs/current/indexes.html.
3
PostgreSQL 16 — Transaction Isolation. postgresql.org/docs/current/transaction-iso.html.
4
ISO/IEC 9075-2:2023 — SQL Foundation. iso.org/standard/76583.html.
Source confidence: High Last verified: Primary source: ISO/IEC 9075 · postgresql.org/docs/

Sources

1
PostgreSQL 16 — WITH Queries (CTEs). postgresql.org/docs/current/queries-with.html.
2
PostgreSQL 16 — Indexes. postgresql.org/docs/current/indexes.html.
3
PostgreSQL 16 — Transaction Isolation. postgresql.org/docs/current/transaction-iso.html.
4
ISO/IEC 9075-2:2023 — SQL Foundation. iso.org/standard/76583.html.