# THE CODEX CODING — PYTHON — FULL CORPUS (llms-full.txt) # Source: https://thecodex.expert/coding/languages/python/ # Free programming reference. Official sources only (docs.python.org, PEPs, Language Reference). # Three reading levels per page: Beginner, Intermediate, Expert. # This file concatenates the full text of every Python reference page for AI ingestion. # Generated: 2026-06-20 | Pages: 60 | Mumbai, India # # STRUCTURE: Each page below is delimited by "================" and gives its URL, # canonical definition, three-level explanation, and primary sources. # Every factual claim traces to a named official source. # # Topics: Python language features (variables, types, control flow, functions, # closures, decorators, generators, context managers, OOP, dataclasses, enums, # match/case, error handling, the data model, iterators, lambda, walrus operator, # type hints, concurrency) and 31 standard-library modules. ================================================================================ # What is Python? ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/what-is-python/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *Python is the world's most popular programming language for beginners and professionals alike. Here's what it actually is, what it does, and why it exists.* ## CANONICAL DEFINITION Python is a high-level, interpreted, general-purpose programming language designed for readability and simplicity, created by Guido van Rossum and first released in 1991, now maintained by the Python Software Foundation. ## SOURCES - Python Software Foundation — What is Python? python.org/about/. - Python Documentation — Python for Beginners. docs.python.org/3/. - Python Language Reference — Execution model. docs.python.org/3/reference/executionmodel.html. **Primary source:** docs.python.org/3/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Setting Up Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/python-setup/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *Install Python, pick an editor, and write your first program. From zero to running code in under 10 minutes.* ## CANONICAL DEFINITION Setting up Python means installing the Python interpreter on your computer, choosing a code editor, and learning to run .py files from the terminal or editor. **Primary source:** docs.python.org/3/using/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Variables and Types in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/variables-and-types/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Python has no type declarations — assign a value and Python figures out the rest. But that doesn't mean it's careless with types.* ## CANONICAL DEFINITION In Python, a variable is a name bound to an object — every value is an instance of some class, types are determined at runtime (dynamic typing), and the language enforces type compatibility without silent coercion (strong typing). ## BEGINNER > The analogy A variable is a label on a box. Python doesn't care what's in the box — you can swap the contents for anything. The label just tells you where to find it. ## Variables In Python, you create a variable by writing a name, an equals sign, and a value. No type declaration needed — Python figures out the type from the value you assign. You can reassign a variable to a completely different type at any time. ```python # Create a variable — no declaration keyword name = "Priya" # str age = 28 # int height = 5.4 # float is_active = True # bool nothing = None # NoneType # Reassign — Python allows changing type age = "twenty-eight" # now a str — perfectly valid # Multiple assignment x, y, z = 1, 2, 3 # tuple unpacking a = b = c = 0 # same value, three names # Check the type print(type(name)) # print(type(age)) # (before reassignment) # Variable names: lowercase, underscores, descriptive user_score = 95 # good userScore = 95 # camelCase — works but not Python style GRAVITY = 9.8 # UPPERCASE = conventional constant ``` ## The built-in types | Type | Example | Mutable? | Notes | | --- | --- | --- | --- | | `int` | `42`, `-7`, `0` | No | Arbitrary precision — no overflow | | `float` | `3.14`, `1e10` | No | IEEE 754 double precision | | `complex` | `2+3j` | No | Built-in complex numbers | | `bool` | `True`, `False` | No | Subclass of int: True==1, False==0 | | `str` | `"hello"`, `'world'` | No | Unicode text, immutable | | `bytes` | `b"data"` | No | Raw binary data | | `list` | `[1, 2, 3]` | **Yes** | Ordered, indexed, heterogeneous | | `tuple` | `(1, 2, 3)` | No | Immutable list — use for fixed data | | `dict` | `{"a": 1}` | **Yes** | Key-value, insertion-ordered (3.7+) | | `set` | `{1, 2, 3}` | **Yes** | Unordered, unique values | | `frozenset` | `frozenset({1,2})` | No | Immutable set | | `NoneType` | `None` | No | The absence of a value | ```python # Numbers x = 42 # int y = 3.14 # float z = 2 + 3j # complex # Python ints have no overflow big = 10 ** 100 # a googol — works fine # Strings — single or double quotes, same thing s1 = "hello" s2 = 'hello' s3 = """Multi line string""" # f-strings (Python 3.6+) — the preferred way name = "Priya" msg = f"Hello, {name}! Age: {age}" # Collections fruits = ["mango", "apple", "banana"] # list — mutable point = (10, 20) # tuple — immutable scores = {"Priya": 95, "Rahul": 88} # dict unique = {1, 2, 3, 2, 1} # set — {1, 2, 3} # None — Python's null equivalent result = None print(result is None) # True — use `is`, not == ``` ▶ **Try it:** Open the [Python Playground](/coding/playground/python/) and type `print(type(42))`, then `print(type("hello"))`, then `print(type([1,2,3]))`. The output shows you the type of each value in real time. ✅ Beginner tab complete — check your understanding - I can create a variable with `name = value` and reassign it to a different type - I know the difference between a `list` (mutable) and a `tuple` (immutable) - I know the difference between a `dict` (key-value pairs) and a `set` (unique values) - I know that `None` represents "no value" and that I check it with `is None` - I know Python variable names use lowercase_underscores, not camelCase **Ticked all five?** Click the **🔵 Intermediate** tab to go deeper — or come back to it after you've done the next two lessons first. The Intermediate tab explains *why* Python's type system works the way it does, and introduces the concept that trips up almost every beginner: mutability. ## INTERMEDIATE ## How Python's type system works Python is **dynamically typed** — types are checked at runtime, not compile time. Python is also **strongly typed** — it will not silently coerce incompatible types. `"5" + 3` raises a `TypeError` in Python (unlike JavaScript where it gives `"53"`). Every object in Python carries its type at runtime — you can always check with `type()` or `isinstance()`. ## Type hints (PEP 484, Python 3.5+) Python 3.5 added optional type annotations. They are not enforced at runtime but are used by static analysis tools (mypy, pyright, Pyrefly) and IDEs. The Python interpreter completely ignores them at runtime — `x: int = "hello"` runs without error. ```python from typing import Optional, Union # Function with type hints def greet(name: str, times: int = 1) -> str: return (f"Hello, {name}! " * times).strip() # Variable annotations score: int = 95 label: str # Optional — the value may be None def find_user(id: int) -> Optional[str]: if id == 1: return "Priya" return None # Union — one of several types (Python 3.10+ shorthand: str | int) def process(value: Union[str, int]) -> str: return str(value) # Modern syntax (3.10+) def divide(a: float, b: float) -> float | None: return a / b if b != 0 else None # Type annotations don't change runtime behaviour x: int = "hello" # No error at runtime! print(type(x)) # ``` ## Mutability in depth Mutability is one of Python's most important concepts. Immutable objects (int, str, tuple, frozenset) cannot be changed after creation — what looks like "changing a string" always creates a new string. Mutable objects (list, dict, set) can be modified in-place. This matters enormously when passing objects to functions and when using objects as dictionary keys (only immutable objects are hashable and can be keys). ```python # Immutable — "changing" creates a new object s = "hello" id_before = id(s) s = s + " world" # new string object print(id(s) == id_before) # False — different object # Mutable — modified in place lst = [1, 2, 3] id_before = id(lst) lst.append(4) print(id(lst) == id_before) # True — same object # Tuple vs list — same data, different mutability point_list = [10, 20] # mutable point_tuple = (10, 20) # immutable # Only immutable objects can be dict keys / set members d = {(1, 2): "a point"} # tuple key — OK # d = {[1, 2]: "a point"} # TypeError: list is unhashable # Gotcha: mutable default argument def add_item(item, lst=[]): # DON'T do this lst.append(item) return lst print(add_item(1)) # [1] print(add_item(2)) # [1, 2] — shared list! def add_item_safe(item, lst=None): # correct pattern if lst is None: lst = [] lst.append(item) return lst ``` ## Names, objects, and identity In Python, variables are names bound to objects — not boxes containing values. When you write `a = b`, you bind the name `a` to the same object that `b` points to. For mutable objects, this means both `a` and `b` can see mutations. `is` checks identity (same object), `==` checks equality (same value). ```python a = [1, 2, 3] b = a # b and a point to the SAME list b.append(4) print(a) # [1, 2, 3, 4] — a sees the change! c = a.copy() # c is a new list with the same values c.append(5) print(a) # [1, 2, 3, 4] — a unchanged # Identity vs equality x = [1, 2, 3] y = [1, 2, 3] print(x == y) # True — same values print(x is y) # False — different objects # Small int caching: CPython caches -5 to 256 a = 256; b = 256 print(a is b) # True — same cached object a = 257; b = 257 print(a is b) # False — not cached (implementation detail) ``` **Commonly confused:** - Dynamic typing is not the same as weak typing. Python is dynamically typed (types checked at runtime) AND strongly typed (no silent coercion between incompatible types). JavaScript is weakly typed — it will coerce "5" + 3 to "53". Python raises TypeError. - is is not ==. is checks object identity (same memory address). == checks equality (same value). Always use == for value comparison. Use is only for None, True, False, and explicit identity checks. How this connects Requires first No prerequisites — this is the starting point Enables [Functions & Scope](/coding/languages/python/functions-and-scope/)[OOP in Python](/coding/languages/python/oop/) Primary source [Python Built-in Types — docs.python.org](https://docs.python.org/3/library/stdtypes.html) ✅ Intermediate tab complete — check your understanding - I can explain the difference between dynamically typed and strongly typed - I understand why `b = a` on a list means both names see the same mutations - I know never to use a mutable object (like `[]` or `{}`) as a default argument - I know to use `==` for value comparison and `is` only for `None` - I understand what a type hint like `def greet(name: str) -> str` means, even if I'm not using them yet **Want to go deeper?** The **🔴 Expert** tab explains how Python's object model works at the CPython level — why `id()` returns a memory address, how reference counting garbage collection works, and the full type hierarchy. Save it for when you're curious about *why*, not just *what*. **Ready to move on?** Go to [Control Flow & Loops →](/coding/languages/python/control-flow/) — that's the next lesson in the course. ## EXPERT ## Python's data model: everything is an object In Python, every value is an instance of some class — integers, strings, functions, classes themselves, and even `None`. The Python data model (Python Language Reference §3) defines the abstract base for all objects: every object has an identity (`id()` returns the memory address in CPython), a type (`type()`), and a value. The `id()` of an object is unique and constant for its lifetime. CPython's implementation uses reference counting: each object stores how many names point to it. When the count reaches zero, the object is freed. ## The type hierarchy Python's type hierarchy (Python Language Reference §3.2): None · NotImplemented · Ellipsis → numbers (int → bool, complex, float) → sequences (immutable: str, bytes, tuple; mutable: list, bytearray) → set types (frozenset, set) → mappings (dict) → callables (functions, methods, classes) → modules → types. All types are instances of `type`; `type` itself is an instance of `type` (the metaclass of everything). ## Integer interning and string interning CPython interns (caches) small integers from -5 to 256 — all accesses to these integers reuse the same object. CPython also interns string literals that look like identifiers (alphanumeric, starting with letter or underscore) at compile time. This is an implementation detail of CPython, not guaranteed by the Python language specification. Code that relies on `is` for integers or strings is implementation-specific and may break in PyPy or other implementations. ✅ Expert tab complete - I can explain why `id(x)` returns a memory address in CPython - I know that `bool` is a subclass of `int`, which is why `True == 1` - I understand reference counting — when no names point to an object, it is freed - I know never to rely on integer or string interning in production code **You've completed all three levels of this page.** Continue the course: [Control Flow & Loops →](/coding/languages/python/control-flow/) ## SOURCES - Python Language Reference §3 — Data model. docs.python.org/3/reference/datamodel.html. - Python Language Reference §4 — Execution model. docs.python.org/3/reference/executionmodel.html. - PEP 484 — Type Hints. peps.python.org/pep-0484/. - Python Built-in Types documentation. docs.python.org/3/library/stdtypes.html. **Primary source:** docs.python.org/3/reference/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # f-strings in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/f-strings/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *f-strings (formatted string literals) embed expressions directly inside string literals using {} — the fastest, most readable way to format strings in modern Python.* ## CANONICAL DEFINITION An f-string (formatted string literal, PEP 498) is a string prefixed with f or F whose contents may contain replacement fields — expressions enclosed in curly braces {} that are evaluated at runtime and formatted into the string. f-strings are evaluated at runtime and are faster than str.format() or %-formatting. ## BEGINNER > The idea An f-string lets you drop variables and expressions directly into a string by putting them in `{curly braces}`. Put an `f` before the opening quote, and everything in braces is evaluated and inserted. ## What is an f-string? Introduced in Python 3.6 (PEP 498), f-strings are the modern standard for string formatting. They are more readable and faster than the older `%` formatting and `str.format()` methods. ```python name = "Priya" age = 30 # f-string — put f before the quote, expressions in {} greeting = f"Hello, {name}! You are {age} years old." print(greeting) # Hello, Priya! You are 30 years old. # Any expression works inside the braces print(f"Next year you will be {age + 1}.") # Next year you will be 31. print(f"Your name has {len(name)} letters.") # Your name has 5 letters. # Compare with older methods (avoid these in new code) old1 = "Hello, %s! You are %d." % (name, age) # %-formatting (oldest) old2 = "Hello, {}! You are {}.".format(name, age) # str.format() new = f"Hello, {name}! You are {age}." # f-string (best) ``` ## Format specifiers After the expression, add a colon and a format specifier to control how the value is displayed — decimal places, padding, alignment, number bases, and more. ```python pi = 3.14159265 # Decimal places — :.2f means 2 decimal places, float print(f"{pi:.2f}") # 3.14 print(f"{pi:.4f}") # 3.1416 # Thousands separator print(f"{1234567:,}") # 1,234,567 # Percentage print(f"{0.875:.1%}") # 87.5% # Padding and alignment (width 10) print(f"{'left':10}|") # right| print(f"{'center':^10}|") # center | # Number bases print(f"{255:b}") # 11111111 (binary) print(f"{255:x}") # ff (hex) print(f"{255:o}") # 377 (octal) # Combine: pad a float to width 8, 2 decimals print(f"{pi:8.2f}") # 3.14 # Currency formatting (Indian Rupees example) amount = 1234567.5 print(f"₹{amount:,.2f}") # ₹1,234,567.50 ``` ✅ Beginner tab complete - I can embed a variable in a string with f"{variable}" - I can embed an expression with f"{a + b}" - I can format to 2 decimal places with f"{value:.2f}" - I can add thousands separators with f"{number:,}" Continue to [functools →](/coding/languages/python/standard-library/functools/) ## INTERMEDIATE ## Advanced f-string features f-strings support several powerful features: the `=` debugging specifier (Python 3.8+), nested expressions, conditional logic, and calling methods inline. ```python x = 42 y = 8 # The = specifier (Python 3.8+) — prints "expr=value", great for debugging print(f"{x=}") # x=42 print(f"{x + y=}") # x + y=50 print(f"{x * y = }") # x * y = 336 (spaces preserved) # Conditional expressions inside f-strings score = 75 print(f"Result: {'PASS' if score >= 50 else 'FAIL'}") # Result: PASS # Calling methods name = "priya kumar" print(f"{name.title()}") # Priya Kumar print(f"{name.upper()}") # PRIYA KUMAR # Nested f-strings and dynamic format specs width = 10 value = 3.14159 print(f"{value:{width}.2f}") # width comes from a variable # Accessing dict values and object attributes user = {"name": "Arjun", "age": 28} print(f"{user['name']} is {user['age']}") # Arjun is 28 # Multi-line f-strings report = ( f"Name: {name.title()}\n" f"Score: {score}\n" f"Status: {'PASS' if score >= 50 else 'FAIL'}" ) ``` **Commonly confused:** - Quotes inside f-strings. Before Python 3.12, you couldn't reuse the same quote type inside an f-string's braces. f"{d["key"]}" was a syntax error — you had to use different quotes: f"{d['key']}". Python 3.12 (PEP 701) lifted this restriction, but for compatibility, still alternate quote types. ✅ Intermediate tab complete - I can use f"{x=}" to print both the expression and its value for debugging - I can put a conditional expression inside an f-string - I can call methods inside braces: f"{name.upper()}" - I know to double braces {{ }} for literal brace characters Continue to [functools →](/coding/languages/python/standard-library/functools/) ## EXPERT ## How f-strings are compiled Unlike `str.format()`, which is a runtime method call, f-strings are parsed and compiled by CPython at compile time into a series of string concatenations and `format()` calls. This is why they are faster — there is no runtime parsing of a format string or dictionary lookup of arguments. ```python import dis def with_fstring(name, age): return f"{name} is {age}" # The bytecode uses FORMAT_VALUE and BUILD_STRING — # no method call, no format-string parsing at runtime dis.dis(with_fstring) # LOAD_FAST name # FORMAT_VALUE 0 # LOAD_CONST ' is ' # LOAD_FAST age # FORMAT_VALUE 0 # BUILD_STRING 3 # RETURN_VALUE # PEP 701 (Python 3.12) reimplemented f-string parsing in the # formal grammar (PEG parser), enabling: # - Reusing quote characters: f"{d["key"]}" # - Multi-line expressions and comments inside braces # - Arbitrary nesting depth # - Backslashes inside the expression part # The __format__ protocol: format specifiers are passed to # the value's __format__ method class Temperature: def __init__(self, celsius): self.celsius = celsius def __format__(self, spec): if spec == "f": # fahrenheit return f"{self.celsius * 9/5 + 32:.1f}°F" return f"{self.celsius:.1f}°C" t = Temperature(25) print(f"{t}") # 25.0°C print(f"{t:f}") # 77.0°F — custom format spec ``` The `__format__` protocol is what makes format specifiers extensible: when you write `f"{value:spec}"`, Python calls `type(value).__format__(value, "spec")`. Built-in types implement this for their format mini-languages (the float format spec, the datetime format spec, etc.), and your own classes can too. ✅ Expert tab complete - I know f-strings compile to FORMAT_VALUE/BUILD_STRING bytecode, not a runtime method call - I know f-strings are faster than .format() and %-formatting - I can implement __format__ on my own class to support custom format specifiers Continue to [functools →](/coding/languages/python/standard-library/functools/) ## SOURCES - PEP 498 — Literal String Interpolation (f-strings). peps.python.org/pep-0498/. - PEP 701 — Syntactic formalization of f-strings (Python 3.12). peps.python.org/pep-0701/. - Python Language Reference §2.4.3 — Formatted string literals. docs.python.org/3/reference/lexical_analysis.html#f-strings. - Python Library Reference — Format Specification Mini-Language. docs.python.org/3/library/string.html#formatspec. **Primary source:** PEP 498 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # String Methods in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/string-methods/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *Python strings come with over 40 built-in methods for searching, splitting, transforming, and validating text — the everyday tools of text processing.* ## CANONICAL DEFINITION Python str objects provide a fixed set of methods defined in the Python Language Reference. Because strings are immutable, every method that "modifies" a string actually returns a new string, leaving the original unchanged. ## BEGINNER > The idea A string method is an action you run on text: make it uppercase, find a word, replace a phrase, strip whitespace. Because Python strings are immutable, these methods never change the original — they return a brand-new string. ## The most-used string methods These ten methods cover the vast majority of everyday text work. Each returns a new string (or a value) — the original is never modified. ```python text = " Hello, World! " # Case print(text.upper()) # " HELLO, WORLD! " print(text.lower()) # " hello, world! " print("priya".capitalize())# "Priya" print("priya kumar".title())# "Priya Kumar" # Whitespace removal print(text.strip()) # "Hello, World!" (both ends) print(text.lstrip()) # "Hello, World! " (left only) print(text.rstrip()) # " Hello, World!" (right only) # The original is unchanged — strings are immutable print(text) # " Hello, World! " — untouched # Length is a function, not a method print(len("Python")) # 6 ``` > Strings are immutable `text.upper()` does not change `text` — it returns a new string. If you want to keep the result, assign it: `text = text.upper()`. This trips up beginners who expect the method to modify the variable in place. ## Searching and replacing ```python s = "the quick brown fox" # Membership — use 'in', not a method print("quick" in s) # True # Find position (returns -1 if not found) print(s.find("brown")) # 10 print(s.find("cat")) # -1 (not found — no error) print(s.index("brown")) # 10 (like find, but raises ValueError if missing) # Count occurrences print(s.count("o")) # 2 # Replace (returns a new string) print(s.replace("quick", "slow")) # "the slow brown fox" # Starts / ends with print(s.startswith("the")) # True print(s.endswith("fox")) # True print("file.pdf".endswith((".pdf", ".doc"))) # True — accepts a tuple ``` ✅ Beginner tab complete - I know upper/lower/strip/replace return NEW strings (the original is unchanged) - I can search text with in, find, and index — and know find returns -1 while index raises - I can use startswith/endswith, including with a tuple of options - I know len() is a function, not a method Continue to [f-strings →](/coding/languages/python/f-strings/) ## INTERMEDIATE ## Splitting and joining Splitting turns a string into a list; joining turns a list back into a string. These two operations are the backbone of text parsing and assembly. ```python # split — string to list csv = "Priya,30,Mumbai" print(csv.split(",")) # ['Priya', '30', 'Mumbai'] # split with no argument splits on ANY whitespace, collapsing runs print("a b c".split()) # ['a', 'b', 'c'] # maxsplit limits the number of splits print("a-b-c-d".split("-", 1)) # ['a', 'b-c-d'] # splitlines — split on line boundaries print("line1\nline2\nline3".splitlines()) # ['line1', 'line2', 'line3'] # join — list to string (called on the SEPARATOR) parts = ['Priya', '30', 'Mumbai'] print(",".join(parts)) # "Priya,30,Mumbai" print(" ".join(["Hello", "World"])) # "Hello World" # join only works on strings — convert numbers first nums = [1, 2, 3] print("-".join(str(n) for n in nums)) # "1-2-3" # partition — split into exactly 3 parts (before, sep, after) print("key=value".partition("=")) # ('key', '=', 'value') ``` > join is called on the separator, not the list It reads backwards at first: `",".join(parts)`, not `parts.join(",")`. The separator string owns the method because join must work on any iterable of strings — so the method lives on str. ## Validation and alignment methods ```python # Validation — all return True/False print("12345".isdigit()) # True print("abc123".isalnum()) # True print("Hello".isalpha()) # True print(" ".isspace()) # True print("HELLO".isupper()) # True print("hello world".islower()) # True # Alignment / padding print("5".zfill(3)) # "005" (zero-fill to width 3) print("hi".ljust(10, ".")) # "hi........" print("hi".rjust(10, ".")) # "........hi" print("hi".center(10, "-")) # "----hi----" # Case-insensitive comparison — use casefold, not lower print("Straße".casefold() == "strasse".casefold()) # True (handles ß) # Remove a prefix/suffix (Python 3.9+) print("test_file.py".removesuffix(".py")) # "test_file" print("get_name".removeprefix("get_")) # "name" ``` **Commonly confused:** - find vs index. Both locate a substring. find returns -1 if not found; index raises ValueError. Use find when "not found" is acceptable; use index when absence is a bug you want to catch. - strip does not remove from the middle. "xxhelloxx".strip("x") gives "hello", but "xhxexllo".strip("x") gives "hxexllo" — it only removes from the ends, stopping at the first non-matching character. ✅ Intermediate tab complete - I can split a string into a list and join a list back into a string - I know join is called on the separator: ",".join(parts) - I can convert numbers to strings before joining - I know removeprefix/removesuffix (3.9+) for clean prefix/suffix removal Continue to [f-strings →](/coding/languages/python/f-strings/) ## EXPERT ## Immutability, interning, and the str C implementation CPython implements `str` as a variable-length object holding a sequence of Unicode code points. Since PEP 393 (Python 3.3), CPython uses a flexible internal representation that stores each string using the smallest unit that fits all its characters — 1 byte (Latin-1), 2 bytes (UCS-2), or 4 bytes (UCS-4). A pure-ASCII string uses one byte per character; adding a single emoji forces the whole string to 4 bytes per character. ```python import sys # PEP 393 — internal size depends on widest character print(sys.getsizeof("abc")) # 52 bytes (1 byte/char + header) print(sys.getsizeof("ab," + "\u00e9")) # wider — 2 bytes/char print(sys.getsizeof("ab" + "\U0001F600")) # widest — 4 bytes/char # String interning — identical literals may share one object a = "hello" b = "hello" print(a is b) # True — compile-time interning of identifiers # But computed strings are not auto-interned c = "hel" + "lo" # constant-folded → interned d = "".join(["h","e","l","l","o"]) # computed → NOT interned print(c is d) # False import sys as _s print(_s.intern(d) is c) # True — manual interning forces sharing ``` Because strings are immutable, building a large string by repeated concatenation in a loop is O(n²) — each `+=` creates a new string and copies all previous characters. The correct pattern is to collect parts in a list and `"".join()` them once at the end, which is O(n). This is one of the most important performance idioms in Python text processing. ✅ Expert tab complete - I know why concatenating in a loop with += is O(n²) - I know to build strings with a list + join for O(n) performance - I understand PEP 393 flexible string representation at a high level Continue to [f-strings →](/coding/languages/python/f-strings/) ## SOURCES - Python Standard Library — Text Sequence Type str. docs.python.org/3/library/stdtypes.html#string-methods. - Python Standard Library — str.removeprefix/removesuffix (PEP 616, Python 3.9). docs.python.org/3/library/stdtypes.html. - PEP 393 — Flexible String Representation. peps.python.org/pep-0393/. - Python Language Reference §6.10 — Membership test operations (in). docs.python.org/3/reference/expressions.html. **Primary source:** Python stdtypes §str *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Control Flow and Loops in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/control-flow/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Python's for loop iterates over anything iterable. Its match/case handles structured data. Here's the complete control flow toolkit.* ## CANONICAL DEFINITION Python control flow uses indentation to define blocks — if/elif/else for branching, for to iterate any iterable, while for condition-based loops, and match/case (Python 3.10+) for structural pattern matching. ## BEGINNER > The analogy Control flow is the signposts on a road — they tell your code which way to go, how many times to go around the roundabout, and when to take the exit. Without them, code runs straight from top to bottom and never makes a decision. ## if, elif, else ```python score = 85 if score >= 90: grade = "A" elif score >= 80: grade = "B" elif score >= 70: grade = "C" else: grade = "F" print(grade) # B # Ternary expression (conditional expression) status = "pass" if score >= 50 else "fail" # Truthy and falsy values # Falsy: False, None, 0, 0.0, "", [], {}, set() # Everything else is truthy name = "" if not name: print("Name is empty") # prints this # Chained comparisons — Python supports them natively x = 5 if 0 < x < 10: print("x is between 0 and 10") # prints # match/case — structural pattern matching (Python 3.10+) command = "quit" match command: case "quit" | "exit" | "q": print("Goodbye!") case "help": print("Available commands: quit, help") case _: print(f"Unknown command: {command}") ``` ## for loops Python's `for` loop iterates over any *iterable* — a list, string, range, dictionary, file, generator, or any object that implements `__iter__`. There is no C-style `for (i=0; i 50: {n}") break # continue — skip to next iteration for n in range(10): if n % 2 == 0: continue # skip evens print(n) # prints 1,3,5,7,9 # else on loops — runs if loop completed without break for n in range(2, 10): for divisor in range(2, n): if n % divisor == 0: break else: print(f"{n} is prime") # only runs if inner loop didn't break # pass — do nothing (syntactic placeholder) def not_yet_implemented(): pass # valid empty function body ``` ✅ Beginner tab complete — check your understanding - I can write an if/elif/else block that handles at least 3 conditions - I can write a for loop that iterates over a list and does something with each item - I can write a while loop that runs until a condition becomes False - I know what break and continue do inside a loop - I understand what "truthy" and "falsy" mean in Python Go to [Comprehensions →](/coding/languages/python/comprehensions/) — the elegant shorthand for the for loops you just learned. ## INTERMEDIATE ## Iterators and the iteration protocol A `for` loop in Python is syntactic sugar for the iterator protocol: Python calls `iter(obj)` to get an iterator, then repeatedly calls `next(iterator)` until `StopIteration` is raised. Any object implementing `__iter__` and `__next__` is an iterator. This protocol is what makes generators, file objects, ranges, and custom classes all work with `for`. ```python # Under the hood of "for x in lst:" lst = [1, 2, 3] iterator = iter(lst) # calls lst.__iter__() print(next(iterator)) # 1 — calls iterator.__next__() print(next(iterator)) # 2 print(next(iterator)) # 3 # next(iterator) # StopIteration # Custom iterable class class Countdown: def __init__(self, start): self.start = start def __iter__(self): return self # this object IS the iterator def __next__(self): if self.start < 0: raise StopIteration val = self.start self.start -= 1 return val for n in Countdown(3): print(n) # 3, 2, 1, 0 ``` ## Comprehension forms Python offers four comprehension forms — concise syntax for creating collections from iterables. ```python # List comprehension squares = [x**2 for x in range(10)] # Set comprehension unique_lengths = {len(word) for word in ["hello", "hi", "hey"]} # Dict comprehension word_lengths = {word: len(word) for word in ["hello", "world"]} # Generator expression (lazy — no [] or {}) sum_squares = sum(x**2 for x in range(1000)) # O(1) memory # Conditions evens_squared = [x**2 for x in range(20) if x % 2 == 0] # Nested (be careful — hard to read beyond 2 levels) flat = [item for row in [[1,2],[3,4],[5,6]] for item in row] # [1, 2, 3, 4, 5, 6] # Walrus operator := (Python 3.8+) # Assign and use in one expression numbers = [1, 2, 3, 4, 5, 6, 7, 8] filtered = [y for x in numbers if (y := x**2) > 10] # [16, 25, 36, 49, 64] ``` ## Structural pattern matching (Python 3.10+) The `match` statement (PEP 634) is much more powerful than a switch/case. It matches on the structure of an object — checking types, destructuring tuples and lists, matching class attributes, and binding matched parts to names. ```python # Match on value def http_error(status): match status: case 400: return "Bad request" case 404: return "Not found" case 418: return "I'm a teapot" case 500 | 502 | 503: return "Server error" case _: return f"Unknown: {status}" # Destructure sequences def process_point(point): match point: case (0, 0): return "origin" case (x, 0): return f"x-axis at {x}" case (0, y): return f"y-axis at {y}" case (x, y): return f"({x}, {y})" # Match class instances class Point: def __init__(self, x, y): self.x = x; self.y = y def classify(p): match p: case Point(x=0, y=0): return "origin" case Point(x=x, y=0): return f"x-axis: {x}" case Point(x=0, y=y): return f"y-axis: {y}" case Point(x=x, y=y) if x == y: return f"diagonal: {x}" case Point(): return "some point" ``` **Commonly confused:** - Python has no do-while loop. The equivalent is a while True: loop with a break at the end: while True: body; if condition: break. This is idiomatic Python. - for/else does not mean "if the loop found nothing." The else clause runs when the loop completes without hitting a break. If the iterable was empty, the else still runs. It's best read as "no break occurred." How this connects Requires first [Variables & Types](/coding/languages/python/variables-and-types/) Enables [Functions & Scope](/coding/languages/python/functions-and-scope/)[OOP](/coding/languages/python/oop/) Primary source [Python Reference — Compound statements](https://docs.python.org/3/reference/compound_stmts.html) ✅ Intermediate tab complete — check your understanding - I can explain what __iter__ and __next__ do and how they power the for loop - I can write a match/case block for at least one real use case - I know Python evaluates "and" and "or" lazily (short-circuit evaluation) Continue to [Comprehensions →](/coding/languages/python/comprehensions/) ## EXPERT ## How Python evaluates boolean expressions Python's boolean operators `and`, `or`, `not` use short-circuit evaluation and return values, not necessarily booleans. `x and y`: if `x` is falsy, return `x`; otherwise return `y`. `x or y`: if `x` is truthy, return `x`; otherwise return `y`. This is specified in Python Language Reference §6.12: "These operators are evaluated left to right. Short-circuit evaluation is performed." This enables idiomatic patterns like `name = name or "Anonymous"` and `value = obj and obj.method()`. ## PEP 634 — Structural Pattern Matching Python 3.10 added structural pattern matching (PEP 634, accepted 2021). Unlike a switch statement (value comparison), pattern matching destructures objects. The `match` statement evaluates the subject once, then tries each `case` pattern in order. Patterns can be: literal (value equality), capture (binds a name), wildcard (`_`), class (checks type + attributes), sequence (matches list/tuple structure), mapping (matches dict keys), OR pattern (`|`), AS pattern (bind after match), guard (`if` condition). The implementation uses structural subtyping, not isinstance checks — the `__match_args__` class attribute controls positional pattern matching. ✅ Expert tab complete - I know the difference between a capture pattern, a literal pattern, and a class pattern - I understand why match/case is not the same as a switch statement Continue to [Comprehensions →](/coding/languages/python/comprehensions/) ## SOURCES - Python Language Reference §6 — Expressions. docs.python.org/3/reference/expressions.html. - Python Language Reference §8 — Compound statements (for, while, if, match). docs.python.org/3/reference/compound_stmts.html. - PEP 634 — Structural Pattern Matching. peps.python.org/pep-0634/. - PEP 572 — Assignment Expressions (walrus operator :=). peps.python.org/pep-0572/. **Primary source:** docs.python.org/3/reference/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Structural Pattern Matching (match/case) ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/match-case/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The match statement (Python 3.10+) matches a value against patterns that describe the structure of data — far more powerful than a simple switch on a single value.* ## CANONICAL DEFINITION Structural pattern matching, introduced by PEP 634 in Python 3.10, is a match statement that compares a subject value against a series of case patterns. Patterns can match literals, capture values, destructure sequences and mappings, and match class instances by attribute. ## BEGINNER > The idea A match statement looks at a value and checks it against several patterns, running the code for the first one that fits. It is like a smarter if/elif chain that can also look *inside* the data — matching not just values but their shape. ## match basics The simplest form of `match` compares a value against literal patterns — close to a switch statement in other languages, but with one crucial extra: the wildcard `_` case. ```python def describe_status(code): match code: case 200: return "OK" case 404: return "Not Found" case 500: return "Server Error" case _: # wildcard — matches anything (the "default") return "Unknown" print(describe_status(200)) # OK print(describe_status(999)) # Unknown # Combine patterns with | (OR) def category(code): match code: case 200 | 201 | 204: return "Success" case 400 | 401 | 403 | 404: return "Client Error" case 500 | 502 | 503: return "Server Error" case _: return "Other" print(category(404)) # Client Error ``` > match needs Python 3.10 or newer Structural pattern matching was added in Python 3.10 (October 2021). On older versions, `match` is a syntax error. Check with `python --version`. `match` and `case` are "soft keywords" — they are only keywords inside a match statement, so existing code using them as variable names still works. ## Capturing and destructuring The real power of `match` appears when patterns capture values and destructure sequences — pulling apart the structure of the data in the pattern itself. ```python def handle_command(command): match command.split(): case ["quit"]: return "Quitting" case ["go", direction]: # capture the second word return f"Moving {direction}" case ["pick", "up", *items]: # capture the rest as a list return f"Picking up: {items}" case _: return "Unknown command" print(handle_command("go north")) # Moving north print(handle_command("pick up sword shield")) # Picking up: ['sword', 'shield'] # Destructuring a coordinate tuple def quadrant(point): match point: case (0, 0): return "origin" case (x, 0): # captures x, requires y == 0 return f"on x-axis at {x}" case (0, y): return f"on y-axis at {y}" case (x, y): return f"point at ({x}, {y})" print(quadrant((0, 0))) # origin print(quadrant((5, 0))) # on x-axis at 5 print(quadrant((3, 4))) # point at (3, 4) ``` ✅ Beginner tab complete - I can write a match with literal cases and a _ wildcard default - I can combine patterns with | (OR) - I can destructure a sequence: case [x, y] or case ["go", direction] - I know a bare name in a case captures, it does not compare Continue to [Comprehensions →](/coding/languages/python/comprehensions/) ## INTERMEDIATE ## Class patterns Patterns can match instances of a class and simultaneously extract their attributes — exceptionally useful for processing structured objects without a chain of `isinstance` checks. ```python from dataclasses import dataclass @dataclass class Circle: radius: float @dataclass class Rectangle: width: float height: float @dataclass class Triangle: base: float height: float def area(shape): match shape: case Circle(radius=r): return 3.14159 * r * r case Rectangle(width=w, height=h): return w * h case Triangle(base=b, height=h): return 0.5 * b * h case _: raise ValueError("Unknown shape") print(area(Circle(5))) # 78.53... print(area(Rectangle(4, 6))) # 24 print(area(Triangle(3, 8))) # 12.0 # Positional class patterns using __match_args__ @dataclass class Point: x: int y: int # dataclasses set __match_args__ = ('x', 'y') automatically def origin_check(p): match p: case Point(0, 0): # positional — uses __match_args__ return "at origin" case Point(x, y): return f"at ({x}, {y})" print(origin_check(Point(0, 0))) # at origin ``` ## Guards and mapping patterns ```python # Guards — an if condition attached to a pattern def classify(point): match point: case (x, y) if x == y: # guard return "on diagonal" case (x, y) if x > 0 and y > 0: return "first quadrant" case (x, y): return "elsewhere" print(classify((3, 3))) # on diagonal print(classify((2, 5))) # first quadrant # Mapping patterns — match dict structure (e.g. JSON) def handle_event(event): match event: case {"type": "click", "x": x, "y": y}: return f"Click at ({x}, {y})" case {"type": "key", "key": key}: return f"Key pressed: {key}" case {"type": event_type}: return f"Unknown event: {event_type}" case _: return "Malformed event" print(handle_event({"type": "click", "x": 10, "y": 20})) # Click at (10, 20) print(handle_event({"type": "key", "key": "Enter"})) # Key pressed: Enter # Capture with 'as' def describe(obj): match obj: case [Point(0, 0) as p]: return f"single origin point: {p}" case [*points]: return f"{len(points)} points" ``` **Commonly confused:** - A bare name in a case captures — it does not compare. case x: matches anything and binds it to x. To compare against an existing constant, use a dotted name (case Color.RED:) or a literal. case RED: where RED is a variable will NOT compare to RED — it captures into a new variable named RED. ✅ Intermediate tab complete - I can match a class instance and extract its attributes in one pattern - I can add a guard: case (x, y) if x == y - I can match dict structure with mapping patterns - I know mapping patterns match a SUBSET of keys (extra keys are ignored) Continue to [Comprehensions →](/coding/languages/python/comprehensions/) ## EXPERT ## Match semantics, irrefutability, and __match_args__ PEP 634 defines the formal semantics. Patterns are tried top to bottom; the first matching case runs and no others (no fall-through, unlike C's switch). A pattern can be *irrefutable* — guaranteed to match — such as a bare capture pattern (`case x:`) or the wildcard (`case _:`). An irrefutable pattern must be last, because any case after it is unreachable; the compiler does not enforce this but the pattern can never trigger. ```python # __match_args__ controls positional class patterns class Point: __match_args__ = ("x", "y") # positional order for matching def __init__(self, x, y): self.x = x self.y = y # Now Point(0, 0) in a pattern checks x==0, y==0 positionally match Point(1, 2): case Point(1, y): # uses __match_args__ to map position 0 -> x print(f"x is 1, y is {y}") # x is 1, y is 2 # Sequence patterns work on any Sequence EXCEPT str/bytes/bytearray # (deliberately — so "abc" is not treated as ['a','b','c']) match [1, 2, 3]: case [first, *rest]: print(first, rest) # 1 [2, 3] # The compiler emits MATCH_SEQUENCE, MATCH_MAPPING, MATCH_CLASS opcodes import dis def f(x): match x: case [a, b]: return a + b case _: return 0 dis.dis(f) # shows MATCH_SEQUENCE, GET_LEN, etc. ``` Sequence patterns deliberately exclude `str`, `bytes`, and `bytearray` — matching `"abc"` against `case [a, b, c]` fails rather than destructuring into characters, because treating strings as character sequences in pattern matching is almost always a bug. Mapping patterns match a subset of keys: `case {"type": t}` matches any dict containing a "type" key, ignoring other keys — unlike sequence patterns which match the full length unless a star pattern is used. ✅ Expert tab complete - I know patterns are tried top-to-bottom with no fall-through - I know __match_args__ defines the positional order for class patterns - I know str/bytes are deliberately excluded from sequence patterns Continue to [Comprehensions →](/coding/languages/python/comprehensions/) ## SOURCES - Python Language Reference §8.6 — The match statement. docs.python.org/3/reference/compound_stmts.html#the-match-statement. - PEP 634 — Structural Pattern Matching: Specification. peps.python.org/pep-0634/. - PEP 636 — Structural Pattern Matching: Tutorial. peps.python.org/pep-0636/. - PEP 635 — Structural Pattern Matching: Motivation and Rationale. peps.python.org/pep-0635/. **Primary source:** Python Language Reference §8.6 (PEP 634) *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Comprehensions in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/comprehensions/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-16 *List, dict, set, and generator comprehensions — Python's concise syntax for building collections from iterables in a single readable expression.* ## CANONICAL DEFINITION A comprehension is a compact syntax for constructing a list, set, dict, or generator from one or more iterables, optionally filtering elements and transforming each one within a single expression, as defined in the Python Language Reference under displays for lists, sets, and dictionaries. ## BEGINNER > The idea A comprehension lets you build a new list (or dict, or set) from an existing sequence in one line, instead of writing a `for` loop with `.append()` on every iteration. It reads almost like English: "give me *x squared*, for each *x* in this range, where *x is even*." ## List comprehensions The list comprehension is the most common form. The syntax is `[expression for item in iterable]`, with an optional `if` condition to filter. According to the Python Tutorial, a list comprehension consists of brackets containing an expression followed by a `for` clause, then zero or more `for` or `if` clauses. ```python # The loop version squares = [] for x in range(10): squares.append(x * x) # The comprehension version — same result, one line squares = [x * x for x in range(10)] # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] # With a filter — only even numbers evens = [x for x in range(20) if x % 2 == 0] # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] # Transform and filter together names = ["priya", "arjun", "rohit", "kavya"] capitalised = [name.title() for name in names if len(name) > 4] # ['Priya', 'Arjun', 'Rohit', 'Kavya'] # Apply a function to each element words = [" hello ", " world "] cleaned = [w.strip() for w in words] # ['hello', 'world'] ``` > Read it right to left, then left To understand `[x * x for x in range(10)]`, first read the `for` part (*for each x in range 10*), then the expression at the front (*compute x times x*). The result of the front expression is what gets collected into the new list. ## Dict and set comprehensions The same pattern works for dictionaries and sets — just change the brackets. Curly braces with a `key: value` expression build a dict; curly braces with a single expression build a set. ```python # Dict comprehension — {key: value for ...} names = ["Priya", "Arjun", "Rohit"] name_lengths = {name: len(name) for name in names} # {'Priya': 5, 'Arjun': 5, 'Rohit': 5} # Build a lookup table squares = {n: n * n for n in range(1, 6)} # {1: 1, 2: 4, 3: 9, 4: 16, 5: 25} # Invert a dictionary (swap keys and values) original = {"a": 1, "b": 2, "c": 3} inverted = {value: key for key, value in original.items()} # {1: 'a', 2: 'b', 3: 'c'} # Set comprehension — {expression for ...}, no key:value numbers = [1, 2, 2, 3, 3, 3, 4] unique_squares = {n * n for n in numbers} # {16, 1, 4, 9} — duplicates removed automatically ``` ✅ Beginner tab complete — check your understanding - I can write a list comprehension that transforms every item in a list - I can write a list comprehension with an if condition to filter items - I can write a dict comprehension to build a key-value mapping from a list - I can write a set comprehension to get unique values Continue to [Functions & Scope →](/coding/languages/python/functions-and-scope/) ## INTERMEDIATE ## Generator expressions A generator expression looks exactly like a list comprehension but uses parentheses instead of square brackets. The crucial difference: it does not build the whole collection in memory. Instead it produces items one at a time, on demand. The Python Tutorial notes that generator expressions are more memory-efficient than equivalent list comprehensions when you only need to iterate once. ```python # List comprehension — builds the entire list in memory sum_squares = sum([x * x for x in range(1_000_000)]) # Generator expression — produces values one at a time, no list built sum_squares = sum(x * x for x in range(1_000_000)) # When a generator expression is the sole argument to a function, # the parentheses can be omitted. # A generator object is lazy — nothing computed until iterated gen = (x * x for x in range(5)) print(gen) # print(next(gen)) # 0 print(next(gen)) # 1 print(list(gen)) # [4, 9, 16] — consumes the rest # Generators are single-use — once exhausted, they yield nothing gen = (x for x in range(3)) print(list(gen)) # [0, 1, 2] print(list(gen)) # [] — already consumed ``` > When to use which Use a **list comprehension** when you need the actual list — to index into it, loop over it multiple times, or check its length. Use a **generator expression** when you only iterate once and the sequence is large (or infinite), to avoid holding everything in memory at once. ## Nested comprehensions and multiple clauses A comprehension can contain multiple `for` clauses, which behave like nested loops. The leftmost `for` is the outermost loop. You can also nest a comprehension inside another comprehension's expression. ```python # Multiple for clauses — flattening a 2D list matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] flat = [num for row in matrix for num in row] # [1, 2, 3, 4, 5, 6, 7, 8, 9] # Read left to right: for each row, for each num in row # Equivalent nested loop (for comparison) flat = [] for row in matrix: for num in row: flat.append(num) # Nested comprehension — transpose a matrix matrix = [[1, 2, 3], [4, 5, 6]] transposed = [[row[i] for row in matrix] for i in range(3)] # [[1, 4], [2, 5], [3, 6]] # Cartesian product with a filter pairs = [(x, y) for x in range(3) for y in range(3) if x != y] # [(0, 1), (0, 2), (1, 0), (1, 2), (2, 0), (2, 1)] ``` **Commonly confused:** - Brackets decide the type. [...] builds a list, {...} with key: value builds a dict, {...} with a single value builds a set, and (...) builds a generator — not a tuple. There is no "tuple comprehension"; (x for x in ...) is a generator expression. To build a tuple, wrap a generator: tuple(x for x in ...). ✅ Intermediate tab complete — check your understanding - I know the difference between [x for x in ...] (list) and (x for x in ...) (generator) - I can use sum(x*x for x in range(n)) without building an intermediate list - I can write a nested comprehension to flatten a 2D list - I know that a generator is single-use — once exhausted, it yields nothing Continue to [Functions & Scope →](/coding/languages/python/functions-and-scope/) ## EXPERT ## Comprehension scope (PEP 572 and the comprehension namespace) Since Python 3, comprehensions and generator expressions have their own enclosing scope. The iteration variable does not leak into the surrounding namespace — after `[x for x in range(10)]`, the name `x` is not defined in the outer scope (in Python 2 it did leak; this was deliberately changed in Python 3). The Python Language Reference states that the comprehension is executed in a separate implicitly nested scope, which ensures that names assigned to in the target list do not "leak" into the enclosing scope. ```python # The loop variable does not leak (Python 3) result = [x * 2 for x in range(5)] # print(x) -> NameError: name 'x' is not defined # The iterable expression of the LEFTMOST for is evaluated # in the enclosing scope, the rest in the comprehension scope. # This matters when the iterable raises: data = [1, 2, 3] squares = [y * y for y in data] # 'data' resolved in enclosing scope # Walrus operator (:=, PEP 572) CAN bind to the enclosing scope # from inside a comprehension — useful for capturing a computed value values = [10, 20, 30, 40] filtered = [smoothed for v in values if (smoothed := v / 10) > 1] # filtered == [2.0, 3.0, 4.0]; 'smoothed' leaks to enclosing scope print(smoothed) # 4.0 — walrus binding survives ``` ## Performance and the LIST_APPEND bytecode A list comprehension is generally faster than an equivalent `for` loop with `.append()`, because CPython compiles it to use a specialised `LIST_APPEND` opcode rather than looking up and calling the `append` method on each iteration. The method-lookup overhead is eliminated. You can confirm this difference with the `dis` module, which shows the comprehension's bytecode is compiled into a nested code object. ```python import dis # A comprehension compiles to its own code object def make_squares(): return [x * x for x in range(10)] dis.dis(make_squares) # You will see LIST_APPEND in the nested code object, # instead of LOAD_METHOD/CALL for .append() # Generator expressions compile to a generator code object # and use YIELD_VALUE — producing items lazily. ``` > Readability is the real limit The Python documentation and PEP 8 favour clarity. A comprehension that spans multiple `for` and `if` clauses can become harder to read than the explicit loop. If a comprehension needs more than two clauses or a complex expression, an ordinary loop is often the better choice — the goal is concise *and* readable, not merely short. ✅ Expert tab complete - I know that the loop variable in a comprehension does NOT leak into the enclosing scope in Python 3 - I know that := (walrus operator) DOES bind to the enclosing scope from inside a comprehension - I understand why LIST_APPEND makes comprehensions faster than a loop with .append() Continue to [Functions & Scope →](/coding/languages/python/functions-and-scope/) ## SOURCES - Python Language Reference §6.2.4 — Displays for lists, sets and dictionaries. docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries. - Python Tutorial §5.1.3 — List Comprehensions. docs.python.org/3/tutorial/datastructures.html#list-comprehensions. - Python Tutorial §5.5 — Dictionaries; §6 — Generator expressions. docs.python.org/3/tutorial/. - PEP 572 — Assignment Expressions (the walrus operator). peps.python.org/pep-0572/. - Python Language Reference §4.2.3 — note on comprehension scope. docs.python.org/3/reference/executionmodel.html. **Primary source:** Python Language Reference §6.2.4 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Iterators and Iterables in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/iterators-iterables/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *An iterable is anything you can loop over; an iterator is the object that does the actual stepping. Understanding the distinction explains how every for loop in Python works.* ## CANONICAL DEFINITION An iterable is any object implementing __iter__, which returns an iterator. An iterator is any object implementing __next__ (returning the next value or raising StopIteration) and __iter__ (returning itself). The Python Language Reference defines the iterator protocol as the mechanism underlying all for loops and comprehensions. ## BEGINNER > The idea An *iterable* is anything you can put after `in` in a for loop — a list, a string, a dict. An *iterator* is the helper Python creates to walk through it one item at a time, remembering where it is. Every for loop quietly turns an iterable into an iterator. ## Iterable vs iterator — the distinction These two words sound alike but mean different things. An iterable can produce an iterator; an iterator produces values. A list is iterable but is not itself an iterator — you can loop over it many times. ```python nums = [1, 2, 3] # a LIST — iterable, but not an iterator # iter() turns an iterable into an iterator it = iter(nums) # now 'it' is an iterator print(next(it)) # 1 print(next(it)) # 2 print(next(it)) # 3 # print(next(it)) # StopIteration — exhausted # A for loop does exactly this for you, automatically: for n in nums: # Python calls iter(nums), then next() repeatedly print(n) # ...and stops cleanly at StopIteration # An iterable can be looped MANY times (fresh iterator each time) for n in nums: pass for n in nums: pass # works — list is reusable # An iterator is ONE-SHOT — once exhausted, it's done it = iter(nums) print(list(it)) # [1, 2, 3] print(list(it)) # [] — already exhausted, nothing left ``` > Iterators are single-use A list can be looped over again and again. An iterator (including a generator) can be walked only once — after it raises StopIteration, it stays exhausted. If you need to iterate twice, either keep the original iterable or convert to a list first. ✅ Beginner tab complete - I know an iterable produces an iterator via iter() - I know an iterator produces values via next() until StopIteration - I know a for loop calls iter() then next() repeatedly - I know iterators are single-use but iterables (like lists) are reusable Continue to [Functions & Scope →](/coding/languages/python/functions-and-scope/) ## INTERMEDIATE ## The iterator protocol The protocol is two methods. An **iterable** implements `__iter__()` returning an iterator. An **iterator** implements `__next__()` returning the next value (or raising `StopIteration`) and `__iter__()` returning itself. ```python # What a for loop REALLY does under the hood: nums = [10, 20, 30] iterator = iter(nums) # 1. get an iterator (calls __iter__) while True: try: item = next(iterator) # 2. get next value (calls __next__) except StopIteration: # 3. stop when exhausted break print(item) # 4. run the loop body # An iterator must implement BOTH __next__ and __iter__ (returning self) class CountUp: def __init__(self, limit): self.current = 0 self.limit = limit def __iter__(self): return self # an iterator returns itself def __next__(self): if self.current >= self.limit: raise StopIteration self.current += 1 return self.current for n in CountUp(3): print(n) # 1, 2, 3 ``` ## Iterable vs iterator as separate objects The cleaner design separates the *iterable* (reusable) from the *iterator* (one-shot), so the collection can be looped multiple times — each loop gets a fresh iterator. This is how built-in containers work. ```python class Deck: """An ITERABLE — reusable, produces a fresh iterator each time.""" def __init__(self): self.cards = ["A", "K", "Q", "J"] def __iter__(self): return DeckIterator(self.cards) # NEW iterator each call class DeckIterator: """An ITERATOR — one-shot, tracks position.""" def __init__(self, cards): self.cards = cards self.index = 0 def __iter__(self): return self def __next__(self): if self.index >= len(self.cards): raise StopIteration card = self.cards[self.index] self.index += 1 return card deck = Deck() print(list(deck)) # ['A','K','Q','J'] print(list(deck)) # ['A','K','Q','J'] — reusable! fresh iterator each time # In practice, a generator is the easy way to write __iter__: class Deck2: def __init__(self): self.cards = ["A", "K", "Q", "J"] def __iter__(self): yield from self.cards # generator = iterator, far less code ``` **Commonly confused:** - Iterable can be looped many times; iterator only once. A list is an iterable — loop it repeatedly. The result of iter(list), or any generator, is an iterator — exhausted after one pass. This is the most important practical distinction. ✅ Intermediate tab complete - I can implement __iter__ and __next__ for a custom iterator - I know an iterator’s __iter__ must return self - I can separate a reusable iterable from a one-shot iterator - I know using a generator in __iter__ is the easy approach Continue to [Functions & Scope →](/coding/languages/python/functions-and-scope/) ## EXPERT ## Lazy evaluation, infinite iterators, and the two-argument iter() Iterators are the foundation of lazy evaluation in Python — values are produced only when requested, so an iterator can represent an infinite or arbitrarily large sequence in constant memory. Generators are the most common iterators, but the standard library's `itertools` module provides composable iterator building blocks, and the built-in `iter()` has a lesser-known two-argument form for sentinel-based iteration. ```python import itertools # Infinite iterator — only computes what you consume counter = itertools.count(start=1, step=2) # 1, 3, 5, 7, ... forever print(list(itertools.islice(counter, 5))) # [1, 3, 5, 7, 9] # Lazy pipeline — no intermediate lists, constant memory squares = map(lambda x: x*x, itertools.count(1)) first_5_big = itertools.takewhile(lambda x: x < 100, squares) print(list(first_5_big)) # [1, 4, 9, 16, 25, 36, 49, 64, 81] # The two-argument iter(callable, sentinel) — call until sentinel returned import io stream = io.StringIO("line1\nline2\nSTOP\nline4\n") # Read lines until "STOP\n" is returned for line in iter(stream.readline, "STOP\n"): print(line.strip()) # line1, line2 (stops at STOP) # A common real use: read fixed-size chunks until empty # with open("file.bin", "rb") as f: # for chunk in iter(lambda: f.read(4096), b""): # process(chunk) # itertools.tee — split one iterator into independent copies it = iter([1, 2, 3, 4]) a, b = itertools.tee(it, 2) print(list(a)) # [1, 2, 3, 4] print(list(b)) # [1, 2, 3, 4] — independent copy # Check the protocols at runtime from collections.abc import Iterable, Iterator print(isinstance([1,2,3], Iterable)) # True print(isinstance([1,2,3], Iterator)) # False — list is iterable, not iterator print(isinstance(iter([1,2,3]), Iterator)) # True ``` The two-argument `iter(callable, sentinel)` form repeatedly calls the zero-argument callable until it returns the sentinel value, then stops. It is the idiomatic way to consume a stream until an end marker — reading chunks from a file until `b""`, or lines until a terminator — without a `while True / break` loop. Combined with `itertools` and generators, the iterator protocol gives Python a complete, memory-efficient, composable model for processing sequences of any size, including infinite ones. ✅ Expert tab complete - I can build lazy pipelines with itertools.count/islice/takewhile - I can use iter(callable, sentinel) to read until an end marker - I know isinstance checks against Iterable vs Iterator from collections.abc Continue to [Functions & Scope →](/coding/languages/python/functions-and-scope/) ## SOURCES - Python Standard Library — Iterator Types. docs.python.org/3/library/stdtypes.html#iterator-types. - Python Language Reference §6.2.9 & §8.3 — for statement and iterator protocol. docs.python.org/3/reference/compound_stmts.html#the-for-statement. - Python Standard Library — itertools. docs.python.org/3/library/itertools.html. - Python Standard Library — iter() built-in (two-argument form). docs.python.org/3/library/functions.html#iter. **Primary source:** Python stdlib — Iterator Types *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Functions and Scope in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/functions-and-scope/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Functions are first-class objects in Python — they can be passed around, returned, and wrapped. Here's everything about how they work.* ## CANONICAL DEFINITION In Python, a function is a first-class object defined with def — it accepts parameters, can have defaults, returns a value (or None), captures its enclosing scope as a closure, and can be passed to and returned from other functions. ## BEGINNER > The analogy A function is a recipe. You define it once, name it, then call it as many times as you need. You hand it ingredients (arguments), it follows steps, and hands you a result (return value). The beauty: the recipe doesn't change what's in your kitchen unless you explicitly tell it to. ## Defining and calling functions ```python # Define a function with def def greet(name): return f"Hello, {name}!" # Call it print(greet("Priya")) # Hello, Priya! # Multiple parameters def add(a, b): return a + b print(add(3, 4)) # 7 # Default parameters — must come after required params def greet_formal(name, title="Ms"): return f"Hello, {title} {name}." print(greet_formal("Sharma")) # Hello, Ms Sharma. print(greet_formal("Sharma", "Dr")) # Hello, Dr Sharma. # Return multiple values (actually returns a tuple) def min_max(numbers): return min(numbers), max(numbers) lo, hi = min_max([3, 1, 4, 1, 5, 9]) print(lo, hi) # 1 9 # Functions are objects — assign to variables, pass around f = greet print(f("Rahul")) # Hello, Rahul! ``` ## Arguments: positional, keyword, *args, **kwargs ```python # Positional — order matters def describe(name, age, city): return f"{name}, {age}, from {city}" describe("Priya", 28, "Mumbai") # positional describe(city="Mumbai", name="Priya", age=28) # keyword — any order # *args — collect extra positional arguments as a tuple def total(*numbers): return sum(numbers) print(total(1, 2, 3, 4, 5)) # 15 # **kwargs — collect extra keyword arguments as a dict def profile(**info): for key, val in info.items(): print(f" {key}: {val}") profile(name="Priya", city="Mumbai", role="Engineer") # Combining — positional, *args, keyword-only, **kwargs def full_example(required, *args, keyword_only=True, **kwargs): print(required, args, keyword_only, kwargs) full_example("a", "b", "c", keyword_only=False, x=1, y=2) # a ('b', 'c') False {'x': 1, 'y': 2} # Unpack when calling nums = [1, 2, 3] print(add(*nums)) # same as add(1, 2, 3) — wait, add takes 2 args opts = {"a": 10, "b": 20} # If we had: def f(a, b): ... # f(**opts) would call f(a=10, b=20) ``` ## Scope: LEGB Python looks up names in four scopes, in order: **L**ocal (inside the current function), **E**nclosing (outer function, for nested functions), **G**lobal (module level), **B**uilt-in (Python's built-ins like `len`, `print`). If a name is not found in any of these, Python raises `NameError`. ```python x = "global" # global scope def outer(): x = "enclosing" # enclosing scope def inner(): x = "local" # local scope print(x) # local — found first inner() print(x) # enclosing outer() print(x) # global # Modify global from inside a function count = 0 def increment(): global count # declare intent to modify global count += 1 increment() print(count) # 1 # Without global: you just create a local variable def bad_increment(): count = count + 1 # UnboundLocalError! # Python sees assignment -> treats count as local # then tries to read count before assignment ``` ## Lambda functions A lambda is a small, anonymous function — useful for short operations passed to functions like `sorted()` or `map()`. They are limited to a single expression. ```python # Lambda: lambda arguments: expression square = lambda x: x * x print(square(5)) # 25 # Useful with sorted(), map(), filter() names = ["Rahul", "Priya", "Anita", "Zara"] sorted_names = sorted(names, key=lambda n: len(n)) print(sorted_names) # by length numbers = [1, 2, 3, 4, 5, 6] evens = list(filter(lambda n: n % 2 == 0, numbers)) doubled = list(map(lambda n: n * 2, numbers)) # For anything more complex, use a regular function # PEP 8: never assign a lambda to a name directly # Wrong: f = lambda x: x*2 # Right: def f(x): return x * 2 ``` ✅ Beginner tab complete — check your understanding - I can define a function with def, give it parameters, and return a value - I can call a function with both positional and keyword arguments - I know how to set default parameter values - I understand the difference between a parameter (the name) and an argument (the value passed in) Continue to [Object-Oriented Python →](/coding/languages/python/oop/) ## INTERMEDIATE ## Closures A closure is a function that remembers the variables from the enclosing scope in which it was created — even after that enclosing scope has finished executing. In Python, a function becomes a closure when it references a free variable (a variable defined in an enclosing scope but not in its own local scope). ```python def make_counter(start=0): count = start # free variable — captured by inner def increment(): nonlocal count # declare we're modifying enclosing count count += 1 return count return increment counter = make_counter(10) print(counter()) # 11 print(counter()) # 12 # Each call to make_counter creates an independent closure counter2 = make_counter(0) print(counter2()) # 1 print(counter()) # 13 — unaffected # Inspect what a closure has captured print(counter.__code__.co_freevars) # ('count',) print(counter.__closure__[0].cell_contents) # 13 ``` ## Decorators A decorator is a function that takes a function, wraps it with additional behaviour, and returns the wrapped version. The `@decorator` syntax is sugar for `func = decorator(func)`. Decorators are how Python adds logging, caching, authentication, retry logic, and timing without touching the underlying function. ```python import functools import time def timer(func): @functools.wraps(func) # preserve original function metadata def wrapper(*args, **kwargs): start = time.perf_counter() result = func(*args, **kwargs) elapsed = time.perf_counter() - start print(f"{func.__name__} took {elapsed:.4f}s") return result return wrapper @timer def slow_sum(n): return sum(range(n)) slow_sum(1_000_000) # slow_sum took 0.04s # Decorator with arguments — needs an extra wrapper layer def repeat(times): def decorator(func): @functools.wraps(func) def wrapper(*args, **kwargs): for _ in range(times): result = func(*args, **kwargs) return result return wrapper return decorator @repeat(3) def say(msg): print(msg) say("hello") # prints hello 3 times # Built-in decorators class Circle: def __init__(self, radius): self._radius = radius @property def area(self): # access as circle.area, not circle.area() return 3.14159 * self._radius ** 2 @staticmethod def info(): return "A circle has no corners." @classmethod def unit(cls): # cls = the class itself return cls(radius=1) ``` ## Generators A generator function uses `yield` instead of `return`. Calling it returns a generator object — an iterator that produces values lazily, one at a time. The function body is paused at each `yield` and resumed on the next `next()` call. Generators are O(1) memory: they produce values on demand instead of building the entire sequence. ```python # Generator function def count_up(start, stop): current = start while current <= stop: yield current # pause here, return value current += 1 # resume here on next call for n in count_up(1, 5): print(n) # 1 2 3 4 5 # Infinite generator — O(1) memory regardless of how far you go def fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b gen = fibonacci() print([next(gen) for _ in range(10)]) # [0,1,1,2,3,5,8,13,21,34] # Generator expression (like list comprehension but lazy) squares_gen = (x**2 for x in range(1_000_000)) # O(1) memory first_five = [next(squares_gen) for _ in range(5)] # send() — pass a value back into the generator def accumulator(): total = 0 while True: value = yield total # yield current total, receive next value if value is None: break total += value acc = accumulator() next(acc) # prime the generator acc.send(10) # 10 acc.send(20) # 30 print(acc.send(5)) # 35 ``` **Commonly confused:** - nonlocal is not the same as global. global refers to the module-level scope. nonlocal refers to the nearest enclosing function scope — not global. Use nonlocal inside a nested function to modify a variable in an outer (but not global) function. - A generator is not a list. Iterating a generator exhausts it — you can only go through it once. list(gen) consumes the generator entirely. To iterate again, you must create a new generator. Lists can be iterated any number of times. How this connects Requires first [Variables & Types](/coding/languages/python/variables-and-types/) Enables [OOP in Python](/coding/languages/python/oop/)[Error Handling](/coding/languages/python/error-handling/) Primary source [Python Reference — Function definitions](https://docs.python.org/3/reference/compound_stmts.html#function) ✅ Intermediate tab complete — check your understanding - I can explain LEGB: Local → Enclosing → Global → Built-in - I can write a function that accepts any number of positional arguments with *args - I can write a function that accepts any keyword arguments with **kwargs - I can write a lambda for a simple one-expression function - I know that using global inside a function is a code smell Continue to [Object-Oriented Python →](/coding/languages/python/oop/) ## EXPERT ## How Python resolves names: the LEGB rule formally The Python Language Reference §4.2.2 defines the LEGB rule: Python looks up names in the local namespace first, then the namespaces of enclosing functions (from inner to outer), then the module's global namespace, then the built-in namespace. Each function creates its own local namespace at call time. The global namespace is the module's `__dict__`. The built-in namespace is `builtins.__dict__`. The `global` statement (§7.13) marks a name as coming from the global namespace; `nonlocal` (§7.14) marks it as coming from the nearest enclosing non-global scope. ## Function objects and the descriptor protocol Functions in Python are first-class objects — instances of `function`. They have attributes: `__name__`, `__doc__`, `__defaults__`, `__code__`, `__globals__`, `__closure__`. When accessed through a class instance, functions become bound methods via the descriptor protocol (Python Reference §3.3.2.4): `instance.method` calls `function.__get__(instance, type(instance))`, which returns a bound method object that prepends `self`. ## PEP 342 — Coroutines via Enhanced Generators PEP 342 (Python 2.5, 2005) enhanced generators with `send()`, `throw()`, and `close()` methods, turning generators into bidirectional coroutines. This was the foundation for Python's async model: PEP 492 (Python 3.5) built `async`/`await` syntax on top of these enhanced generators. A `async def` function is, at the bytecode level, a specialised kind of generator — `await` is syntactic sugar for `yield from` in a coroutine context. ✅ Expert tab complete - I can write a closure that captures a variable from the enclosing scope - I can write a decorator that wraps a function and preserves its signature with functools.wraps - I can write a generator function with yield and iterate it with next() or a for loop - I understand why decorators are just syntactic sugar for func = decorator(func) Continue to [Object-Oriented Python →](/coding/languages/python/oop/) ## SOURCES - Python Language Reference §4 — Execution model. docs.python.org/3/reference/executionmodel.html. - Python Language Reference §7.13 — The global statement. docs.python.org/3/reference/simple_stmts.html#global. - PEP 342 — Coroutines via Enhanced Generators. peps.python.org/pep-0342/. - PEP 318 — Decorators for Functions and Methods. peps.python.org/pep-0318/. **Primary source:** docs.python.org/3/reference/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Lambda Functions in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/lambda/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *A lambda is a small anonymous function written in a single expression — most useful as a throwaway argument to functions like sorted(), map(), and filter().* ## CANONICAL DEFINITION A lambda expression creates an anonymous function object. Defined in the Python Language Reference, it consists of the keyword lambda, a parameter list, a colon, and a single expression whose value is returned. Lambdas are syntactically restricted to one expression — no statements, assignments, or annotations. ## BEGINNER > The idea A lambda is a tiny function with no name, written in one line. `lambda x: x * 2` is a function that doubles its input. You use lambdas when you need a small function briefly — usually to hand to another function — and naming it with `def` would be overkill. ## Lambda vs def A lambda and a `def` create the same kind of object — a function. The difference is that a lambda is an expression (it can appear inline), has no name, and is limited to a single expression. ```python # These two are equivalent def double(x): return x * 2 double = lambda x: x * 2 # same behaviour, anonymous origin print(double(5)) # 10 in both cases # Lambdas can take multiple arguments add = lambda a, b: a + b print(add(3, 4)) # 7 # ...and default arguments greet = lambda name, greeting="Hello": f"{greeting}, {name}!" print(greet("Priya")) # Hello, Priya! print(greet("Arjun", "Namaste")) # Namaste, Arjun! # The body is a single EXPRESSION whose value is returned automatically # (no 'return' keyword, no statements allowed) square = lambda x: x ** 2 print(square(6)) # 36 ``` > PEP 8: don't assign a lambda to a name If you are giving a lambda a name (`double = lambda x: x*2`), just use `def` instead — it is more readable and gives the function a proper name in tracebacks. The whole point of a lambda is to be *anonymous* and *inline*. The examples above assign names only to show equivalence. ✅ Beginner tab complete - I can write a lambda with one or more arguments - I know a lambda body is a single expression (no statements, no return keyword) - I know lambda and def create the same kind of object - I know not to assign a lambda to a name (use def instead) Continue to [Closures →](/coding/languages/python/closures/) ## INTERMEDIATE ## The main use: sort keys The single most common and most appropriate use of a lambda is as the `key` argument to `sorted()`, `min()`, `max()`, and `list.sort()` — telling them *what* to compare. ```python people = [ {"name": "Priya", "age": 30}, {"name": "Arjun", "age": 25}, {"name": "Rohit", "age": 35}, ] # Sort by age by_age = sorted(people, key=lambda p: p["age"]) print([p["name"] for p in by_age]) # ['Arjun', 'Priya', 'Rohit'] # Sort by name length, then alphabetically (tuple key) words = ["banana", "kiwi", "apple", "fig"] print(sorted(words, key=lambda w: (len(w), w))) # ['fig', 'kiwi', 'apple', 'banana'] # max / min with a key print(max(people, key=lambda p: p["age"])["name"]) # Rohit print(min(words, key=len)) # 'fig' (key=len, no lambda needed) # Reverse sort print(sorted([5, 2, 8, 1], key=lambda x: -x)) # [8, 5, 2, 1] # (or simpler: sorted([5,2,8,1], reverse=True)) ``` ## map, filter, and functools.reduce ```python nums = [1, 2, 3, 4, 5, 6] # map — apply a function to every item doubled = list(map(lambda x: x * 2, nums)) print(doubled) # [2, 4, 6, 8, 10, 12] # filter — keep items where the function returns True evens = list(filter(lambda x: x % 2 == 0, nums)) print(evens) # [2, 4, 6] # reduce — fold a sequence into a single value from functools import reduce product = reduce(lambda acc, x: acc * x, nums) print(product) # 720 (1*2*3*4*5*6) # BUT: a comprehension is usually clearer than map/filter+lambda doubled = [x * 2 for x in nums] # clearer than map evens = [x for x in nums if x % 2 == 0] # clearer than filter # Reach for map/filter mainly when you already HAVE a named function: nums_as_str = list(map(str, nums)) # map with a builtin — clean ``` **Commonly confused:** - Comprehension vs map/filter + lambda. [f(x) for x in xs] is generally more readable than map(lambda x: f(x), xs). Use a comprehension by default; use map/filter when passing an already-named function (e.g. map(str, nums)). ✅ Intermediate tab complete - I can use lambda as a key= argument to sorted/min/max - I can sort by a tuple key for multi-level sorting - I know a comprehension is usually clearer than map/filter + lambda - I can recognise the late-binding closure trap in a loop of lambdas Continue to [Closures →](/coding/languages/python/closures/) ## EXPERT ## Syntactic restrictions and the function object A lambda produces exactly the same type of object as `def` — a `function` instance with `__code__`, `__defaults__`, and `__closure__` attributes. The only differences are syntactic: a lambda's body must be a single expression (no statements, no `return`, no `=` assignment, no annotations), and its `__name__` is always the string `""`, which is why lambdas produce less helpful tracebacks than named functions. ```python f = lambda x, y=10: x + y # Same object type as a def function print(type(f)) # print(f.__name__) # '' — always, hence poor tracebacks print(f.__defaults__) # (10,) print(f.__code__.co_argcount) # 2 # Lambdas capture by reference (closures) — the late-binding trap funcs = [lambda: i for i in range(3)] print([fn() for fn in funcs]) # [2, 2, 2] — all see final i # Fix 1: default argument binds the CURRENT value funcs = [lambda i=i: i for i in range(3)] print([fn() for fn in funcs]) # [0, 1, 2] # Fix 2: functools.partial from functools import partial def identity(i): return i funcs = [partial(identity, i) for i in range(3)] print([fn() for fn in funcs]) # [0, 1, 2] # Conditional expression inside a lambda (allowed — it's an expression) sign = lambda x: "positive" if x > 0 else "negative" if x < 0 else "zero" print(sign(-5)) # negative # What you CANNOT do in a lambda (all SyntaxError): # lambda x: x = 5 # no assignment # lambda x: print(x); x # no statement sequences # lambda x: return x # no return keyword ``` Because the body is restricted to one expression, anything requiring a statement — a loop, a `try`, multiple steps, or an early `return` — cannot be a lambda and must be a `def`. The conditional expression (`a if cond else b`) is permitted because it is an expression, which lets lambdas express simple branching. For anything beyond a single clear expression, a named function is both required and more readable. ✅ Expert tab complete - I know a lambda is the same object type as a def function - I know lambda __name__ is always "", giving poor tracebacks - I can fix late binding with a default argument: lambda i=i: i - I know a conditional expression is allowed in a lambda but statements are not Continue to [Closures →](/coding/languages/python/closures/) ## SOURCES - Python Language Reference §6.14 — Lambdas. docs.python.org/3/reference/expressions.html#lambda. - Python Standard Library — sorted(), map(), filter(). docs.python.org/3/library/functions.html. - Python Standard Library — functools.reduce. docs.python.org/3/library/functools.html#functools.reduce. - PEP 8 — Style Guide: Programming Recommendations (lambda assignment). peps.python.org/pep-0008/. **Primary source:** Python Language Reference §6.14 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Closures in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/closures/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *A closure is a function that remembers and accesses variables from its enclosing scope even after that scope has finished executing — the mechanism behind decorators, callbacks, and factory functions.* ## CANONICAL DEFINITION A closure in Python is a nested function that captures free variables from its enclosing scope. The Python Language Reference defines a closure as a function object that refers to variables from its enclosing scope — these captured variables are stored in the function object's __closure__ attribute as cell objects. ## BEGINNER > The idea A closure is a function that "remembers" values from where it was created — even after the outer function has returned. Think of it as a function with a backpack: the backpack contains the variables it captured from its birthplace, and it carries them wherever it goes. ## What is a closure? A closure forms when a nested function (a function defined inside another function) references a variable from the enclosing function's scope. Python keeps that variable alive so the nested function can access it later — even after the outer function has returned and would normally have been cleaned up. ```python def make_greeting(greeting): """Outer function — defines the greeting.""" def greet(name): """Inner function — a closure that captures 'greeting'.""" return f"{greeting}, {name}!" return greet # return the inner function, not its result # make_greeting has finished — but the captured variable lives on say_hello = make_greeting("Hello") say_namaste = make_greeting("Namaste") print(say_hello("Priya")) # Hello, Priya! print(say_namaste("Arjun")) # Namaste, Arjun! print(say_hello("Rohit")) # Hello, Rohit! # 'greeting' is remembered by each closure independently ``` ## Three conditions for a closure The Python Language Reference specifies that a closure forms when three things are true: there is a nested function; the nested function refers to a variable defined in the enclosing scope; and the enclosing function returns the nested function (or passes it somewhere else). ```python # A counter factory — each call creates an independent counter def make_counter(start=0): count = start # captured variable def counter(): nonlocal count # needed to REASSIGN (not just read) the captured var count += 1 return count return counter a = make_counter() b = make_counter(10) # independent counter starting at 10 print(a()) # 1 print(a()) # 2 print(b()) # 11 print(a()) # 3 — a and b are independent print(b()) # 12 ``` > nonlocal is required to reassign, not just read Reading a captured variable works without `nonlocal`. But if you want to reassign it (`count += 1` is `count = count + 1`), Python treats it as a local variable and raises `UnboundLocalError`. Add `nonlocal count` to tell Python it lives in the enclosing scope. ✅ Beginner tab complete - I can write a factory function that returns a customised inner function - I know the three conditions for a closure: nested function, captured variable, returned function - I know why nonlocal is required to REASSIGN (not just read) a captured variable - I can explain the classic loop closure bug and how to fix it Continue to [Decorators →](/coding/languages/python/decorators/) — which are built on closures ## INTERMEDIATE ## The __closure__ attribute and cell objects Python stores captured variables in *cell objects* — a special wrapper that allows the inner and outer functions to share the same variable. The closure function's `__closure__` attribute holds a tuple of these cells. You can inspect them to see what was captured. ```python def make_multiplier(factor): def multiply(x): return x * factor return multiply double = make_multiplier(2) triple = make_multiplier(3) # Inspect the closure print(double.__closure__) # (,) print(double.__closure__[0].cell_contents) # 2 print(triple.__closure__[0].cell_contents) # 3 # __code__.co_freevars names the captured variables print(double.__code__.co_freevars) # ('factor',) ``` ## Real-world uses of closures Closures appear throughout Python — often without being named. Every decorator is built on closures. Every callback that remembers context is a closure. Every factory function that produces customised functions is a closure. ```python # 1. Memoisation using a closure def memoised(func): cache = {} def wrapper(*args): if args not in cache: cache[args] = func(*args) return cache[args] return wrapper # wrapper closes over 'cache' @memoised def fib(n): if n < 2: return n return fib(n-1) + fib(n-2) print(fib(35)) # Fast — results are cached in the closure's 'cache' # 2. Event handlers — remembering which button was clicked def make_button_handler(button_id): def handler(event): print(f"Button {button_id} clicked") return handler # handler closes over button_id handlers = [make_button_handler(i) for i in range(5)] # 3. Partial application — fixing some arguments def power(base, exponent): return base ** exponent square = lambda x: power(x, 2) # closure via lambda cube = lambda x: power(x, 3) # Better: use functools.partial (same concept, more explicit) from functools import partial square = partial(power, exponent=2) ``` **Commonly confused:** - The classic loop closure bug. [lambda: i for i in range(5)] — all five lambdas return 4, not 0,1,2,3,4. They all close over the same i variable, which ends at 4 after the loop. Fix: [lambda i=i: i for i in range(5)] — bind the current value as a default argument, creating a snapshot instead of a reference. ✅ Intermediate tab complete - I can use __closure__[0].cell_contents to inspect a captured variable - I can explain what __code__.co_freevars contains - I can write a memoisation closure using a dict captured from the enclosing scope Continue to [Decorators →](/coding/languages/python/decorators/) ## EXPERT ## How CPython implements cell objects When CPython's compiler sees a variable referenced in an inner function and assigned in an outer function, it marks that variable as a *cell variable* in the outer function and a *free variable* in the inner function. At runtime, the outer function allocates a cell object on the heap (not the stack) and stores the variable's value inside it. Both the outer and inner functions access the variable through this shared cell — which is why the inner function continues to see changes to the variable even after calling `nonlocal`. ```python import dis def outer(): x = 10 def inner(): return x return inner # outer's bytecode uses STORE_DEREF (not STORE_FAST) for x # inner's bytecode uses LOAD_DEREF (not LOAD_FAST) for x dis.dis(outer) # MAKE_CELL x ← allocates the cell object f = outer() dis.dis(f) # LOAD_DEREF x ← loads through the cell pointer ``` The key bytecode instructions: `MAKE_CELL` allocates the heap-based cell; `STORE_DEREF` writes to the cell; `LOAD_DEREF` reads from it. These are distinct from the stack-based `STORE_FAST`/`LOAD_FAST` used for regular local variables, because the cell must outlive the outer function's stack frame. ✅ Expert tab complete - I know Python allocates cell objects on the heap (not the stack) for captured variables - I know STORE_DEREF and LOAD_DEREF are the bytecode instructions for cell variable access - I can use dis.dis() to see the difference between a closure variable and a regular local Continue to [Decorators →](/coding/languages/python/decorators/) ## SOURCES - Python Language Reference §4.2.2 — Naming and binding: free variables, cell variables, and closures. docs.python.org/3/reference/executionmodel.html. - Python Language Reference §4.2.4 — Interaction with dynamic features. docs.python.org/3/reference/executionmodel.html. - Python Data Model § — __closure__ attribute. docs.python.org/3/reference/datamodel.html#index-55. - Python Built-in Functions — nonlocal statement. docs.python.org/3/reference/simple_stmts.html#the-nonlocal-statement. **Primary source:** Python Language Reference §4.2 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Decorators in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/decorators/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *A decorator is a function that wraps another function to extend or modify its behaviour without changing its source code — the mechanism behind @staticmethod, @property, @dataclass, Flask routes, and pytest fixtures.* ## CANONICAL DEFINITION A decorator in Python is a callable that takes a function (or class) as its argument and returns a modified version of it. The @syntax is syntactic sugar: @decorator above a def is exactly equivalent to func = decorator(func) immediately after the def. ## BEGINNER > The idea A decorator modifies a function the same way a coffee shop "modifies" a coffee order — you add milk, sugar, or flavour without changing what coffee fundamentally is. `@decorator` is simply a clean way of writing `func = decorator(func)`. ## What a decorator is The Python Language Reference specifies that the `@expression` syntax above a function definition is syntactic sugar. It evaluates the expression to get a callable, passes the function to it, and rebinds the function name to the result. Nothing more — no magic, just a function that wraps another function. ```python # WITHOUT decorator syntax def shout(func): def wrapper(*args, **kwargs): result = func(*args, **kwargs) return result.upper() return wrapper def greet(name): return f"hello, {name}" greet = shout(greet) # manually wrap print(greet("Priya")) # HELLO, PRIYA # WITH decorator syntax — identical result def shout(func): def wrapper(*args, **kwargs): result = func(*args, **kwargs) return result.upper() return wrapper @shout # exactly the same as: greet = shout(greet) def greet(name): return f"hello, {name}" print(greet("Priya")) # HELLO, PRIYA ``` ## Writing a useful decorator A well-written decorator uses `functools.wraps` to preserve the original function's metadata — name, docstring, and signature — which is critical for debuggability and tools that inspect function metadata. ```python import time import functools def timer(func): """Decorator that prints how long the function takes.""" @functools.wraps(func) # preserves func.__name__, __doc__, etc. def wrapper(*args, **kwargs): start = time.perf_counter() result = func(*args, **kwargs) elapsed = time.perf_counter() - start print(f"{func.__name__} took {elapsed:.4f}s") return result return wrapper @timer def slow_sum(n): """Add numbers from 1 to n.""" return sum(range(n)) print(slow_sum(10_000_000)) # slow_sum took 0.3124s # 49999995000000 # Without @functools.wraps, slow_sum.__name__ would be 'wrapper' print(slow_sum.__name__) # 'slow_sum' ← preserved correctly ``` ✅ Beginner tab complete - I know @decorator is EXACTLY equivalent to func = decorator(func) - I always use @functools.wraps(func) inside my wrapper functions - I can write a decorator that adds timing, logging, or validation to any function - I know decorators execute at definition time, not call time Continue to [Generators →](/coding/languages/python/generators/) ## INTERMEDIATE ## Stacking decorators and decorators with arguments Multiple decorators can be stacked — they apply from bottom to top (the one closest to the function definition applies first). A decorator with arguments is a function that returns a decorator, adding one more layer of nesting. ```python import functools # Stacking — applied bottom-up @decorator_a @decorator_b def func(): ... # equivalent to: func = decorator_a(decorator_b(func)) # Decorator WITH arguments — requires an extra layer def repeat(times): """Decorator factory — takes an argument, returns a decorator.""" def decorator(func): @functools.wraps(func) def wrapper(*args, **kwargs): for _ in range(times): result = func(*args, **kwargs) return result return wrapper return decorator @repeat(3) # repeat(3) returns a decorator, which wraps greet def greet(name): print(f"Hello, {name}!") greet("Priya") # Hello, Priya! # Hello, Priya! # Hello, Priya! # Practical decorator with arguments: retry on failure def retry(max_attempts=3, delay=1.0): def decorator(func): @functools.wraps(func) def wrapper(*args, **kwargs): import time for attempt in range(1, max_attempts + 1): try: return func(*args, **kwargs) except Exception as e: if attempt == max_attempts: raise print(f"Attempt {attempt} failed: {e}. Retrying...") time.sleep(delay) return wrapper return decorator @retry(max_attempts=3, delay=0.5) def fetch_data(url): import urllib.request return urllib.request.urlopen(url).read() ``` ## Class decorators Decorators work on classes too — the same @syntax applies, and the class is passed to the decorator. Python's built-in `@dataclass` is the most widely used class decorator; it inspects the class's annotations and auto-generates `__init__`, `__repr__`, and `__eq__`. ```python from dataclasses import dataclass # @dataclass is a class decorator — Python's most used @dataclass class Point: x: float y: float p = Point(1.0, 2.0) print(p) # Point(x=1.0, y=2.0) ← generated __repr__ print(p == Point(1.0, 2.0)) # True ← generated __eq__ # Writing your own class decorator def singleton(cls): """Ensure only one instance of cls can exist.""" instances = {} @functools.wraps(cls) def get_instance(*args, **kwargs): if cls not in instances: instances[cls] = cls(*args, **kwargs) return instances[cls] return get_instance @singleton class DatabaseConnection: def __init__(self, url): self.url = url db1 = DatabaseConnection("postgresql://...") db2 = DatabaseConnection("postgresql://...") print(db1 is db2) # True — same instance ``` **Commonly confused:** - Always use @functools.wraps(func) in your wrapper. Without it, the decorated function loses its name, docstring, and signature. This breaks help(), logging, error messages, and any tool that uses function introspection. It takes one line and should be considered mandatory. ✅ Intermediate tab complete - I can write a decorator factory: @repeat(3) requires def repeat(times): def decorator(func): def wrapper(...)... - I know decorators stack bottom-up: @a @b def f is a(b(f)) - I can write a class decorator (takes a class, returns modified class) - I understand that @dataclass is just a class decorator that generates __init__, __repr__, __eq__ Continue to [Generators →](/coding/languages/python/generators/) ## EXPERT ## The descriptor protocol and method decorators Python's built-in method decorators — `@staticmethod`, `@classmethod`, and `@property` — are implemented as descriptors, not as ordinary closures. A descriptor is an object that defines `__get__` (and optionally `__set__`, `__delete__`). When an attribute is accessed on a class or instance, Python calls the descriptor's `__get__` method. This is how `@property` turns a method into an attribute — the property object's `__get__` calls the underlying function transparently. ```python # @property is a descriptor — here's a simplified version class property_like: def __init__(self, fget): self.fget = fget def __get__(self, obj, objtype=None): if obj is None: return self # class access — return descriptor itself return self.fget(obj) # instance access — call the getter class Circle: def __init__(self, radius): self.radius = radius @property_like # equivalent to built-in @property def area(self): import math return math.pi * self.radius ** 2 c = Circle(5) print(c.area) # 78.539... — __get__ called the fget function ``` Understanding the descriptor protocol explains why `@staticmethod` returns a function regardless of whether it's accessed on the class or an instance, while `@classmethod` always passes `cls` as the first argument — these are implemented as distinct descriptor types (`staticmethod` and `classmethod` objects), not as closures. ✅ Expert tab complete - I know @property is a descriptor object, not a simple closure - I can implement a basic descriptor with __get__ - I understand how __get__(self, obj, objtype) receives both the instance and the class Continue to [Generators →](/coding/languages/python/generators/) ## SOURCES - Python Language Reference §8.7 — Function definitions: decorator syntax. docs.python.org/3/reference/compound_stmts.html#function-definitions. - Python Standard Library — functools.wraps. docs.python.org/3/library/functools.html#functools.wraps. - Python Reference — Descriptor HowTo Guide. docs.python.org/3/howto/descriptor.html. - PEP 318 — Decorators for Functions and Methods. peps.python.org/pep-0318/. **Primary source:** Python Language Reference §8.7 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Generators in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/generators/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *A generator is a function that uses yield to produce a sequence of values lazily — one at a time, on demand — without building the entire sequence in memory at once.* ## CANONICAL DEFINITION A generator function is defined like a normal function but uses yield instead of return. Calling it returns a generator iterator — an object that produces values one at a time when iterated. The function body is suspended at each yield and resumes from that exact point on the next call to next(). ## BEGINNER > The idea A regular function runs to completion and returns one value. A generator function pauses at each `yield`, returns that value, and waits — resuming exactly where it left off when you ask for the next value. Think of it as a vending machine: it dispenses one item at a time, on request, rather than dumping its entire inventory at once. ## What makes a generator Any function containing a `yield` statement is a generator function. Calling it does not execute the body — it returns a generator iterator object. The body only runs when you iterate the generator (with `next()` or a `for` loop). ```python def count_up(start, stop): """A generator that counts from start to stop.""" n = start while n <= stop: yield n # pause here, return n, resume next call n += 1 # Calling count_up does NOT run the body yet gen = count_up(1, 5) print(type(gen)) # # Each call to next() runs until the next yield print(next(gen)) # 1 print(next(gen)) # 2 print(next(gen)) # 3 # A for loop calls next() automatically until StopIteration for value in count_up(1, 5): print(value, end=" ") # 1 2 3 4 5 # The generator version of range — for illustration def my_range(n): i = 0 while i < n: yield i i += 1 ``` ## Why generators matter — memory A list comprehension builds the entire list in memory. A generator expression produces values one at a time. For large or infinite sequences, this difference is enormous. ```python import sys # List — all million numbers in memory at once big_list = [x * x for x in range(1_000_000)] print(sys.getsizeof(big_list)) # ~8 MB # Generator — one number at a time, constant memory big_gen = (x * x for x in range(1_000_000)) print(sys.getsizeof(big_gen)) # 104 bytes — always # Same computation, radically different memory use print(sum(big_list)) # 333332833333500000 print(sum(big_gen)) # 333332833333500000 — identical result # Infinite generator — impossible as a list def natural_numbers(): n = 1 while True: # runs forever — but yields one at a time yield n n += 1 # Safe to use with islice from itertools import islice first_10 = list(islice(natural_numbers(), 10)) print(first_10) # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ``` ✅ Beginner tab complete - I know that calling a generator function returns a generator object — it does NOT execute the body - I know each call to next() runs until the next yield, then pauses - I understand why generators use O(1) memory regardless of sequence length - I know generators are single-use — once exhausted, they yield nothing on re-iteration Continue to [Context Managers →](/coding/languages/python/context-managers/) ## INTERMEDIATE ## Generator pipelines Because generators are lazy, you can chain them together into a processing pipeline where data flows through each stage on demand — no intermediate lists, constant memory regardless of input size. ```python def read_lines(filename): """Generator: yield one line at a time from a file.""" with open(filename, encoding="utf-8") as f: for line in f: yield line.rstrip() def filter_non_empty(lines): """Generator: skip blank lines.""" for line in lines: if line: yield line def parse_csv_line(lines): """Generator: split each line into a list.""" for line in lines: yield line.split(",") # Chain them — nothing runs until we iterate lines = read_lines("data.csv") non_empty = filter_non_empty(lines) rows = parse_csv_line(non_empty) for row in rows: print(row) # processes one line at a time, constant memory # Even with a 10GB file — memory use stays tiny ``` ## yield from and generator delegation `yield from` (PEP 380) delegates to another iterable or generator, yielding all its values. It also passes `send()` values and exceptions through to the inner generator, enabling clean composition of generators. ```python # yield from — delegate to another iterable def chain_iterables(*iterables): for it in iterables: yield from it # yield each item from it in turn result = list(chain_iterables([1, 2], [3, 4], [5, 6])) print(result) # [1, 2, 3, 4, 5, 6] # Flatten nested lists of arbitrary depth def flatten(nested): for item in nested: if isinstance(item, list): yield from flatten(item) # recursive delegation else: yield item print(list(flatten([1, [2, [3, 4]], [5, 6]]))) # [1, 2, 3, 4, 5, 6] ``` **Commonly confused:** - Generators are single-use. Once a generator is exhausted (all values yielded), iterating it again yields nothing — StopIteration is raised immediately. To iterate again, create a new generator object by calling the generator function again. ✅ Intermediate tab complete - I can build a multi-stage generator pipeline where each stage is a generator function - I know yield from delegates to another iterable (or generator) completely - I can use yield from for recursive flattening of nested structures - I know return inside a generator raises StopIteration(value) Continue to [Context Managers →](/coding/languages/python/context-managers/) ## EXPERT ## Generator internals: frame suspension and the gi_ attributes When a generator is created, Python allocates a frame object for the generator function — the same structure used for regular function calls, but stored on the heap rather than the stack. The generator iterator holds a reference to this frame. When `next()` is called, CPython resumes execution in that frame from the last `yield` point. The frame stores the instruction pointer (`f_lasti`) and all local variables. ```python def my_gen(): yield 1 yield 2 g = my_gen() # Generator attributes (CPython implementation) print(g.gi_frame) # the suspended frame object print(g.gi_frame.f_lasti) # -1 before first next() next(g) print(g.gi_frame.f_lasti) # byte offset of last executed instruction print(g.gi_code) # the code object (shared across all instances) print(g.gi_running) # True only while executing inside next() # send() — inject a value into the generator at the yield point def accumulator(): total = 0 while True: value = yield total # yield sends total OUT, receives value IN total += value acc = accumulator() next(acc) # prime the generator (advance to first yield) print(acc.send(10)) # 10 print(acc.send(20)) # 30 print(acc.send(5)) # 35 ``` PEP 342 (Python 2.5) added `send()`, `throw()`, and `close()` to turn generators into coroutines. PEP 380 (Python 3.3) added `yield from` for delegation. PEP 492 (Python 3.5) introduced `async def`/`await` as native syntax for coroutines — asyncio's coroutines are implemented on top of generators at the CPython level. ✅ Expert tab complete - I can explain what gi_frame, gi_code, and gi_running contain - I can write a generator that uses yield as an expression to receive sent values - I understand that asyncio coroutines are generators under the hood in CPython Continue to [Context Managers →](/coding/languages/python/context-managers/) ## SOURCES - Python Language Reference §6.2.9 — Yield expressions. docs.python.org/3/reference/expressions.html#yield-expressions. - Python Language Reference §8.8 — The yield statement. docs.python.org/3/reference/simple_stmts.html#the-yield-statement. - PEP 255 — Simple Generators; PEP 342 — Coroutines via Enhanced Generators; PEP 380 — yield from. peps.python.org. - Python Tutorial §9.10 — Generators. docs.python.org/3/tutorial/classes.html#generators. **Primary source:** Python Language Reference §6.2.9 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Context Managers in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/context-managers/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *A context manager is an object that sets up and tears down a resource automatically using the with statement — guaranteeing cleanup happens even if an exception occurs.* ## CANONICAL DEFINITION A context manager is an object that defines __enter__ and __exit__ methods. When used in a with statement, Python calls __enter__ at the start (setup) and __exit__ at the end (teardown, even if an exception was raised). The with statement is defined in the Python Language Reference as the primary mechanism for resource management. ## BEGINNER > The idea A context manager is like a responsible adult who not only opens a door for you but guarantees they'll close it when you're done — even if you trip and fall while going through. The `with` statement is the door; `__exit__` is the closing. ## The with statement The most common context manager in Python is file handling. Without `with`, you must remember to call `f.close()` — and if an exception occurs before that line, the file stays open. With `with`, the file is always closed, regardless of what happens. ```python # WITHOUT with — dangerous: close() may not run on exception f = open("data.txt", "r") try: content = f.read() finally: f.close() # must remember this # WITH with — always closes, even on exception with open("data.txt", encoding="utf-8") as f: content = f.read() # f.close() is called automatically here — always # Multiple context managers in one with (Python 3.1+) with open("input.txt") as src, open("output.txt", "w") as dst: dst.write(src.read()) # Parenthesised form for many managers (Python 3.10+) with ( open("a.txt") as a, open("b.txt") as b, open("c.txt") as c, ): data = a.read() + b.read() + c.read() ``` ## Common built-in context managers Many Python built-ins and standard library modules support the context manager protocol: | Context Manager | Manages | On exit | | --- | --- | --- | | `open()` | File handles | Closes the file | | `threading.Lock()` | Thread locks | Releases the lock | | `decimal.localcontext()` | Decimal precision | Restores old precision | | `unittest.mock.patch()` | Mocked objects | Restores original | | `contextlib.suppress()` | Expected exceptions | Silences named exceptions | | Database connections | DB connections | Commits or rolls back | ✅ Beginner tab complete - I always use "with open(...) as f:" — never open() without with - I know with guarantees __exit__ runs even if an exception occurs inside the block - I can list 4 common Python objects that support the context manager protocol - I can use multiple context managers in a single with statement Continue to [Object-Oriented Python →](/coding/languages/python/oop/) ## INTERMEDIATE ## The __enter__ and __exit__ protocol Any object implementing `__enter__` and `__exit__` can be used in a `with` statement. `__enter__` is called at the start and its return value is bound to the `as` variable. `__exit__` is called at the end, receiving exception information if one occurred — it can suppress the exception by returning a truthy value. ```python class Timer: """Context manager that measures elapsed time.""" import time def __enter__(self): import time self.start = time.perf_counter() return self # bound to the 'as' variable def __exit__(self, exc_type, exc_val, exc_tb): import time self.elapsed = time.perf_counter() - self.start print(f"Elapsed: {self.elapsed:.4f}s") return False # False = don't suppress exceptions with Timer() as t: total = sum(range(10_000_000)) # Elapsed: 0.312s print(t.elapsed) # also accessible after the with block # __exit__ signature: exc_type, exc_val, exc_tb # If no exception: all three are None # If exception: they contain the exception info # Return True to suppress the exception; False/None to propagate it class SuppressIOError: def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): return exc_type is IOError # suppress IOErrors only ``` ## Writing context managers with contextlib The `contextlib` module's `@contextmanager` decorator lets you write a context manager as a generator function — simpler than implementing the full class protocol. Everything before `yield` is the `__enter__` body; everything after is the `__exit__` body. ```python from contextlib import contextmanager import os, tempfile @contextmanager def temporary_directory(): """Create a temp dir, yield it, delete it on exit.""" tmpdir = tempfile.mkdtemp() try: yield tmpdir # __enter__ returns this value finally: import shutil shutil.rmtree(tmpdir) # __exit__ cleanup with temporary_directory() as tmpdir: # work inside the temp directory filepath = os.path.join(tmpdir, "data.txt") with open(filepath, "w") as f: f.write("temporary data") # tmpdir is deleted here, regardless of exceptions # contextlib.suppress — the cleanest way to ignore specific exceptions from contextlib import suppress with suppress(FileNotFoundError): os.remove("might_not_exist.txt") # No try/except needed — FileNotFoundError is silently ignored ``` **Commonly confused:** - The as variable is NOT the context manager itself — it's what __enter__ returns. For open(), __enter__ returns the file object, not the open() return value. For threading.Lock(), __enter__ returns the lock. For your own context managers, return whatever the user needs to work with inside the block. ✅ Intermediate tab complete - I can implement __enter__ and __exit__ on a class to make it a context manager - I know __exit__ receives (exc_type, exc_val, exc_tb) — all None if no exception - I can write a context manager using @contextmanager with yield as the boundary - I know contextlib.suppress(FileNotFoundError) as an alternative to try/except/pass Continue to [Object-Oriented Python →](/coding/languages/python/oop/) ## EXPERT ## contextlib advanced tools The `contextlib` module provides several advanced context manager utilities. `ExitStack` manages a dynamic stack of context managers — useful when the number of managers is not known until runtime. `nullcontext` is a no-op context manager useful for optional context management. `AsyncExitStack` and `asynccontextmanager` provide async equivalents. ```python from contextlib import ExitStack, nullcontext # ExitStack — dynamic number of context managers def process_files(filenames): with ExitStack() as stack: files = [stack.enter_context(open(f)) for f in filenames] # All files opened; all will be closed on exit regardless of exceptions return [f.read() for f in files] # nullcontext — conditional context management def process(data, output_file=None): with (open(output_file, "w") if output_file else nullcontext()) as f: result = compute(data) if f: f.write(result) return result # asynccontextmanager for async with from contextlib import asynccontextmanager @asynccontextmanager async def async_timer(): import time start = time.perf_counter() try: yield finally: print(f"Async elapsed: {time.perf_counter() - start:.4f}s") async def main(): async with async_timer(): import asyncio await asyncio.sleep(0.1) ``` The Python Language Reference specifies that the `with` statement desugars to a precise sequence of operations: evaluate the context expression, call `__enter__`, bind the result to the as target, execute the body, and unconditionally call `__exit__` with exception information (or all-None if no exception). This precise specification means the `with` statement's behaviour is guaranteed to be consistent regardless of what the context manager does internally. ✅ Expert tab complete - I can use ExitStack to manage a dynamic number of context managers - I can write an async context manager with @asynccontextmanager for use with async with - I can trace through the exact desugar: __enter__, body execution, __exit__ call sequence Continue to [Object-Oriented Python →](/coding/languages/python/oop/) ## SOURCES - Python Language Reference §8.5 — The with statement. docs.python.org/3/reference/compound_stmts.html#the-with-statement. - Python Standard Library — contextlib. docs.python.org/3/library/contextlib.html. - PEP 343 — The "with" Statement. peps.python.org/pep-0343/. - Python Reference — Context Manager Types. docs.python.org/3/library/stdtypes.html#context-manager-types. **Primary source:** Python Language Reference §8.5 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Object-Oriented Python in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/oop/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Classes, instances, dunder methods, properties, dataclasses — Python's complete object model in one place.* ## CANONICAL DEFINITION Python OOP uses classes as blueprints for objects — instances carry their own state (instance attributes), methods are functions bound to instances via the descriptor protocol, and inheritance follows C3 linearisation (MRO). ## BEGINNER > The analogy A class is a blueprint. An instance is the house built from it. The blueprint defines what rooms every house has and what they do — the actual house has its own rooms filled with its own furniture. ## Classes and instances ```python class BankAccount: # Class attribute — shared by ALL instances bank_name = "The Codex Bank" def __init__(self, owner, balance=0): # Instance attributes — unique to each instance self.owner = owner self.balance = balance def deposit(self, amount): if amount self.balance: raise ValueError("Insufficient funds") self.balance -= amount def __repr__(self): return f"BankAccount({self.owner!r}, {self.balance})" def __str__(self): return f"{self.owner}'s account: ₹{self.balance:,}" # Create instances acc1 = BankAccount("Priya", 10000) acc2 = BankAccount("Rahul") acc1.deposit(5000) print(acc1) # Priya's account: ₹15,000 print(repr(acc1)) # BankAccount('Priya', 15000) print(acc1.bank_name) # The Codex Bank print(BankAccount.bank_name) # same — class attribute ``` ## Inheritance ```python class Animal: def __init__(self, name, sound): self.name = name self.sound = sound def speak(self): return f"{self.name} says {self.sound}" def __repr__(self): return f"{type(self).__name__}({self.name!r})" class Dog(Animal): def __init__(self, name): super().__init__(name, "Woof") # call parent __init__ def fetch(self, item): return f"{self.name} fetches the {item}!" class Cat(Animal): def __init__(self, name): super().__init__(name, "Meow") def speak(self): # override parent method return f"{self.name} purrs softly" d = Dog("Rex") c = Cat("Whiskers") print(d.speak()) # Rex says Woof print(c.speak()) # Whiskers purrs softly print(d.fetch("ball")) # Polymorphism — same interface, different behaviour animals = [Dog("Rex"), Cat("Whiskers"), Dog("Bruno")] for animal in animals: print(animal.speak()) # calls correct speak() for each type ``` ## Special methods (dunder methods) Special methods let your class integrate with Python's built-in syntax. Define `__add__` and your objects work with `+`. Define `__len__` and `len()` works. Define `__iter__` and `__next__` and your class is iterable with `for`. ```python class Vector: def __init__(self, x, y): self.x, self.y = x, y def __repr__(self): return f"Vector({self.x}, {self.y})" def __add__(self, other): return Vector(self.x + other.x, self.y + other.y) def __mul__(self, scalar): return Vector(self.x * scalar, self.y * scalar) def __len__(self): return 2 # always 2 components def __eq__(self, other): return self.x == other.x and self.y == other.y def __abs__(self): return (self.x**2 + self.y**2) ** 0.5 def __bool__(self): return self.x != 0 or self.y != 0 # zero vector is falsy v1 = Vector(1, 2) v2 = Vector(3, 4) print(v1 + v2) # Vector(4, 6) print(v1 * 3) # Vector(3, 6) print(len(v1)) # 2 print(abs(v2)) # 5.0 print(v1 == Vector(1, 2)) # True ``` ✅ Beginner tab complete — check your understanding - I can define a class with __init__ and at least one other method - I understand what self refers to (the specific object being operated on) - I can create two objects from the same class and give them different attribute values - I can write a subclass that inherits from a parent class and adds new behaviour Continue to [Error Handling →](/coding/languages/python/error-handling/) ## INTERMEDIATE ## Properties and descriptors The `@property` decorator lets you define getter, setter, and deleter for an attribute while accessing it like a plain attribute — not a method call. This is Python's way of enforcing encapsulation without ugly `getX()`/`setX()` Java-style methods. ```python class Temperature: def __init__(self, celsius=0): self._celsius = celsius # private by convention (single underscore) @property def celsius(self): return self._celsius @celsius.setter def celsius(self, value): if value < -273.15: raise ValueError(f"Temperature below absolute zero: {value}") self._celsius = value @property def fahrenheit(self): return self._celsius * 9/5 + 32 @fahrenheit.setter def fahrenheit(self, value): self.celsius = (value - 32) * 5/9 # delegates validation to celsius setter t = Temperature(100) print(t.celsius) # 100 print(t.fahrenheit) # 212.0 t.fahrenheit = 32 print(t.celsius) # 0.0 # t.celsius = -300 # raises ValueError ``` ## Dataclasses (Python 3.7+) `@dataclass` (PEP 557) automatically generates `__init__`, `__repr__`, `__eq__`, and optionally `__lt__`/`__hash__` for a class based on its field annotations. It replaces 20+ lines of boilerplate with a decorator. ```python from dataclasses import dataclass, field @dataclass class Point: x: float y: float @dataclass class Employee: name: str department: str salary: float = 50000.0 # default value skills: list = field(default_factory=list) # mutable default def give_raise(self, percent: float) -> None: self.salary *= (1 + percent / 100) # Auto-generated __init__, __repr__, __eq__ p1 = Point(1.0, 2.0) p2 = Point(1.0, 2.0) print(p1) # Point(x=1.0, y=2.0) print(p1 == p2) # True emp = Employee("Priya", "Engineering") emp.skills.append("Python") emp.give_raise(10) print(emp) # Employee(name='Priya', department='Engineering', salary=55000.0, skills=['Python']) # @dataclass(frozen=True) makes it immutable (and hashable) @dataclass(frozen=True) class FrozenPoint: x: float y: float fp = FrozenPoint(1.0, 2.0) # fp.x = 3.0 # FrozenInstanceError ``` ## Multiple inheritance and MRO Python supports multiple inheritance. When a method is called, Python uses the C3 linearisation algorithm (Method Resolution Order — MRO) to determine which class's method to call. `ClassName.__mro__` shows the resolution order. `super()` always follows the MRO, which is why `super()` is safe in multiple inheritance scenarios. ```python class A: def method(self): return "A" class B(A): def method(self): return "B" class C(A): def method(self): return "C" class D(B, C): # inherits from both B and C pass d = D() print(d.method()) # B — follows MRO print(D.__mro__) # (, , , , ) # Mixin pattern — add behaviour without deep inheritance class JSONMixin: def to_json(self): import json return json.dumps(self.__dict__) class TimestampMixin: def created_at(self): from datetime import datetime return datetime.now().isoformat() class User(JSONMixin, TimestampMixin): def __init__(self, name, email): self.name = name self.email = email user = User("Priya", "priya@example.com") print(user.to_json()) # {"name": "Priya", "email": "priya@example.com"} print(user.created_at()) ``` **Commonly confused:** - Name mangling (__name) is not true privacy. Attributes starting with __ (double underscore) are name-mangled to _ClassName__name by Python — they are still accessible, just renamed. This is a convention for subclass collision avoidance, not access control. Single underscore (_name) is the idiomatic convention for "internal use". - Class attributes and instance attributes are different. A class attribute is shared by all instances. If an instance assigns to the same name, it creates an instance attribute that shadows the class attribute — the class attribute is unchanged. Mutating a mutable class attribute (like a list) through one instance affects all instances. How this connects Requires first [Variables & Types](/coding/languages/python/variables-and-types/)[Functions & Scope](/coding/languages/python/functions-and-scope/) Enables [Error Handling](/coding/languages/python/error-handling/)[Inheritance](/coding/concepts/inheritance/) Primary source [Python Data Model — docs.python.org](https://docs.python.org/3/reference/datamodel.html) ✅ Intermediate tab complete — check your understanding - I can implement __str__ so print(my_object) shows something readable - I can implement __eq__ so two objects with the same data compare as equal - I can use @property to create a computed attribute - I can write a dataclass and know when it's preferable to a regular class Continue to [Error Handling →](/coding/languages/python/error-handling/) ## EXPERT ## The Python object model: metaclasses In Python, classes are themselves objects — instances of a metaclass. The default metaclass is `type`. When Python executes a `class` statement, it calls the metaclass to create the class object. A custom metaclass can intercept class creation to add or modify attributes, enforce constraints, register classes, or implement the class pattern. The process: `type.__new__(mcs, name, bases, namespace)` creates the class; `type.__init__(cls, name, bases, namespace)` initialises it. Abstract Base Classes (`abc.ABCMeta`) use a custom metaclass to enforce abstract method implementation. ## Descriptor protocol Python Language Reference §3.3.2: a descriptor is any object that defines `__get__`, `__set__`, or `__delete__`. When a descriptor is accessed through a class or instance, Python calls these methods instead of returning the object directly. `property`, `classmethod`, `staticmethod`, and `__slots__` are all implemented as descriptors. Non-data descriptors (only `__get__`) are overridden by instance variables. Data descriptors (have `__set__` or `__delete__`) take precedence over instance variables — this is how `property` setters work. ✅ Expert tab complete - I can use super() correctly in a multiple inheritance hierarchy - I can explain what MRO is and use type.mro() to inspect it - I know what a metaclass is and can give one real-world example of where they're used - I understand why @property works (it's a descriptor object) Continue to [Error Handling →](/coding/languages/python/error-handling/) ## SOURCES - Python Language Reference §3 — Data model. docs.python.org/3/reference/datamodel.html. - Python Language Reference §3.3.2 — Customising attribute access and the descriptor protocol. - PEP 557 — Data Classes. peps.python.org/pep-0557/. - PEP 3119 — Introducing Abstract Base Classes. peps.python.org/pep-3119/. **Primary source:** docs.python.org/3/reference/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Dataclasses in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/dataclasses/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The @dataclass decorator auto-generates __init__, __repr__, and __eq__ from type-annotated fields — the modern, concise way to write classes that mainly hold data.* ## CANONICAL DEFINITION A dataclass is a class decorated with @dataclass from the dataclasses module (PEP 557, Python 3.7+). The decorator reads the annotated fields of the class and generates boilerplate methods automatically, including __init__, __repr__, and __eq__. ## BEGINNER > The idea A dataclass writes the tedious parts of a class for you. You declare what fields the class has; Python generates the constructor, a readable printout, and equality comparison automatically. Less code, fewer bugs. ## The problem dataclasses solve Writing a simple data-holding class the traditional way involves a lot of repetition. The `@dataclass` decorator eliminates it. ```python # WITHOUT dataclass — manual boilerplate class PointOld: def __init__(self, x, y): self.x = x self.y = y def __repr__(self): return f"PointOld(x={self.x}, y={self.y})" def __eq__(self, other): return (self.x, self.y) == (other.x, other.y) # WITH dataclass — same behaviour, far less code from dataclasses import dataclass @dataclass class Point: x: float y: float p = Point(1.0, 2.0) print(p) # Point(x=1.0, y=2.0) ← generated __repr__ print(p == Point(1.0, 2.0)) # True ← generated __eq__ print(p.x, p.y) # 1.0 2.0 ``` > Type annotations are required A dataclass field must have a type annotation — that is how the decorator finds it. `x: float` is a field. `x = 5` without an annotation is a plain class attribute and is ignored by the dataclass machinery. If you don't want to constrain the type, annotate with `typing.Any`. ## Fields and default values ```python from dataclasses import dataclass @dataclass class User: name: str age: int = 0 # default value active: bool = True # default value role: str = "member" u1 = User("Priya") # uses defaults print(u1) # User(name='Priya', age=0, active=True, role='member') u2 = User("Arjun", 30, role="admin") print(u2) # User(name='Arjun', age=30, active=True, role='admin') # Fields WITHOUT defaults must come before fields WITH defaults # (same rule as function parameters) # @dataclass # class Bad: # x: int = 0 # y: int # TypeError — non-default after default ``` ✅ Beginner tab complete - I can write a @dataclass with annotated fields and no manual __init__ - I know fields must have type annotations to be recognised - I can give fields default values (non-default fields must come first) - I know the generated __repr__ and __eq__ work automatically Continue to [Enums →](/coding/languages/python/enums/) ## INTERMEDIATE ## Decorator options The `@dataclass` decorator accepts arguments that control what it generates. The most useful are `frozen` (immutability), `order` (comparison operators), and `slots` (memory optimisation). ```python from dataclasses import dataclass # frozen=True — immutable, and hashable (usable as dict key / in a set) @dataclass(frozen=True) class Coordinate: lat: float lon: float c = Coordinate(19.07, 72.87) # c.lat = 0 # raises FrozenInstanceError locations = {c: "Mumbai"} # works — frozen dataclasses are hashable # order=True — generates __lt__, __le__, __gt__, __ge__ @dataclass(order=True) class Version: major: int minor: int patch: int versions = [Version(1, 2, 0), Version(1, 0, 5), Version(2, 0, 0)] print(sorted(versions)) # sorted by (major, minor, patch) tuple order # [Version(1,0,5), Version(1,2,0), Version(2,0,0)] # slots=True (Python 3.10+) — less memory, faster attribute access @dataclass(slots=True) class Particle: x: float y: float z: float # No per-instance __dict__ — significant memory savings at scale ``` ## field() and __post_init__ Mutable defaults (lists, dicts) cannot be used directly — the same gotcha as mutable default arguments. The `field()` function with `default_factory` solves this. `__post_init__` runs validation or derived-value computation after the generated `__init__`. ```python from dataclasses import dataclass, field @dataclass class ShoppingCart: customer: str # WRONG: items: list = [] — shared mutable default, raises ValueError # RIGHT: use default_factory items: list = field(default_factory=list) tags: dict = field(default_factory=dict) c1 = ShoppingCart("Priya") c1.items.append("book") c2 = ShoppingCart("Arjun") print(c2.items) # [] — independent, not shared # field() options @dataclass class Account: name: str balance: float = 0.0 # exclude from __repr__ and __init__ _id: int = field(default=0, repr=False) # computed, not passed to __init__ display: str = field(init=False, default="") def __post_init__(self): # runs after __init__ — validation and derived values if self.balance < 0: raise ValueError("Balance cannot be negative") self.display = f"{self.name} (${self.balance:.2f})" a = Account("Priya", 100.0) print(a.display) # "Priya ($100.00)" ``` **Commonly confused:** - Never use a mutable default directly. items: list = [] raises ValueError in a dataclass (Python protects you here, unlike plain functions). Always use field(default_factory=list). ✅ Intermediate tab complete - I use field(default_factory=list) for mutable defaults — never items: list = [] - I can make an immutable, hashable dataclass with frozen=True - I can generate comparison operators with order=True - I can validate or compute derived values in __post_init__ Continue to [Enums →](/coding/languages/python/enums/) ## EXPERT ## How @dataclass generates code at class-creation time The `@dataclass` decorator runs once, when the class is defined. It inspects `__annotations__`, builds the source code for each method as a string, compiles it with `exec()`, and attaches the resulting functions to the class. This is why dataclasses have zero runtime overhead compared to hand-written classes — the generated methods are ordinary Python methods, indistinguishable from ones you wrote yourself. ```python from dataclasses import dataclass, fields, asdict, astuple @dataclass class Point: x: int y: int # Introspection API p = Point(1, 2) for f in fields(Point): print(f.name, f.type, f.default) # x int , y int # Conversion helpers print(asdict(p)) # {'x': 1, 'y': 2} — recursive print(astuple(p)) # (1, 2) # The generated __init__ is a real function you can inspect import inspect print(inspect.getsource(Point.__init__)) # def __init__(self, x: int, y: int) -> None: # self.x = x # self.y = y # __dataclass_fields__ holds the field metadata print(Point.__dataclass_fields__.keys()) # dict_keys(['x', 'y']) ``` The `InitVar` type marks a parameter that is passed to `__init__` and forwarded to `__post_init__` but not stored as a field. Combined with `field(init=False)` and `__post_init__`, this gives precise control over the initialisation pipeline — useful for fields whose values are derived from other fields rather than passed in directly. ✅ Expert tab complete - I know @dataclass generates real methods at definition time with zero runtime overhead - I can use fields() and asdict() to introspect and convert dataclasses - I know InitVar marks init-only parameters forwarded to __post_init__ Continue to [Enums →](/coding/languages/python/enums/) ## SOURCES - Python Standard Library — dataclasses. docs.python.org/3/library/dataclasses.html. - PEP 557 — Data Classes. peps.python.org/pep-0557/. - PEP 681 — Data Class Transforms. peps.python.org/pep-0681/. - Python Standard Library — dataclasses.field, asdict, fields. docs.python.org/3/library/dataclasses.html#dataclasses.field. **Primary source:** Python dataclasses (PEP 557) *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Enumerations (enum) in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/enums/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *An enum is a set of named constant values bound to a single type — replacing scattered magic numbers and strings with readable, comparable, self-documenting members.* ## CANONICAL DEFINITION An enumeration is a class derived from enum.Enum whose members are unique constant values. Defined in the enum module (PEP 435, Python 3.4+), each member has a name and a value, and members are singletons that compare by identity. ## BEGINNER > The idea An enum gives names to a fixed set of related values. Instead of writing `status = 1` and trying to remember what 1 means, you write `status = Status.ACTIVE` — readable, typo-proof, and self-documenting. ## Why enums exist Without enums, code is full of "magic" numbers and strings whose meaning is unclear and whose typos are silent bugs. Enums replace them with named members. ```python # WITHOUT enum — magic values, error-prone status = "active" # typo "activ" would be a silent bug if status == "active": ... # WITH enum — named, typo-proof, self-documenting from enum import Enum class Status(Enum): ACTIVE = "active" INACTIVE = "inactive" PENDING = "pending" BANNED = "banned" status = Status.ACTIVE print(status) # Status.ACTIVE print(status.name) # "ACTIVE" print(status.value) # "active" if status == Status.ACTIVE: print("User is active") # A typo is now an AttributeError, caught immediately: # Status.ACTIV -> AttributeError ``` > Enum members are singletons Each enum member exists exactly once. `Status.ACTIVE is Status.ACTIVE` is always True, and you compare members with `is` or `==`. Two members can never accidentally be equal unless you deliberately alias them. ## Defining and accessing enums ```python from enum import Enum class Color(Enum): RED = 1 GREEN = 2 BLUE = 3 # Three ways to access a member print(Color.RED) # Color.RED (by attribute) print(Color(1)) # Color.RED (by value) print(Color["RED"]) # Color.RED (by name) # Members have .name and .value print(Color.RED.name) # "RED" print(Color.RED.value) # 1 # Iterate over all members (in definition order) for color in Color: print(color.name, color.value) # RED 1 / GREEN 2 / BLUE 3 # Membership and count print(len(Color)) # 3 print(Color.RED in Color) # True ``` ✅ Beginner tab complete - I can define an Enum and access members by attribute, value, or name - I know member.name and member.value - I know enum members are singletons compared with is or == - I can iterate over all members with a for loop Continue to [Error Handling →](/coding/languages/python/error-handling/) ## INTERMEDIATE ## IntEnum, StrEnum, Flag, and auto() The enum module provides specialised base classes. `auto()` assigns values automatically. `IntEnum` and `StrEnum` make members behave as int/str. `Flag` supports bitwise combination. ```python from enum import Enum, IntEnum, Flag, auto # auto() — assigns 1, 2, 3, ... automatically class Direction(Enum): NORTH = auto() SOUTH = auto() EAST = auto() WEST = auto() print(Direction.NORTH.value) # 1 # IntEnum — members ARE integers, comparable with int class Priority(IntEnum): LOW = 1 MEDIUM = 2 HIGH = 3 print(Priority.HIGH > Priority.LOW) # True — int comparison print(Priority.HIGH == 3) # True — equals the int print(sorted([Priority.HIGH, Priority.LOW])) # [LOW, HIGH] # StrEnum (Python 3.11+) — members ARE strings from enum import StrEnum class Env(StrEnum): DEV = "development" PROD = "production" print(Env.PROD == "production") # True print(Env.PROD.upper()) # "PRODUCTION" — string methods work # Flag — combine members with bitwise operators class Permission(Flag): READ = auto() WRITE = auto() EXECUTE = auto() access = Permission.READ | Permission.WRITE print(Permission.READ in access) # True print(Permission.EXECUTE in access) # False ``` ## Methods, aliases, and @unique ```python from enum import Enum, unique # Enums can have methods like any class class Planet(Enum): MERCURY = (3.303e+23, 2.4397e6) EARTH = (5.976e+24, 6.37814e6) def __init__(self, mass, radius): self.mass = mass self.radius = radius @property def surface_gravity(self): G = 6.67300e-11 return G * self.mass / (self.radius ** 2) print(f"{Planet.EARTH.surface_gravity:.2f}") # 9.80 # Aliases — two names, same value (second is an alias) class Status(Enum): ACTIVE = 1 RUNNING = 1 # alias for ACTIVE print(Status.RUNNING is Status.ACTIVE) # True print(list(Status)) # [Status.ACTIVE] — aliases excluded from iteration # @unique — forbid aliases, raise ValueError on duplicate values @unique class Weekday(Enum): MON = 1 TUE = 2 # WED = 1 # would raise ValueError: duplicate values ``` **Commonly confused:** - Enum vs IntEnum. A plain Enum member is NOT equal to its value: Color.RED == 1 is False. An IntEnum member IS: Priority.LOW == 1 is True. Use IntEnum only when you genuinely need integer behaviour (e.g. comparing or sorting); otherwise plain Enum is safer because it prevents accidental comparison with raw numbers. ✅ Intermediate tab complete - I can use auto() to assign member values automatically - I know IntEnum members equal their int value but plain Enum members do not - I can combine Flag members with bitwise | and test with in - I know @unique forbids aliases (duplicate values) Continue to [Error Handling →](/coding/languages/python/error-handling/) ## EXPERT ## EnumMeta, member creation, and the functional API Enums are implemented through a metaclass, `EnumMeta` (also exposed as `EnumType`). When you define an Enum subclass, the metaclass intercepts class creation, collects the member definitions, converts each into a singleton member instance, and stores them in `_member_map_` and `_value2member_map_`. This is why members are singletons and why `Color(1)` can look up a member by value — the metaclass built a reverse mapping. ```python from enum import Enum # Functional API — create an enum without the class syntax Animal = Enum("Animal", ["DOG", "CAT", "BIRD"]) print(list(Animal)) # [Animal.DOG, Animal.CAT, Animal.BIRD] Animal2 = Enum("Animal2", {"DOG": 1, "CAT": 2}) # with explicit values # Internal mappings built by EnumMeta class Color(Enum): RED = 1 GREEN = 2 print(Color._member_map_) # {'RED': Color.RED, 'GREEN': Color.GREEN} print(Color._value2member_map_) # {1: Color.RED, 2: Color.GREEN} # _missing_ — hook for handling unknown values class Status(Enum): ACTIVE = 1 INACTIVE = 2 @classmethod def _missing_(cls, value): # called when Status(value) finds no match return cls.INACTIVE # default fallback instead of ValueError print(Status(99)) # Status.INACTIVE — via _missing_ # __members__ includes aliases; iteration does not class S(Enum): A = 1 B = 1 # alias print(list(S.__members__)) # ['A', 'B'] (includes alias) print([m.name for m in S]) # ['A'] (excludes alias) ``` The `_missing_` classmethod is a powerful hook: it is invoked when a value lookup (`Status(value)`) finds no matching member, letting you implement default values, case-insensitive matching, or aliasing logic. Combined with the functional API — `Enum("Name", names)` — and mix-in types like `IntEnum`, the enum module supports everything from simple constants to rich domain types with behaviour. ✅ Expert tab complete - I know enums are built by the EnumMeta metaclass at class creation - I can create an enum dynamically with the functional API: Enum("Name", [...]) - I can implement _missing_ to handle unknown values gracefully Continue to [Error Handling →](/coding/languages/python/error-handling/) ## SOURCES - Python Standard Library — enum. docs.python.org/3/library/enum.html. - PEP 435 — Adding an Enum type to the Python standard library. peps.python.org/pep-0435/. - Python Standard Library — enum.StrEnum (Python 3.11). docs.python.org/3/library/enum.html#enum.StrEnum. - Python HOWTO — Enum HOWTO. docs.python.org/3/howto/enum.html. **Primary source:** Python enum module (PEP 435) *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Abstract Base Classes (abc) in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/abc/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *An abstract base class defines an interface that subclasses must implement — Python’s way of declaring "every subclass must provide these methods" and enforcing it at instantiation time.* ## CANONICAL DEFINITION An abstract base class (ABC), provided by the abc module, is a class that cannot be instantiated directly and may declare abstract methods that concrete subclasses are required to override. Defined in PEP 3119, ABCs formalise interfaces and enable isinstance() checks against capabilities. ## BEGINNER > The idea An abstract base class is a contract. It says "any class that claims to be one of these must provide these specific methods." Python refuses to create an object from a class that hasn't fulfilled the contract — so mistakes are caught immediately, not deep in your program. ## The problem ABCs solve Without an ABC, nothing stops you from creating an incomplete subclass and only discovering the missing method when it is called — possibly much later, in production. An ABC moves that error to the moment of object creation. ```python from abc import ABC, abstractmethod class Shape(ABC): # inherit from ABC @abstractmethod def area(self): ... # no implementation — subclasses must provide it @abstractmethod def perimeter(self): ... class Circle(Shape): def __init__(self, radius): self.radius = radius def area(self): return 3.14159 * self.radius ** 2 def perimeter(self): return 2 * 3.14159 * self.radius c = Circle(5) # works — both methods implemented print(c.area()) # 78.53... # You cannot instantiate the ABC itself # Shape() # TypeError: Can't instantiate abstract class Shape # A subclass that forgets a method also can't be instantiated class Square(Shape): def area(self): return 4 # forgot perimeter() # Square() # TypeError: Can't instantiate abstract class Square # # with abstract method perimeter ``` > The error happens at instantiation, not definition Defining an incomplete subclass is allowed; the `TypeError` fires the moment you try to create an instance. This is exactly when you want it — you cannot accidentally pass around an object that is missing required methods. ✅ Beginner tab complete - I can define an ABC with @abstractmethod - I know an ABC cannot be instantiated directly - I know a subclass missing an abstract method cannot be instantiated - I know the error fires at instantiation, not definition Continue to [Data Model →](/coding/languages/python/python-data-model/) ## INTERMEDIATE ## Abstract properties and partial implementations ABCs can declare abstract properties and can also provide concrete (working) methods alongside abstract ones — giving subclasses a base to build on while still enforcing the required pieces. ```python from abc import ABC, abstractmethod class Employee(ABC): def __init__(self, name): self.name = name @property @abstractmethod def monthly_salary(self): ... # abstract property — must be overridden @abstractmethod def role(self): ... def annual_salary(self): # CONCRETE method — inherited as-is return self.monthly_salary * 12 def describe(self): # concrete, uses abstract methods return f"{self.name} ({self.role()}): {self.annual_salary()}/yr" class Manager(Employee): @property def monthly_salary(self): return 150_000 def role(self): return "Manager" m = Manager("Priya") print(m.annual_salary()) # 1800000 — concrete method works print(m.describe()) # Priya (Manager): 1800000/yr ``` ## Virtual subclasses with register() An ABC can recognise a class as a subclass without that class inheriting from it — via `register()`. This is how Python lets unrelated classes satisfy an interface, the basis of "duck typing made checkable." ```python from abc import ABC class Serialisable(ABC): @abstractmethod def to_json(self): ... # An existing class we don't want to (or can't) modify class LegacyRecord: def to_json(self): return "{}" # Register it as a virtual subclass Serialisable.register(LegacyRecord) print(issubclass(LegacyRecord, Serialisable)) # True print(isinstance(LegacyRecord(), Serialisable)) # True # NOTE: register() does NOT enforce the methods exist — it's a promise. # Use it for retrofitting interfaces onto existing/third-party classes. # __subclasshook__ — recognise ANY class with the right method class Sized(ABC): @classmethod def __subclasshook__(cls, C): if cls is Sized: if any("__len__" in B.__dict__ for B in C.__mro__): return True return NotImplemented print(issubclass(list, Sized)) # True — list has __len__ ``` **Commonly confused:** - ABC vs Protocol. An ABC uses nominal typing — a class must inherit (or be registered) to count. A Protocol (PEP 544) uses structural typing — any class with the right methods counts, no inheritance needed. Use ABC when you want enforcement at instantiation; use Protocol for static duck typing checked by type checkers. ✅ Intermediate tab complete - I can declare an abstract property - I can mix concrete methods with abstract ones in an ABC - I can register a virtual subclass with ABC.register() - I know register() does not verify the methods exist Continue to [Data Model →](/coding/languages/python/python-data-model/) ## EXPERT ## collections.abc and the ABCMeta machinery ABCs are implemented through the `ABCMeta` metaclass. When a class uses `ABCMeta` (which `ABC` does via inheritance), the metaclass tracks abstract methods in `__abstractmethods__` and overrides `__instancecheck__` and `__subclasscheck__` so that `isinstance` and `issubclass` consult registered virtual subclasses and `__subclasshook__`. The standard library ships a complete set of ABCs in `collections.abc` describing Python's container protocols. ```python from collections.abc import ( Iterable, Iterator, Sequence, MutableSequence, Mapping, MutableMapping, Set, Hashable, Callable ) # These ABCs let you check capabilities, not concrete types print(isinstance([], Sequence)) # True print(isinstance({}, Mapping)) # True print(isinstance("abc", Iterable)) # True print(isinstance(len, Callable)) # True print(isinstance(42, Hashable)) # True # Inheriting from a collections.abc ABC gives you mixin methods free class MyList(Sequence): def __init__(self, data): self._data = data def __getitem__(self, i): return self._data[i] def __len__(self): return len(self._data) # Sequence provides __contains__, __iter__, __reversed__, # index(), and count() automatically from these two methods ml = MyList([10, 20, 30]) print(20 in ml) # True — __contains__ provided by Sequence print(list(reversed(ml)))# [30,20,10] — __reversed__ provided print(ml.index(20)) # 1 — index() provided # __abstractmethods__ holds the set still needing implementation from abc import ABC, abstractmethod class Base(ABC): @abstractmethod def foo(self): ... print(Base.__abstractmethods__) # frozenset({'foo'}) ``` The mixin behaviour is the practical payoff: inherit from `collections.abc.Sequence` and implement just `__getitem__` and `__len__`, and you get `__contains__`, `__iter__`, `__reversed__`, `index`, and `count` for free. This is why building custom containers on the `collections.abc` ABCs is far less work than implementing every protocol method by hand. ✅ Expert tab complete - I know collections.abc defines Iterable, Sequence, Mapping, etc. - I know inheriting from Sequence gives __contains__/__iter__/index/count free - I know __subclasshook__ can recognise any class with the right methods Continue to [Data Model →](/coding/languages/python/python-data-model/) ## SOURCES - Python Standard Library — abc (Abstract Base Classes). docs.python.org/3/library/abc.html. - Python Standard Library — collections.abc. docs.python.org/3/library/collections.abc.html. - PEP 3119 — Introducing Abstract Base Classes. peps.python.org/pep-3119/. - PEP 544 — Protocols: Structural subtyping (contrast with ABCs). peps.python.org/pep-0544/. **Primary source:** Python abc module (PEP 3119) *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # The Python Data Model (Dunder Methods) ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/python-data-model/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The data model is the set of special "dunder" methods (__init__, __len__, __getitem__, and dozens more) that let your own classes integrate seamlessly with Python’s built-in syntax and functions.* ## CANONICAL DEFINITION The Python data model, defined in the Python Language Reference, specifies the special methods — named with double underscores ("dunders") — that the interpreter calls to implement built-in operations. Implementing these methods lets custom objects support syntax like len(obj), obj[key], obj + other, and for x in obj. ## BEGINNER > The idea When you write `len(x)`, Python actually calls `x.__len__()`. When you write `a + b`, Python calls `a.__add__(b)`. These double-underscore methods are hooks. Implement them on your own class and your objects work with Python's built-in syntax as if they were native types. ## "Dunder" methods are hooks Dunder (double-underscore) methods are how Python's syntax maps onto method calls. You rarely call them directly — instead you use the syntax, and Python calls the dunder for you. ```python class Money: def __init__(self, rupees): # called by Money(100) self.rupees = rupees def __repr__(self): # called by repr() and the REPL return f"Money({self.rupees})" def __str__(self): # called by str() and print() return f"₹{self.rupees:,}" def __eq__(self, other): # called by == return self.rupees == other.rupees def __lt__(self, other): # called by < return self.rupees < other.rupees m = Money(1500) print(m) # ₹1,500 — __str__ print(repr(m)) # Money(1500) — __repr__ print(Money(100) == Money(100)) # True — __eq__ print(Money(50) < Money(80)) # True — __lt__ # You never call m.__str__() yourself — print() does it for you. ``` > __repr__ vs __str__ `__str__` is for humans (what `print()` shows). `__repr__` is for developers (what the REPL and debuggers show) and should ideally be valid Python that recreates the object. If you only write one, write `__repr__` — Python falls back to it when `__str__` is missing. ✅ Beginner tab complete - I know len(x) calls x.__len__() and a+b calls a.__add__(b) - I can implement __init__, __repr__, __str__, __eq__ - I know __repr__ is for developers and __str__ is for humans - I know to write __repr__ if I write only one of them Continue to [Error Handling →](/coding/languages/python/error-handling/) ## INTERMEDIATE ## The container protocol Implementing a handful of dunders makes your object behave like a built-in container — supporting `len()`, indexing, membership tests, and iteration. ```python class Playlist: def __init__(self, songs): self._songs = list(songs) def __len__(self): # len(playlist) return len(self._songs) def __getitem__(self, index): # playlist[0], playlist[1:3], and iteration return self._songs[index] def __setitem__(self, index, value): # playlist[0] = "new" self._songs[index] = value def __contains__(self, song): # "song" in playlist return song in self._songs p = Playlist(["Song A", "Song B", "Song C"]) print(len(p)) # 3 — __len__ print(p[0]) # Song A — __getitem__ print(p[1:]) # ['Song B', 'Song C'] — slicing via __getitem__ print("Song B" in p) # True — __contains__ p[0] = "New Song" # __setitem__ # __getitem__ alone is enough to make it iterable! for song in p: # Python calls __getitem__ with 0, 1, 2... until IndexError print(song) ``` ## Operator overloading and callable objects ```python class Vector: def __init__(self, x, y): self.x, self.y = x, y def __repr__(self): return f"Vector({self.x}, {self.y})" def __add__(self, other): # self + other return Vector(self.x + other.x, self.y + other.y) def __sub__(self, other): # self - other return Vector(self.x - other.x, self.y - other.y) def __mul__(self, scalar): # self * scalar return Vector(self.x * scalar, self.y * scalar) def __abs__(self): # abs(self) return (self.x ** 2 + self.y ** 2) ** 0.5 def __bool__(self): # bool(self), truthiness return bool(self.x or self.y) v1 = Vector(2, 3) v2 = Vector(1, 1) print(v1 + v2) # Vector(3, 4) print(v1 * 3) # Vector(6, 9) print(abs(Vector(3, 4))) # 5.0 print(bool(Vector(0, 0))) # False # __call__ makes an instance callable like a function class Multiplier: def __init__(self, factor): self.factor = factor def __call__(self, x): # instance(x) return x * self.factor triple = Multiplier(3) print(triple(10)) # 30 — the instance behaves like a function ``` **Commonly confused:** - Implementing __eq__ disables __hash__. If you define __eq__, Python sets __hash__ to None, making instances unhashable (unusable as dict keys / set members). Add __hash__ back, or use @dataclass(frozen=True) which handles both. ✅ Intermediate tab complete - I can make a class support len(), indexing, and "in" - I can overload +, -, * with __add__, __sub__, __mul__ - I can make an instance callable with __call__ - I know defining __eq__ disables __hash__ unless I add it back Continue to [Error Handling →](/coding/languages/python/error-handling/) ## EXPERT ## Reflected operators, __new__, and attribute hooks The data model includes reflected (right-hand) operator methods, called when the left operand does not know how to handle the operation. `a + b` first tries `a.__add__(b)`; if that returns `NotImplemented`, Python tries `b.__radd__(a)`. This is how `2 * vector` can work even though `int` has no idea what a Vector is — Python falls back to `vector.__rmul__(2)`. ```python class Vector: def __init__(self, x, y): self.x, self.y = x, y def __mul__(self, scalar): # vector * 2 return Vector(self.x * scalar, self.y * scalar) def __rmul__(self, scalar): # 2 * vector (reflected) return self.__mul__(scalar) v = Vector(1, 2) # print((2 * v)) # works via __rmul__ since int.__mul__(v) returns NotImplemented # __new__ vs __init__ — creation vs initialisation class Singleton: _instance = None def __new__(cls): # __new__ CREATES the instance if cls._instance is None: cls._instance = super().__new__(cls) return cls._instance def __init__(self): # __init__ INITIALISES it (may run repeatedly) pass print(Singleton() is Singleton()) # True — same instance # Attribute access hooks class Proxy: def __getattr__(self, name): # called ONLY when normal lookup fails return f"no attribute named {name}" def __setattr__(self, name, value):# called on EVERY attribute assignment print(f"setting {name} = {value}") super().__setattr__(name, value) p = Proxy() p.foo = 10 # prints: setting foo = 10 print(p.missing) # "no attribute named missing" # Context manager protocol — __enter__ / __exit__ # Iterator protocol — __iter__ / __next__ # Descriptor protocol — __get__ / __set__ / __delete__ # These are all part of the data model too. ``` The data model is the unifying theory behind much of Python: context managers (`__enter__`/`__exit__`), iterators (`__iter__`/`__next__`), descriptors (`__get__`/`__set__`), numeric coercion, attribute access, and object creation (`__new__` before `__init__`) are all specified there. Learning to read the data model chapter of the Language Reference is what turns "Python has a lot of magic" into "Python has a small set of consistent protocols." ✅ Expert tab complete - I know 2*vector works via vector.__rmul__ when int.__mul__ returns NotImplemented - I know __new__ creates the instance and __init__ initialises it - I know __getattr__ is called only when normal lookup fails - I can see how the data model unifies most of Python’s "magic" Continue to [Error Handling →](/coding/languages/python/error-handling/) ## SOURCES - Python Language Reference §3 — Data Model (special method names). docs.python.org/3/reference/datamodel.html. - Python Language Reference §3.3.1 — Basic customization (__new__, __init__, __repr__, __eq__, __hash__). docs.python.org/3/reference/datamodel.html#basic-customization. - Python Language Reference §3.3.7 — Emulating container types. docs.python.org/3/reference/datamodel.html#emulating-container-types. - Python Language Reference §3.3.8 — Emulating numeric types (reflected operands). docs.python.org/3/reference/datamodel.html#emulating-numeric-types. **Primary source:** Python Language Reference §3 (Data Model) *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Error Handling in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/error-handling/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Exceptions are events, not catastrophes. Python's error handling model makes catching, raising, and recovering clean and explicit.* ## CANONICAL DEFINITION Python error handling uses exceptions — objects that inherit from BaseException, raised with raise and caught with try/except — with finally for guaranteed cleanup and context managers (the with statement) for resource management. ## BEGINNER > The analogy Error handling is like a safety net under a tightrope. You hope you don't fall, but you have a plan for when you do. In Python, errors are not crashes waiting to happen — they're events your program can catch, handle, and recover from. ## Python's exception hierarchy In Python, all exceptions are objects that inherit from `BaseException`. Most exceptions you'll deal with inherit from `Exception` (which inherits from `BaseException`). System-exiting exceptions (`SystemExit`, `KeyboardInterrupt`, `GeneratorExit`) inherit directly from `BaseException` — they should almost never be caught. | Exception | When it occurs | | --- | --- | | `ValueError` | Right type, wrong value: `int("hello")` | | `TypeError` | Wrong type: `"3" + 5` | | `NameError` | Undefined name: `print(x)` before assigning x | | `IndexError` | Out-of-bounds list access: `lst[100]` | | `KeyError` | Missing dict key: `d["missing"]` | | `AttributeError` | No such attribute: `None.name` | | `FileNotFoundError` | File does not exist | | `ZeroDivisionError` | `10 / 0` | | `StopIteration` | Iterator exhausted (normal — do not catch) | | `ImportError` | Module not found | ## try / except / else / finally ```python # Basic try/except try: result = 10 / 0 except ZeroDivisionError: print("Cannot divide by zero") # Multiple except clauses def parse_number(text): try: return int(text) except ValueError: try: return float(text) except ValueError: return None # Catch multiple exceptions in one clause try: data = int(input("Enter a number: ")) except (ValueError, EOFError) as e: print(f"Input error: {e}") # else: runs only if no exception was raised # finally: ALWAYS runs — cleanup code def read_file(path): f = None try: f = open(path) return f.read() except FileNotFoundError: return None else: print("File read successfully") # only if no exception finally: if f: f.close() # ALWAYS closes the file # Better: use context manager def read_file_clean(path): try: with open(path) as f: return f.read() except FileNotFoundError: return None ``` ## Raising exceptions ```python # Raise a built-in exception def divide(a, b): if b == 0: raise ZeroDivisionError("b cannot be zero") return a / b # Custom exception — inherit from Exception class InsufficientFundsError(Exception): def __init__(self, requested, available): self.requested = requested self.available = available super().__init__( f"Requested ₹{requested:,} but only ₹{available:,} available" ) def withdraw(balance, amount): if amount > balance: raise InsufficientFundsError(amount, balance) return balance - amount try: withdraw(1000, 2000) except InsufficientFundsError as e: print(e) # Requested ₹2,000 but only ₹1,000 available print(e.requested) # 2000 print(e.available) # 1000 # Re-raise — preserve original exception try: result = 1 / 0 except ZeroDivisionError: print("Handling...") raise # re-raises the same exception # Exception chaining (Python 3) try: 1 / 0 except ZeroDivisionError as original: raise ValueError("Bad input") from original # chains context ``` ✅ Beginner tab complete — check your understanding - I can wrap risky code in try/except and handle the error gracefully - I know the 5 most common exception types: TypeError, ValueError, KeyError, IndexError, FileNotFoundError - I can use the else clause (runs if no exception) and finally clause (always runs) - I can raise my own exception with raise ValueError("message") Continue to [File I/O →](/coding/languages/python/file-io/) — where error handling becomes essential (files that don't exist, wrong permissions, etc.) ## INTERMEDIATE ## Context managers and the with statement A context manager implements `__enter__` (called on entry, can return a value bound by `as`) and `__exit__` (called on exit, receives exception info if one occurred). If `__exit__` returns truthy, the exception is suppressed. This is how `with open(...)` guarantees the file closes even if an exception is raised inside the block. ```python # Custom context manager using a class class Timer: def __enter__(self): import time self.start = time.perf_counter() return self def __exit__(self, exc_type, exc_val, exc_tb): import time self.elapsed = time.perf_counter() - self.start print(f"Elapsed: {self.elapsed:.4f}s") return False # do not suppress exceptions with Timer() as t: sum(range(1_000_000)) # Elapsed: 0.0432s # contextlib.contextmanager — generator-based context manager from contextlib import contextmanager, suppress @contextmanager def managed_resource(): print("Acquiring resource") resource = {"status": "active"} try: yield resource # body of the with block runs here finally: resource["status"] = "closed" print("Released resource") with managed_resource() as r: print(r["status"]) # active # Released resource # suppress — silently ignore specific exceptions with suppress(FileNotFoundError): open("nonexistent.txt") # No error raised ``` ## Exception groups (Python 3.11+) PEP 654 (Python 3.11) introduced `ExceptionGroup` for handling multiple concurrent exceptions — essential for async code where many tasks may fail simultaneously. The `except*` syntax handles exception groups. ```python # Python 3.11+ import asyncio async def failing_task(n): raise ValueError(f"Task {n} failed") async def main(): async with asyncio.TaskGroup() as tg: tg.create_task(failing_task(1)) tg.create_task(failing_task(2)) # If multiple tasks fail, raises ExceptionGroup try: asyncio.run(main()) except* ValueError as eg: for exc in eg.exceptions: print(exc) # Task 1 failed, Task 2 failed # Create and handle manually try: raise ExceptionGroup("multiple errors", [ ValueError("bad value"), TypeError("bad type"), ]) except* ValueError as eg: print(f"ValueErrors: {eg.exceptions}") except* TypeError as eg: print(f"TypeErrors: {eg.exceptions}") ``` **Commonly confused:** - Never use a bare except:. except: without a type catches everything — including KeyboardInterrupt, SystemExit, and programming errors. Use except Exception: at the minimum, or better, name specific exceptions. Catching everything silences bugs. - Don't use exceptions for flow control. In Python, exceptions are for errors and genuinely exceptional conditions. The EAFP style ("Easier to Ask Forgiveness than Permission") — try it and handle the exception — is idiomatic. But using StopIteration outside a generator, or raising exceptions in normal code paths, is an anti-pattern that harms performance and readability. How this connects Requires first [Functions & Scope](/coding/languages/python/functions-and-scope/) Enables [Standard Library](/coding/languages/python/standard-library/)[Error Handling concept](/coding/concepts/error-handling/) Primary source [Python Built-in Exceptions — docs.python.org](https://docs.python.org/3/library/exceptions.html) ✅ Intermediate tab complete — check your understanding - I understand what __enter__ and __exit__ do and why the with statement calls them - I can write a custom exception class that inherits from Exception - I know when to catch a specific exception vs catching Exception base class - I know never to use bare except: (always specify an exception type) Continue to [File I/O →](/coding/languages/python/file-io/) ## EXPERT ## Exception handling and the CPython implementation In CPython, exception handling uses a zero-cost model at the bytecode level (since Python 3.11, PEP 654 and related changes): when no exception is raised, try/except blocks have essentially zero overhead — the interpreter does not set up any data structure on the hot path. When an exception is raised, CPython uses a combination of the exception table (generated by the compiler — a mapping from bytecode range to handler) and the exception value stored in the thread state. This is documented in CPython's ceval.c and the compiler module. ## PEP 3151 — Reworking the OS and IO exception hierarchy Python 3.3 (PEP 3151) unified the OS error hierarchy. Before 3.3, you had to check `errno` to distinguish `ENOENT` from `EACCES`. After 3.3, there are specific exception classes: `FileNotFoundError`, `PermissionError`, `IsADirectoryError`, `TimeoutError`, etc. — all subclasses of `OSError`. This is why you can write `except FileNotFoundError` instead of `except OSError as e: if e.errno == errno.ENOENT:`. ✅ Expert tab complete - I know that Python's exceptions have zero overhead when NOT raised - I can use raise X from Y to chain exceptions and preserve context - I know the difference between __cause__ (explicit chaining) and __context__ (implicit chaining) Continue to [File I/O →](/coding/languages/python/file-io/) ## SOURCES - Python Language Reference §8 — Compound statements (try). docs.python.org/3/reference/compound_stmts.html#try. - Python Built-in Exceptions. docs.python.org/3/library/exceptions.html. - PEP 343 — The "with" Statement. peps.python.org/pep-0343/. - PEP 654 — Exception Groups and except*. peps.python.org/pep-0654/. - PEP 3151 — Reworking the OS and IO exception hierarchy. peps.python.org/pep-3151/. **Primary source:** docs.python.org/3/reference/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # The Python Exception Hierarchy ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/exception-hierarchy/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *Every Python exception is a class in a single inheritance tree rooted at BaseException — understanding that tree is what lets you catch exactly the errors you mean to, and no more.* ## CANONICAL DEFINITION All built-in exceptions in Python form a class hierarchy rooted at BaseException. Most application-level exceptions inherit from Exception, a direct subclass of BaseException. When you catch an exception class, you also catch every class that inherits from it. ## BEGINNER > The idea Every error in Python is an object built from a class, and those classes are arranged in a family tree. When you catch a parent class, you automatically catch all its children. Knowing the tree means you catch exactly what you intend. ## The shape of the tree At the very top sits `BaseException`. Almost everything you handle inherits from `Exception`, which is one level down. A few special exceptions — the ones that should usually *not* be caught — sit beside `Exception` rather than under it. ```python BaseException # the root — catches absolutely everything ├── SystemExit # raised by sys.exit() — let it propagate ├── KeyboardInterrupt # raised by Ctrl-C — let it propagate ├── GeneratorExit └── Exception # ← catch THIS or its children, not BaseException ├── ArithmeticError │ └── ZeroDivisionError ├── LookupError │ ├── IndexError # list[999] │ └── KeyError # dict["missing"] ├── TypeError # "5" + 3 ├── ValueError # int("abc") ├── OSError │ ├── FileNotFoundError │ └── PermissionError ├── AttributeError # obj.missing_attribute ├── NameError # using an undefined variable └── ... many more ``` > Catch Exception, not BaseException `except Exception:` catches ordinary errors but deliberately lets `KeyboardInterrupt` (Ctrl-C) and `SystemExit` through — so the user can always stop your program. `except BaseException:` (or a bare `except:`) swallows even those, which is almost always a bug. ## The exceptions you will meet most ```python int("abc") # ValueError: invalid literal for int() "5" + 3 # TypeError: can only concatenate str to str [1, 2, 3][99] # IndexError: list index out of range {"a": 1}["b"] # KeyError: 'b' open("nope.txt") # FileNotFoundError undefined_name # NameError: name 'undefined_name' is not defined "hi".missing() # AttributeError: 'str' object has no attribute 'missing' 10 / 0 # ZeroDivisionError: division by zero # Each name tells you the category of problem: # ValueError = right type, wrong value # TypeError = wrong type entirely # LookupError = asked for something not there (IndexError/KeyError) ``` ✅ Beginner tab complete - I know all exceptions inherit from BaseException, and most from Exception - I know to catch Exception, never BaseException or bare except - I can name what ValueError, TypeError, KeyError, IndexError each mean - I know catching a parent class catches all its child classes Continue to [Walrus Operator →](/coding/languages/python/walrus-operator/) ## INTERMEDIATE ## Catching by level — specific vs broad Because catching a parent catches all children, you choose *how broadly* to catch by choosing *how high* in the tree you reach. Catch as specifically as you can. ```python # Catching a parent catches all children try: data = {"a": 1} print(data["b"]) except LookupError: # catches BOTH KeyError and IndexError print("key or index missing") # Catch multiple specific types in one except try: value = int(user_input) result = 100 / value except (ValueError, ZeroDivisionError) as e: print(f"Bad input: {e}") # Order matters — most specific FIRST try: risky() except FileNotFoundError: # specific child first print("file missing") except OSError: # broader parent second print("some OS error") # If reversed, OSError would catch FileNotFoundError and the # specific handler would never run. # Inspect the exception object try: int("abc") except ValueError as e: print(type(e).__name__) # 'ValueError' print(str(e)) # the message print(e.args) # ('invalid literal...',) ``` ## Custom exceptions and chaining ```python # Define a custom exception by inheriting from Exception class ValidationError(Exception): """Raised when user input fails validation.""" pass class AgeError(ValidationError): # a more specific subclass pass def set_age(age): if not isinstance(age, int): raise AgeError(f"Age must be an integer, got {type(age).__name__}") if age < 0: raise AgeError("Age cannot be negative") return age # Callers can catch the specific type OR the parent try: set_age(-5) except ValidationError as e: # catches AgeError too (it's a subclass) print(f"Validation failed: {e}") # Exception chaining — preserve the original cause def load_config(path): try: with open(path) as f: return parse(f.read()) except FileNotFoundError as e: raise ValidationError("Config missing") from e # 'from e' records the original — the traceback shows BOTH ``` **Commonly confused:** - except order: specific before general. Python tries except clauses top to bottom and uses the first that matches. If you put except Exception first, it catches everything and your specific handlers below never run. ✅ Intermediate tab complete - I put specific exceptions before general ones in except clauses - I can catch multiple types with except (A, B) - I can define a custom exception by inheriting from Exception - I use raise X from e to preserve the original cause Continue to [Walrus Operator →](/coding/languages/python/walrus-operator/) ## EXPERT ## BaseExceptionGroup, except*, and the OSError consolidation Two structural changes shaped the modern hierarchy. PEP 3151 (Python 3.3) collapsed the old tangle of `IOError`, `OSError`, `WindowsError`, and `socket.error` into a single `OSError` tree with meaningful subclasses like `FileNotFoundError` and `PermissionError` — so you can catch `FileNotFoundError` directly instead of inspecting `errno`. PEP 654 (Python 3.11) added `ExceptionGroup` and the `except*` syntax for handling multiple simultaneous exceptions, which arise in concurrent code where several tasks may fail at once. ```python # PEP 654 — ExceptionGroup and except* (Python 3.11+) def run_tasks(): errors = [] for task in tasks: try: task.run() except Exception as e: errors.append(e) if errors: raise ExceptionGroup("task failures", errors) # except* handles groups — each clause peels off matching exceptions try: run_tasks() except* ValueError as eg: print(f"{len(eg.exceptions)} value errors") except* TypeError as eg: print(f"{len(eg.exceptions)} type errors") # Both clauses can run — a group may contain both kinds # __cause__ vs __context__ — the two chaining attributes try: try: 1 / 0 except ZeroDivisionError as e: raise ValueError("bad") from e except ValueError as e: print(e.__cause__) # the ZeroDivisionError (explicit, via 'from') print(e.__context__) # also set (implicit — what we were handling) print(e.__suppress_context__) # True when 'from' was used ``` Every raised exception records two chaining attributes: `__context__` is set implicitly to whatever exception was being handled when the new one was raised, and `__cause__` is set explicitly by `raise ... from ...`. The `from None` form sets `__suppress_context__` to hide the implicit chain entirely — useful when re-raising a clean domain exception and you do not want the internal cause cluttering the traceback. ✅ Expert tab complete - I know ExceptionGroup and except* handle multiple simultaneous errors (3.11+) - I know __cause__ is set by raise...from and __context__ is set implicitly - I know FileNotFoundError/PermissionError are OSError subclasses (PEP 3151) Continue to [Walrus Operator →](/coding/languages/python/walrus-operator/) ## SOURCES - Python Standard Library — Built-in Exceptions (full hierarchy). docs.python.org/3/library/exceptions.html. - PEP 3151 — Reworking the OS and IO exception hierarchy. peps.python.org/pep-3151/. - PEP 654 — Exception Groups and except*. peps.python.org/pep-0654/. - Python Language Reference §8.4 — The try statement. docs.python.org/3/reference/compound_stmts.html#try. **Primary source:** Python stdlib — Built-in Exceptions *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # The Walrus Operator (:=) in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/walrus-operator/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The walrus operator := assigns a value to a variable as part of a larger expression — letting you capture and test a value in a single step.* ## CANONICAL DEFINITION The walrus operator (:=), formally an assignment expression, was introduced by PEP 572 in Python 3.8. Unlike the = statement, := is an expression that both assigns to a name and evaluates to the assigned value, so it can be used inside conditions, comprehensions, and other expressions. ## BEGINNER > The idea Normally `=` is a statement on its own line. The walrus `:=` lets you assign a value *inside* a bigger expression — for example, computing a value and testing it in the same `if`. The name comes from its resemblance to a walrus's eyes and tusks. ## Assigning inside an expression The walrus operator captures a value into a variable at the same moment you use that value. This removes the common pattern of computing something, then immediately testing it on the next line. ```python # WITHOUT walrus — compute, then test on separate lines name = input("Name: ") if len(name) > 50: print("Too long") # WITH walrus — assign and test in one expression if (length := len(name)) > 50: print(f"Too long: {length} chars") # 'length' is now available for use afterwards too # Useful when you need the value AND the test import re text = "order #12345" if (match := re.search(r"#(\d+)", text)): print(f"Order number: {match.group(1)}") # 12345 # Without walrus you'd call re.search twice or use a temp variable # Avoid recomputing an expensive value data = [1, 2, 3, 4, 5] if (total := sum(data)) > 10: print(f"Sum {total} exceeds limit") # reuses total, no recompute ``` > Walrus needs parentheses in most places To avoid ambiguity, `:=` usually must be wrapped in parentheses: `if (n := len(x)) > 10:`. A bare `:=` at statement level (`n := 5` on its own line) is a syntax error — use the normal `=` there. The walrus is for when you are *inside* a larger expression. ✅ Beginner tab complete - I can use := to assign and test in one if statement - I know := requires parentheses in most contexts - I know := is an expression while = is a statement - I know a bare := at top level is a syntax error Continue to [File I/O →](/coding/languages/python/file-io/) ## INTERMEDIATE ## The classic use: while loops reading input The walrus operator's most celebrated use is collapsing the "read, test, process, read again" loop pattern into a single clean line — eliminating the duplicated read call. ```python # WITHOUT walrus — the read appears twice (before loop AND inside) line = input("> ") while line != "quit": print(f"You said: {line}") line = input("> ") # duplicated read — easy to forget # WITH walrus — read once, in the condition while (line := input("> ")) != "quit": print(f"You said: {line}") # Reading a file in fixed-size chunks — a canonical example with open("big.bin", "rb") as f: while (chunk := f.read(8192)): # read until empty bytes process(chunk) # Consuming from a queue until empty import queue q = queue.Queue() # ... fill q ... while not q.empty(): item = q.get() process(item) ``` ## In comprehensions — compute once, use twice Inside a comprehension, the walrus lets you compute a value once and use it both in the filter condition and the output expression — avoiding a double computation. ```python def expensive(x): # imagine this is slow return x * x - 10 # WITHOUT walrus — expensive() called TWICE per item result = [expensive(x) for x in range(10) if expensive(x) > 0] # WITH walrus — expensive() called ONCE per item result = [y for x in range(10) if (y := expensive(x)) > 0] print(result) # [6, 15, 26, 39, 54, 71] # Capture intermediate values while filtering data = ["1,2", "3,4", "bad", "5,6"] parsed = [ (a, b) for item in data if len(parts := item.split(",")) == 2 and parts[0].isdigit() for a, b in [parts] ] # Walrus scope in comprehensions LEAKS to the enclosing scope # (unlike the loop variable, which does not) nums = [y := x + 1 for x in range(3)] print(y) # 3 — y escaped the comprehension (last assigned value) ``` **Commonly confused:** - := vs =. = is a statement; it cannot appear inside an if or comprehension. := is an expression; it can. You cannot use := as a plain top-level assignment (x := 5 alone is a syntax error) — use = there. ✅ Intermediate tab complete - I can write while (line := input()) != "quit" - I can read file chunks with while (chunk := f.read(n)) - I can use := in a comprehension to avoid computing a value twice - I know walrus names leak from comprehensions into the enclosing scope Continue to [File I/O →](/coding/languages/python/file-io/) ## EXPERT ## Scoping rules, prohibited forms, and readability PEP 572 specifies precise rules for assignment expressions. The bound name belongs to the *containing* scope, not any comprehension it appears in — the one deliberate exception to the rule that comprehensions have their own scope. Several forms are explicitly prohibited to prevent confusing or redundant code: you cannot use `:=` at the top level of an expression statement, cannot assign to attributes or subscripts (`obj.x := 1` is illegal), and cannot mix it with `=` on the same target. ```python # Comprehension scope exception — y binds OUTSIDE the comprehension total = 0 values = [total := total + x for x in range(5)] print(values) # [0, 1, 3, 6, 10] — running total print(total) # 10 — the walrus name persists in enclosing scope # Prohibited forms (all SyntaxError): # x := 5 # bare top-level — use x = 5 # obj.attr := 5 # cannot target an attribute # d["key"] := 5 # cannot target a subscript # x = (y := 5) = 3 # cannot chain with = # Legitimate advanced use: avoid redundant work in any() / all() lines = ["", " ", "hello", "world"] # Find first non-blank line AND keep it if any((stripped := ln.strip()) for ln in lines): pass # 'stripped' holds the LAST evaluated value (short-circuit aware) # Use in f-strings for debugging (3.8+) combined with walrus import math r = 5 print(f"{(area := math.pi * r * r) = :.2f}") # area = 78.54 print(area) # 78.539... — still available ``` The PEP's stated motivation is to reduce a specific, common redundancy — computing a value, naming it, and immediately testing it — without encouraging dense one-liners. The style guidance from the PEP authors is explicit: use the walrus where it genuinely removes duplication or a throwaway temporary, and avoid it where a plain assignment on the preceding line would read more clearly. Overuse produces expressions that are hard to scan, which defeats the purpose. ✅ Expert tab complete - I know walrus binds in the containing scope, even inside a comprehension - I know obj.attr := and d[key] := are illegal - I use walrus only where it removes genuine duplication Continue to [File I/O →](/coding/languages/python/file-io/) ## SOURCES - Python Language Reference — Assignment expressions. docs.python.org/3/reference/expressions.html#assignment-expressions. - PEP 572 — Assignment Expressions. peps.python.org/pep-0572/. - Python What's New in 3.8 — Assignment expressions. docs.python.org/3/whatsnew/3.8.html. **Primary source:** PEP 572 (Python 3.8) *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # File I/O in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/file-io/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-16 *Reading and writing files in Python — the open() function, file modes, text vs binary, encodings, and the with statement for guaranteed cleanup.* ## CANONICAL DEFINITION File input/output in Python is performed through file objects returned by the built-in open() function, which connects a program to a file using a specified mode and encoding; the with statement is the idiomatic way to use them because it guarantees the file is closed even if an error occurs. ## BEGINNER > The idea To work with a file, you **open** it, **read** or **write** its contents, then **close** it. Python's `with` statement handles the closing automatically — so the safe, standard pattern is `with open(...) as f:`. ## Opening a file with open() The built-in `open()` function returns a file object. Its two most important arguments are the file path and the *mode* — a short string saying whether you want to read, write, or append, and whether the data is text or binary. | Mode | Meaning | | --- | --- | | `'r'` | Read (default). Error if the file does not exist. | | `'w'` | Write. Creates the file, or **truncates** (empties) an existing one. | | `'a'` | Append. Writes to the end; creates the file if needed. | | `'x'` | Exclusive creation. Error if the file already exists. | | `'r+'` | Read and write (file must exist). | | `'b'` | Binary mode, combined with another: `'rb'`, `'wb'`. | | `'t'` | Text mode (default), combined: `'rt'`. | ## Reading a file ```python # The standard, safe pattern — with statement closes the file for you with open("notes.txt", "r", encoding="utf-8") as f: content = f.read() # entire file as one string print(content) # Read line by line — memory-efficient for large files with open("notes.txt", encoding="utf-8") as f: for line in f: # iterating a file yields lines print(line.rstrip()) # rstrip removes the trailing newline # Read all lines into a list with open("notes.txt", encoding="utf-8") as f: lines = f.readlines() # ['first line\n', 'second line\n', ...] # Read one line at a time with open("notes.txt", encoding="utf-8") as f: first = f.readline() # just the first line ``` > Always specify encoding Pass `encoding="utf-8"` explicitly when opening text files. Without it, Python uses a platform-dependent default that can differ between machines, causing the same code to behave differently on Windows, macOS, and Linux. UTF-8 is the safe, portable choice for almost all text. ## Writing a file ```python # Write mode 'w' — creates or OVERWRITES the file with open("output.txt", "w", encoding="utf-8") as f: f.write("First line\n") # write() does NOT add a newline f.write("Second line\n") # so add \n yourself # Write a list of lines at once lines = ["Priya\n", "Arjun\n", "Rohit\n"] with open("names.txt", "w", encoding="utf-8") as f: f.writelines(lines) # writelines adds no newlines either # Append mode 'a' — adds to the end, keeps existing content with open("log.txt", "a", encoding="utf-8") as f: f.write("New log entry\n") # print() can write to a file via the file= argument with open("report.txt", "w", encoding="utf-8") as f: print("Total:", 42, file=f) # print adds a newline automatically ``` ✅ Beginner tab complete — check your understanding - I use "with open(...) as f:" every time — never open() without with - I always pass encoding="utf-8" when opening text files - I know the difference between mode "r" (read), "w" (write — erases), and "a" (append) - I can read a file line by line with "for line in f:" instead of loading the whole thing Continue to [Modules & Packages →](/coding/languages/python/modules-and-packages/) ## INTERMEDIATE ## Binary mode and encodings Text mode returns `str` objects and handles encoding/decoding for you. Binary mode (`'b'`) returns `bytes` objects with no decoding — use it for images, audio, compressed files, or any non-text data. In text mode, the file's bytes are decoded using the specified encoding; in binary mode you receive the raw bytes unchanged. ```python # Binary read — get raw bytes, no decoding with open("photo.jpg", "rb") as f: data = f.read() # data is a bytes object print(type(data)) # print(len(data), "bytes") # Binary write — must write bytes, not str with open("copy.jpg", "wb") as f: f.write(data) # Copy a binary file in chunks (memory-safe for large files) with open("big.zip", "rb") as src, open("backup.zip", "wb") as dst: while chunk := src.read(8192): # read 8 KB at a time dst.write(chunk) # Explicit encoding for non-UTF-8 text with open("legacy.txt", "r", encoding="latin-1") as f: text = f.read() # Handle decoding errors gracefully with open("messy.txt", "r", encoding="utf-8", errors="replace") as f: text = f.read() # invalid bytes become the replacement char ``` ## Seeking and the file position A file object maintains a current position. `tell()` reports it; `seek()` moves it. This lets you re-read or jump within a file without reopening it. ```python with open("data.txt", "r", encoding="utf-8") as f: print(f.tell()) # 0 — at the start chunk = f.read(5) # read 5 characters print(f.tell()) # position after 5 chars f.seek(0) # jump back to the beginning print(f.read()) # read the whole file again # In binary mode, seek can use a reference point: # 0 = start (default), 1 = current, 2 = end with open("data.bin", "rb") as f: f.seek(-10, 2) # 10 bytes before the end tail = f.read() ``` > pathlib is the modern path tool For building, joining, and inspecting file *paths*, Python's `pathlib` module is the modern, object-oriented approach — and `Path` objects work directly with `open()`. See the dedicated reference: [pathlib in the Standard Library](/coding/languages/python/standard-library/pathlib/). ```python from pathlib import Path # Path objects have convenient read/write shortcuts p = Path("notes.txt") text = p.read_text(encoding="utf-8") # read whole file p.write_text("new content", encoding="utf-8") # write whole file data = Path("photo.jpg").read_bytes() # read binary # Path objects also work directly with open() with open(Path("logs") / "today.txt", "w", encoding="utf-8") as f: f.write("entry\n") ``` **Commonly confused:** - 'w' erases the file immediately. Opening an existing file in 'w' mode truncates it to zero length the moment open() is called — before you write anything. If you want to add to a file, use 'a' (append). To avoid accidental overwrites, use 'x', which raises FileExistsError if the file already exists. - write() does not add newlines. Unlike print(), the file write() and writelines() methods write exactly what you give them. You must include \n yourself where you want line breaks. ✅ Intermediate tab complete — check your understanding - I know that text mode returns str, binary mode returns bytes, and mixing them raises TypeError - I can copy a binary file in chunks without loading it all into memory - I know why CSV files need newline="" in the open() call - I can use Path("file.txt").read_text(encoding="utf-8") as a one-liner Continue to [Modules & Packages →](/coding/languages/python/modules-and-packages/) ## EXPERT ## The io module and the three layers of file objects The objects returned by `open()` are defined in the `io` module. There is a layered hierarchy: `RawIOBase` for raw unbuffered binary I/O, `BufferedIOBase` for buffered binary I/O, and `TextIOBase` for text I/O. When you open a file in text mode, `open()` returns a `TextIOWrapper` that wraps a `BufferedReader`/`BufferedWriter`, which in turn wraps a `FileIO` raw stream. This layering is why text mode can transparently decode bytes and translate newlines. ```python import io # What open() actually returns f = open("notes.txt", "r", encoding="utf-8") print(type(f)) # f.close() f = open("notes.txt", "rb") print(type(f)) # f.close() # In-memory file objects — same interface, no disk text_stream = io.StringIO() text_stream.write("in-memory text\n") print(text_stream.getvalue()) # 'in-memory text\n' bytes_stream = io.BytesIO() bytes_stream.write(b"\x00\x01\x02") print(bytes_stream.getvalue()) # b'\x00\x01\x02' # Control buffering with the buffering argument # 0 = unbuffered (binary only), 1 = line-buffered (text), # >1 = buffer size in bytes with open("log.txt", "w", encoding="utf-8", buffering=1) as f: f.write("flushed each line\n") ``` ## Newline translation and universal newlines In text mode, Python performs *universal newline* translation by default: on reading, any of `\n`, `\r\n`, or `\r` is translated to `\n`; on writing, `\n` is translated to the platform default line separator (`os.linesep`). The `newline` argument to `open()` controls this. Setting `newline=""` disables translation — which is exactly what the `csv` module requires to handle quoted fields containing embedded newlines correctly. ```python # Default: universal newlines translate \r\n -> \n on read with open("windows_file.txt", "r", encoding="utf-8") as f: text = f.read() # \r\n becomes \n automatically # csv module requires newline="" to avoid double-translation import csv with open("data.csv", "w", encoding="utf-8", newline="") as f: writer = csv.writer(f) writer.writerow(["name", "city"]) writer.writerow(["Priya", "Mumbai"]) # Preserve original newlines exactly (no translation on read) with open("exact.txt", "r", encoding="utf-8", newline="") as f: raw = f.read() # \r\n stays \r\n ``` > Why with is non-negotiable for files Without `with` (or an explicit `try/finally`), a file may stay open if an exception occurs before `close()`. Open file handles are a limited OS resource, and buffered writes are not guaranteed to reach disk until the file is flushed or closed. The `with` statement calls the file's `__exit__`, which closes and flushes it — even on error. This is the single most important habit in Python file handling. ✅ Expert tab complete - I know the three io layers: raw binary → buffered binary → text - I can use io.StringIO to create an in-memory file object for testing - I understand what universal newline translation does and why csv needs newline="" Continue to [Modules & Packages →](/coding/languages/python/modules-and-packages/) ## SOURCES - Python Built-in Functions — open(). docs.python.org/3/library/functions.html#open. - Python Standard Library — io — Core tools for working with streams. docs.python.org/3/library/io.html. - Python Tutorial §7.2 — Reading and Writing Files. docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files. - Python Standard Library — pathlib. docs.python.org/3/library/pathlib.html. - Python Standard Library — csv (newline handling). docs.python.org/3/library/csv.html. **Primary source:** Python Built-in Functions · open() *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Modules & Packages in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/modules-and-packages/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-16 *How Python's import system works — modules, packages, __init__.py, the module search path, pip, and virtual environments for managing dependencies.* ## CANONICAL DEFINITION A module is a single file of Python code that can be imported; a package is a directory of modules identified to Python as importable. The import system, defined in the Python Language Reference, locates modules via sys.path, executes them once, and binds their names into the importing namespace. ## BEGINNER > The idea A **module** is just a `.py` file. When you `import` it, you get access to the functions, classes, and variables it defines. A **package** is a folder of modules. This is how Python code is organised and shared — your own code, the standard library, and third-party libraries all work the same way. ## The import statement There are a few forms of `import`. Each one runs the target module once (the first time) and binds names so you can use them. The Python Tutorial describes a module as a file containing Python definitions and statements, with the file name being the module name plus `.py`. ```python # Import the whole module — access names with module.name import math print(math.sqrt(16)) # 4.0 print(math.pi) # 3.141592653589793 # Import specific names from a module from math import sqrt, pi print(sqrt(16)) # 4.0 — no 'math.' prefix needed # Import with an alias (very common for long names) import datetime as dt now = dt.datetime.now() # Import a name with an alias from collections import OrderedDict as OD # Importing your own module — a file called helpers.py # helpers.py defines: def greet(name): return f"Hi {name}" import helpers print(helpers.greet("Priya")) # Hi Priya from helpers import greet print(greet("Arjun")) # Hi Arjun ``` > Avoid from module import * The form `from module import *` imports every public name into your namespace. It is discouraged because it makes it unclear where names came from and can silently overwrite existing names. Import only what you need, or import the module and use its prefix. ## Packages and __init__.py A package is a directory containing modules. Historically, a directory became a package by including an `__init__.py` file (which may be empty). That file runs when the package is first imported and can expose a curated public API. The Python Tutorial explains that the `__init__.py` files are required to make Python treat directories containing the file as packages (for regular packages). ```python myproject/ main.py store/ <- a package (a directory) __init__.py <- marks 'store' as a package cart.py <- a module: store.cart payment.py <- a module: store.payment utils/ <- a sub-package __init__.py format.py <- store.utils.format ``` ```python # From main.py — import from the package import store.cart store.cart.add_item("book") from store import payment payment.charge(500) from store.utils.format import to_currency print(to_currency(500)) # ₹500.00 # __init__.py can re-export names for a clean public API. # If store/__init__.py contains: from .cart import add_item # then users can simply write: from store import add_item ``` ✅ Beginner tab complete — check your understanding - I can import a module and use its functions with module.function() syntax - I can use "from module import name" to import specific names - I understand what a package is (a directory of modules with __init__.py) - I can create a virtual environment with python -m venv .venv and install packages with pip Continue to [Type Hints →](/coding/languages/python/type-hints/) ## INTERMEDIATE ## The module search path (sys.path) When you import a module, Python searches a list of locations in order, stored in `sys.path`. The Python Language Reference specifies that the search begins with built-in modules, then proceeds through the directories in `sys.path`: the directory of the input script (or the current directory), then `PYTHONPATH` environment variable directories, then installation-dependent defaults (including site-packages where third-party libraries live). ```python import sys # See exactly where Python looks for modules, in order for path in sys.path: print(path) # sys.modules is a cache of already-imported modules. # A module is executed only ONCE; later imports reuse the cache. import math print("math" in sys.modules) # True # The name of the current module print(__name__) # In the file you run directly, __name__ == "__main__" # In an imported module, __name__ is the module's name ``` > The if __name__ == "__main__" guard Because an imported module's code runs on import, you wrap "run this only when executed directly" code in a guard. When the file is imported, `__name__` is the module name; when run directly, it is `"__main__"`. ```python # calculator.py def add(a, b): return a + b def main(): print(add(2, 3)) # This block runs ONLY when the file is executed directly: # python calculator.py -> runs main() # import calculator -> does NOT run main() if __name__ == "__main__": main() ``` ## Virtual environments and pip Third-party packages are installed with `pip`, the package installer for Python, from the Python Package Index (PyPI). A *virtual environment* is an isolated Python environment with its own set of installed packages, created with the standard-library `venv` module. The Python documentation recommends using a virtual environment per project so dependencies do not conflict between projects. ```python # Create a virtual environment named .venv python -m venv .venv # Activate it source .venv/bin/activate # macOS / Linux .venv\Scripts\activate # Windows # Now pip installs into THIS environment only pip install requests pip install "django>=5.0" # List installed packages pip list # Save the exact dependency versions to a file pip freeze > requirements.txt # Recreate the environment elsewhere from that file pip install -r requirements.txt # Leave the virtual environment deactivate ``` **Commonly confused:** - Module vs package. A module is one .py file. A package is a directory of modules (traditionally with an __init__.py). Both are imported with import — the difference is structure, not syntax. - import runs the module once. The first import executes the module top to bottom; subsequent imports reuse the cached module object from sys.modules. Top-level side effects (like print statements) happen only on that first import. ✅ Intermediate tab complete — check your understanding - I know what sys.path is and in what order Python searches it - I understand that modules are executed once and then cached in sys.modules - I know that if __name__ == "__main__" runs only when the file is executed directly, not when imported - I can use relative imports (from .module import name) within a package Continue to [Type Hints →](/coding/languages/python/type-hints/) ## EXPERT ## Absolute vs relative imports inside packages Within a package, modules can import sibling modules using *absolute* imports (the full path from the project root) or *relative* imports (using leading dots to indicate the current and parent packages). PEP 328 introduced explicit relative imports. The Python documentation recommends absolute imports for clarity, but relative imports are useful within large packages. A single dot means the current package; two dots means the parent package. ```python # Inside store/cart.py # Absolute import — full path from the project root (preferred) from store.payment import charge from store.utils.format import to_currency # Relative import — dots indicate current/parent package from .payment import charge # . = current package (store) from .utils.format import to_currency from ..config import SETTINGS # .. = parent package # Relative imports only work inside a package being imported, # NOT in a module run directly as a script. Running a file with # relative imports directly raises: # ImportError: attempted relative import with no known parent package # Run it as a module instead: python -m store.cart ``` ## The import system internals and importlib The import statement is implemented by `importlib`. The machinery has two stages: *finders* locate a module and return a *loader*; the loader executes the module and populates its namespace. The result is cached in `sys.modules`. This system is extensible — you can import from zip files, URLs, or custom sources by registering finders on `sys.meta_path`. The Python Language Reference documents this as the import protocol. ```python import importlib # Import a module whose name is only known at runtime (a string) module_name = "math" math = importlib.import_module(module_name) print(math.sqrt(25)) # 5.0 # Reload a module that has changed (rarely needed, useful in REPL) import mymodule importlib.reload(mymodule) # Check if a module is available without importing it spec = importlib.util.find_spec("numpy") if spec is not None: import numpy else: print("numpy not installed") ``` > Namespace packages (PEP 420) Since Python 3.3 (PEP 420), a directory *without* an `__init__.py` can still be a "namespace package" — a package whose contents may be split across multiple directories on `sys.path`. This is an advanced feature used by large frameworks to let separate distributions contribute to the same package name. For ordinary projects, including `__init__.py` (a regular package) remains the clear, recommended choice. > Circular imports If module A imports module B while B imports A, you can hit a circular import — one module sees the other only partially initialised, often raising `ImportError` or `AttributeError`. The fixes: restructure to remove the cycle, move the import inside the function that needs it (deferring it until call time), or import the module object rather than names from it. ✅ Expert tab complete - I can use importlib.import_module() to import a module whose name is only known at runtime - I know the difference between a regular package (__init__.py) and a namespace package (no __init__.py) - I can diagnose and fix a circular import by restructuring or deferring the import Continue to [Type Hints →](/coding/languages/python/type-hints/) ## SOURCES - Python Language Reference §5 — The import system. docs.python.org/3/reference/import.html. - Python Tutorial §6 — Modules (including packages and __init__.py). docs.python.org/3/tutorial/modules.html. - Python Standard Library — venv — Creation of virtual environments. docs.python.org/3/library/venv.html. - PEP 328 — Imports: Multi-Line and Absolute/Relative; PEP 420 — Implicit Namespace Packages. peps.python.org. - Python Standard Library — importlib. docs.python.org/3/library/importlib.html. **Primary source:** Python Language Reference §5 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Type Hints in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/type-hints/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-16 *Optional static typing for Python — annotations, the typing module, Optional and Union, generics, Protocols, and static checking with tools like mypy.* ## CANONICAL DEFINITION Type hints are optional annotations, introduced by PEP 484, that declare the expected types of variables, function parameters, and return values. They do not affect runtime behaviour; they are used by static type checkers, IDEs, and documentation tools to catch type errors before the program runs. ## BEGINNER > The idea Type hints let you write down what type a value is *supposed* to be — a string, an integer, a list of numbers. Python ignores them at runtime, but editors and tools like **mypy** read them to catch mistakes (like passing a string where a number is expected) *before* you run the code. ## Basic annotations You annotate a function's parameters and return type with colons and an arrow. You can also annotate variables. PEP 484 defines the function syntax; PEP 526 defines variable annotations. Crucially, these are hints only — Python does not enforce them. ```python # Function annotations: param: type, and -> return_type def greet(name: str) -> str: return f"Hello, {name}" def add(a: int, b: int) -> int: return a + b # A function that returns nothing uses -> None def log(message: str) -> None: print(message) # Variable annotations (PEP 526) age: int = 30 name: str = "Priya" price: float = 99.99 is_active: bool = True # Annotations do NOT enforce types at runtime: def double(n: int) -> int: return n * 2 print(double("ab")) # 'abab' — Python runs it anyway! # A type checker (mypy) would flag this as an error before running. ``` ## Built-in container types Since Python 3.9 (PEP 585), you can use the built-in collection types directly as generic hints — `list[int]`, `dict[str, int]`, `tuple[int, ...]` — without importing from `typing`. Before 3.9 you used capitalised versions (`List`, `Dict`) from the `typing` module. ```python # Python 3.9+ — use built-in types directly (PEP 585) def total(prices: list[float]) -> float: return sum(prices) def lookup(table: dict[str, int], key: str) -> int: return table[key] # A tuple of fixed types point: tuple[int, int] = (3, 5) # A tuple of variable length, all the same type scores: tuple[int, ...] = (90, 85, 78, 92) # Nested containers matrix: list[list[int]] = [[1, 2], [3, 4]] records: dict[str, list[str]] = {"fruits": ["apple", "mango"]} # Pre-3.9 equivalent (still valid, needs imports) from typing import List, Dict def total_old(prices: List[float]) -> float: return sum(prices) ``` ✅ Beginner tab complete — check your understanding - I can annotate a function: def greet(name: str, age: int) -> str - I can annotate a variable: count: int = 0 - I know that type hints are not enforced at runtime — they're documentation + static analysis - I know that Optional[str] means "str or None" Continue to [Concurrency →](/coding/languages/python/concurrency/) ## INTERMEDIATE ## Optional, Union, and the | operator When a value can be one of several types, use a union. PEP 604 (Python 3.10+) added the `X | Y` syntax as a cleaner alternative to `typing.Union[X, Y]`. `Optional[X]` means "X or None" and is equivalent to `X | None`. ```python # Python 3.10+ — the | union operator (PEP 604) def parse(text: str) -> int | float: if "." in text: return float(text) return int(text) # Optional[X] means "X or None" — extremely common for defaults def find_user(user_id: int) -> str | None: users = {1: "Priya", 2: "Arjun"} return users.get(user_id) # returns str or None result = find_user(99) if result is not None: print(result.upper()) # safe — checked for None first # Pre-3.10 equivalents from the typing module from typing import Optional, Union def parse_old(text: str) -> Union[int, float]: ... def find_old(uid: int) -> Optional[str]: ... # = Union[str, None] # Other common typing constructs from typing import Any, Callable handler: Callable[[int, str], bool] # takes (int, str), returns bool anything: Any = "could be literally anything" # opt out of checking ``` ## Generics and TypeVar A generic function or class works with multiple types while preserving the relationship between them. A `TypeVar` is a type variable — a placeholder that a checker fills in. Python 3.12 (PEP 695) added cleaner built-in syntax for declaring type parameters. ```python from typing import TypeVar T = TypeVar("T") # This function returns the SAME type it receives def first(items: list[T]) -> T: return items[0] n: int = first([1, 2, 3]) # checker knows this is int s: str = first(["a", "b"]) # checker knows this is str # A generic class from typing import Generic class Box(Generic[T]): def __init__(self, content: T) -> None: self.content = content def get(self) -> T: return self.content int_box: Box[int] = Box(42) print(int_box.get()) # checker knows this is int # Python 3.12+ syntax (PEP 695) — no TypeVar import needed def first_new[T](items: list[T]) -> T: return items[0] class Box2[T]: def __init__(self, content: T) -> None: self.content = content ``` **Commonly confused:** - Type hints are not enforced at runtime. Python does not check them when the program runs — passing the wrong type does not raise an error from the annotation itself. Hints are for static checkers (mypy, pyright), editors, and humans. To validate types at runtime you need explicit checks or a library like pydantic. - Optional[X] means "X or None", not "optional argument". It is purely about the value possibly being None. Whether an argument has a default value is a separate matter. Optional[int] is exactly int | None. ✅ Intermediate tab complete — check your understanding - I know that int | None (3.10+) and Optional[int] mean the same thing - I can write list[int] instead of List[int] for container type hints (Python 3.9+) - I understand what TypeVar is for: preserving type relationships between input and output - I have run mypy on at least one file and fixed the type errors it found Continue to [Concurrency →](/coding/languages/python/concurrency/) ## EXPERT ## Protocols and structural typing (PEP 544) PEP 544 introduced `Protocol`, which brings *structural* subtyping (often called "duck typing" made static) to Python's type system. A class satisfies a Protocol if it has the required methods and attributes — it does not need to explicitly inherit from the Protocol. This matches how Python code actually works: it cares whether an object *behaves* the right way, not what it inherits from. ```python from typing import Protocol # Define behaviour, not a base class class Drawable(Protocol): def draw(self) -> str: ... # Circle does NOT inherit from Drawable — but it matches the shape class Circle: def draw(self) -> str: return "drawing a circle" class Square: def draw(self) -> str: return "drawing a square" # A function accepting anything that has a draw() method def render(shape: Drawable) -> None: print(shape.draw()) render(Circle()) # accepted — Circle structurally matches Drawable render(Square()) # accepted — same reason # @runtime_checkable allows isinstance() checks against the protocol from typing import runtime_checkable @runtime_checkable class Sized(Protocol): def __len__(self) -> int: ... print(isinstance([1, 2, 3], Sized)) # True — lists have __len__ ``` ## Static checking with mypy, and runtime introspection Type hints are checked by separate tools, not the interpreter. `mypy` is the reference type checker (others include pyright and pyre). You run it over your code, and it reports type errors without executing the program. Although hints are not enforced at runtime, they are stored in the `__annotations__` attribute and can be inspected — frameworks like dataclasses and pydantic use this. ```python # Install the reference checker pip install mypy # Check a file or a whole project mypy myscript.py mypy src/ # Example output when a type is wrong: # error: Argument 1 to "double" has incompatible type "str"; # expected "int" ``` ```python # Annotations are stored and inspectable at runtime def add(a: int, b: int) -> int: return a + b print(add.__annotations__) # {'a': , 'b': , 'return': } # typing.get_type_hints resolves string annotations properly from typing import get_type_hints print(get_type_hints(add)) # dataclasses use annotations to generate __init__ from fields from dataclasses import dataclass @dataclass class User: name: str # these annotations drive the generated __init__ age: int active: bool = True u = User("Priya", 30) print(u) # User(name='Priya', age=30, active=True) ``` > Forward references and from __future__ import annotations If an annotation references a name not yet defined (such as a class referring to itself), wrap it in quotes as a string forward reference, or add `from __future__ import annotations` at the top of the file. That future import (PEP 563) makes all annotations be stored as strings and evaluated lazily, which sidesteps definition-order problems and slightly improves import time. ✅ Expert tab complete - I can write a Protocol class that defines an interface without inheritance - I know that Protocol uses structural typing (duck typing made static) - I can use get_type_hints() to introspect annotations at runtime Continue to [Concurrency →](/coding/languages/python/concurrency/) ## SOURCES - PEP 484 — Type Hints. peps.python.org/pep-0484/. - PEP 526 — Syntax for Variable Annotations; PEP 585 — Type Hinting Generics In Standard Collections. peps.python.org. - PEP 604 — Allow writing union types as X | Y; PEP 695 — Type Parameter Syntax. peps.python.org. - PEP 544 — Protocols: Structural subtyping. peps.python.org/pep-0544/. - Python Standard Library — typing. docs.python.org/3/library/typing.html. mypy documentation: mypy.readthedocs.io. **Primary source:** PEP 484 — Type Hints *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Concurrency in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/concurrency/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-16 *Python's three concurrency models — threading, multiprocessing, and asyncio — when to use each, the role of the GIL, and the unified concurrent.futures interface.* ## CANONICAL DEFINITION Python offers three concurrency approaches: threading for I/O-bound work using OS threads constrained by the Global Interpreter Lock, multiprocessing for CPU-bound work using separate processes that bypass the GIL, and asyncio for high-volume I/O-bound work using a single-threaded cooperative event loop. ## BEGINNER > The idea Concurrency means doing more than one thing at a time. Python gives you three tools, and choosing the right one depends on what is slowing you down: **waiting** (network, disk, database) or **computing** (heavy number-crunching). The wrong choice can make things slower, so the distinction matters. ## Which model to use | Model | Best for | How it runs | | --- | --- | --- | | `threading` | I/O-bound (waiting on network, disk) | Multiple OS threads, one at a time due to the GIL | | `multiprocessing` | CPU-bound (heavy computation) | Multiple processes, truly parallel | | `asyncio` | I/O-bound at high volume (thousands of connections) | One thread, cooperative event loop | > I/O-bound vs CPU-bound — the key question **I/O-bound:** your program spends most of its time *waiting* — for a web response, a file, a database. Threads and asyncio help because while one task waits, another runs. **CPU-bound:** your program spends its time *calculating* — image processing, maths. Only multiprocessing gives a real speed-up here, because it sidesteps the GIL. ## Threading — for I/O-bound work A thread runs a function alongside others in the same process. Because of the GIL (explained in the Expert tab), threads do not speed up pure computation — but they are excellent when tasks spend time waiting, because Python releases the GIL during I/O. ```python import threading import time def download(name: str) -> None: print(f"Starting {name}") time.sleep(2) # simulates waiting for a network response print(f"Finished {name}") # Run three "downloads" concurrently threads = [] for site in ["site-a", "site-b", "site-c"]: t = threading.Thread(target=download, args=(site,)) t.start() # begins running the function threads.append(t) # Wait for all threads to complete for t in threads: t.join() # Without threads this takes ~6s (2s x 3). # With threads it takes ~2s — they wait at the same time. print("All downloads done") ``` ✅ Beginner tab complete — check your understanding - I can explain the difference between I/O-bound and CPU-bound work - I know: I/O-bound → threading or asyncio; CPU-bound → multiprocessing - I can write a basic threading.Thread example - I can write a basic asyncio coroutine with async/await - I understand why threads do NOT speed up CPU-heavy Python code Continue to [Testing →](/coding/languages/python/testing/) ## INTERMEDIATE ## Multiprocessing — for CPU-bound work The `multiprocessing` module creates separate Python processes, each with its own interpreter and memory — and crucially, its own GIL. This achieves true parallelism on multiple CPU cores for computation-heavy work. The trade-off: starting processes is more expensive than threads, and data must be sent between them (pickled), so it suits coarse-grained, CPU-bound tasks. ```python from multiprocessing import Pool def heavy_compute(n: int) -> int: # A CPU-bound task — pure calculation return sum(i * i for i in range(n)) if __name__ == "__main__": # required guard for multiprocessing numbers = [1_000_000, 2_000_000, 3_000_000, 4_000_000] # Pool distributes the work across CPU cores, truly in parallel with Pool(processes=4) as pool: results = pool.map(heavy_compute, numbers) print(results) # On a 4-core machine this runs ~4x faster than a plain loop, # because each process runs on its own core with its own GIL. ``` > The if __name__ == "__main__" guard is required On platforms that start processes via "spawn" (Windows, and macOS by default), the child process re-imports your script. Without the main-guard, that would recursively start more processes. Always wrap multiprocessing entry code in `if __name__ == "__main__":`. ## asyncio — single-threaded concurrency `asyncio` runs many tasks on one thread using an event loop and the `async`/`await` keywords. When a coroutine hits an `await` on something that would wait (a network call), it yields control so another coroutine can run. This scales to thousands of simultaneous connections with very little overhead — ideal for network servers and clients. The Codex has a dedicated deep-dive: [asyncio in the Standard Library](/coding/languages/python/standard-library/asyncio/). ```python import asyncio async def fetch(name: str) -> str: print(f"Starting {name}") await asyncio.sleep(2) # yields control while "waiting" print(f"Finished {name}") return name async def main() -> None: # Run three coroutines concurrently and wait for all results = await asyncio.gather( fetch("api-1"), fetch("api-2"), fetch("api-3"), ) print(results) # ['api-1', 'api-2', 'api-3'] # asyncio.run starts the event loop and runs main() to completion asyncio.run(main()) # Total time ~2s, not 6s — all three wait concurrently on one thread. ``` **Commonly confused:** - Threads do not speed up CPU-bound Python. Because of the GIL, only one thread executes Python bytecode at a time. For heavy computation, threading gives little or no speed-up — use multiprocessing instead. Threads shine only when tasks wait. - asyncio is concurrency, not parallelism. It runs on a single thread. It interleaves many waiting tasks extremely efficiently, but it does not use multiple CPU cores. For parallel computation you still need multiprocessing. ✅ Intermediate tab complete — check your understanding - I can use ThreadPoolExecutor for I/O-bound work and ProcessPoolExecutor for CPU-bound work - I can use pool.map() to apply a function to a list in parallel - I can explain what the GIL is and why Python threads don't run Python bytecode in parallel - I always include if __name__ == "__main__": when using multiprocessing Continue to [Testing →](/coding/languages/python/testing/) ## EXPERT ## The Global Interpreter Lock (GIL) In CPython, the Global Interpreter Lock is a mutex that allows only one thread to execute Python bytecode at a time. It exists because CPython's memory management (reference counting) is not thread-safe; the GIL makes the interpreter simpler and single-threaded code faster. The consequence: threads cannot run Python bytecode in parallel, so CPU-bound multithreaded code sees no speed-up. The GIL *is* released during blocking I/O and inside many C extensions (such as NumPy operations), which is why threading still helps I/O-bound and C-heavy workloads. > PEP 703 — the free-threaded build PEP 703 introduced an experimental free-threaded build of CPython (available from Python 3.13 as a separate build) that can run without the GIL, enabling true multithreaded parallelism. As of 3.13 it is optional and experimental; the standard build still uses the GIL. This is the most significant change to Python's concurrency story in its history, but production code today should still assume the GIL is present. ## concurrent.futures — the unified high-level interface The `concurrent.futures` module provides a single high-level API over both threads and processes via `ThreadPoolExecutor` and `ProcessPoolExecutor`. You submit callables and get back `Future` objects representing pending results. Switching between threads and processes is often a one-line change — making it the recommended starting point for most parallel work. ```python from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor def fetch_url(url: str) -> int: import time time.sleep(1) # simulate an I/O wait return len(url) # I/O-bound -> ThreadPoolExecutor urls = ["a.com", "b.com", "c.com", "d.com"] with ThreadPoolExecutor(max_workers=4) as pool: results = list(pool.map(fetch_url, urls)) print(results) # CPU-bound -> ProcessPoolExecutor (same API, different executor) def square(n: int) -> int: return n * n if __name__ == "__main__": with ProcessPoolExecutor() as pool: results = list(pool.map(square, range(10))) print(results) # submit() returns a Future; as_completed yields them as they finish from concurrent.futures import as_completed with ThreadPoolExecutor() as pool: futures = [pool.submit(fetch_url, u) for u in urls] for future in as_completed(futures): print("Got result:", future.result()) ``` > A decision summary Start with `concurrent.futures`. Use `ThreadPoolExecutor` for I/O-bound tasks, `ProcessPoolExecutor` for CPU-bound tasks. Reach for `asyncio` when you have very high volumes of concurrent I/O (thousands of network operations) and libraries that support it. Use raw `threading` or `multiprocessing` only when you need fine-grained control the executors do not provide. How this connects Goes deeper [asyncio (Standard Library)](/coding/languages/python/standard-library/asyncio/) Related concept [Concurrency concept](/coding/concepts/concurrency/) Primary source [Concurrent Execution — docs.python.org](https://docs.python.org/3/library/concurrency.html) ✅ Expert tab complete - I can explain why CPython needs the GIL (reference counting is not thread-safe) - I know the GIL IS released during blocking I/O and inside many C extensions - I know that PEP 703 introduces an optional free-threaded CPython build in Python 3.13 Continue to [Testing →](/coding/languages/python/testing/) ## SOURCES - Python Standard Library — Concurrent Execution. docs.python.org/3/library/concurrency.html. - Python Standard Library — threading, multiprocessing, asyncio. docs.python.org/3/library/. - Python Standard Library — concurrent.futures. docs.python.org/3/library/concurrent.futures.html. - Python Language Reference & CPython docs — Global Interpreter Lock. docs.python.org/3/glossary.html#term-global-interpreter-lock. - PEP 703 — Making the Global Interpreter Lock Optional in CPython. peps.python.org/pep-0703/. **Primary source:** Python Standard Library · concurrency *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Testing in Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/testing/ **Type:** language-concept | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-16 *Writing automated tests in Python — the built-in unittest framework, the popular pytest library, fixtures, parametrization, and mocking with unittest.mock.* ## CANONICAL DEFINITION Testing in Python is the practice of writing code that automatically verifies other code behaves correctly. The standard library provides unittest (an xUnit-style framework) and unittest.mock (for replacing dependencies); the third-party pytest library is the most widely used test runner, offering a lighter syntax. ## BEGINNER > The idea A test is code that checks your other code does what you expect. Instead of manually running your program and eyeballing the output, you write assertions — "this function should return 5 for these inputs" — and a test runner checks them all automatically. When you change code later, the tests tell you instantly if you broke something. ## unittest — the built-in framework Python ships with `unittest`, an xUnit-style testing framework (modelled on JUnit). You write test classes that inherit from `unittest.TestCase` and methods that begin with `test_`. Inside, you use assertion methods like `assertEqual` to state what should be true. ```python import unittest # The code we want to test def add(a, b): return a + b def divide(a, b): if b == 0: raise ValueError("Cannot divide by zero") return a / b # A test case — a class inheriting from TestCase class TestMath(unittest.TestCase): def test_add_positive(self): self.assertEqual(add(2, 3), 5) def test_add_negative(self): self.assertEqual(add(-1, -1), -2) def test_divide(self): self.assertEqual(divide(10, 2), 5.0) def test_divide_by_zero_raises(self): # assertRaises checks that an exception is raised with self.assertRaises(ValueError): divide(10, 0) if __name__ == "__main__": unittest.main() # run all tests when executed directly # Run from the terminal: # python test_with_unittest.py # python -m unittest (auto-discovers test files) ``` | Assertion | Checks | | --- | --- | | `assertEqual(a, b)` | `a == b` | | `assertTrue(x)` / `assertFalse(x)` | `x` is truthy / falsy | | `assertIs(a, b)` / `assertIsNone(x)` | identity / `x is None` | | `assertIn(a, b)` | `a in b` | | `assertRaises(Error)` | the block raises `Error` | | `assertAlmostEqual(a, b)` | floats equal to N decimal places | ✅ Beginner tab complete — check your understanding - I can write a TestCase class with at least 3 test methods - I can use assertEqual, assertTrue, assertIn, and assertRaises - I can run tests with python -m unittest and understand the output - I know that each test should test one thing and be named descriptively Continue to Stage 3: [os — Operating System Interface →](/coding/languages/python/standard-library/os/) ## INTERMEDIATE ## pytest — the popular alternative `pytest` is a third-party framework (installed with `pip install pytest`) and the most widely used Python test runner. Its appeal is simplicity: tests are plain functions, and you use Python's built-in `assert` statement rather than special methods. pytest rewrites assertions to give detailed failure messages. It also runs existing `unittest` test cases without changes. ```python import pytest def add(a, b): return a + b def divide(a, b): if b == 0: raise ValueError("Cannot divide by zero") return a / b # Just functions named test_* — no class needed def test_add_positive(): assert add(2, 3) == 5 # plain assert statement def test_add_negative(): assert add(-1, -1) == -2 def test_divide(): assert divide(10, 2) == 5.0 # Checking exceptions with pytest.raises def test_divide_by_zero_raises(): with pytest.raises(ValueError): divide(10, 0) # Run from the terminal: # pytest (discovers all test_*.py files) # pytest test_file.py (one file) # pytest -v (verbose, shows each test) # pytest -k "divide" (only tests matching 'divide') ``` ## Fixtures and parametrization A *fixture* provides a fixed baseline for tests — shared setup like a database connection or sample data. In pytest, fixtures are functions decorated with `@pytest.fixture` that tests receive as arguments. *Parametrization* runs the same test against many inputs without duplication. ```python import pytest # A fixture supplies reusable test data or resources @pytest.fixture def sample_users(): return [{"name": "Priya"}, {"name": "Arjun"}] # Tests request the fixture by naming it as a parameter def test_user_count(sample_users): assert len(sample_users) == 2 def test_first_user(sample_users): assert sample_users[0]["name"] == "Priya" # Parametrize — run one test with many input/expected pairs @pytest.mark.parametrize("a, b, expected", [ (2, 3, 5), (0, 0, 0), (-1, 1, 0), (100, 200, 300), ]) def test_add(a, b, expected): assert a + b == expected # This produces 4 separate test results, one per row. # unittest equivalent of fixtures: setUp and tearDown methods import unittest class TestWithSetup(unittest.TestCase): def setUp(self): # runs before EACH test self.data = [1, 2, 3] def tearDown(self): # runs after EACH test self.data = None def test_length(self): self.assertEqual(len(self.data), 3) ``` **Commonly confused:** - unittest vs pytest. unittest is built in and uses classes with self.assertEqual methods. pytest is a third-party install that uses plain functions and the bare assert statement. pytest can run unittest tests, so many projects write new tests in pytest style while keeping old unittest ones. ✅ Intermediate tab complete — check your understanding - I can write pytest tests as plain functions (no class, no self) - I can write a @pytest.fixture that provides test data - I can use @pytest.mark.parametrize to run one test with 5 different inputs - I can run pytest -v to see verbose output and pytest -k "pattern" to run matching tests Continue to Stage 3: [os — Operating System Interface →](/coding/languages/python/standard-library/os/) ## EXPERT ## Mocking with unittest.mock Tests should be fast and deterministic, so you replace slow or unpredictable dependencies (network calls, databases, the current time) with controlled stand-ins. The standard library's `unittest.mock` provides `Mock` and `MagicMock` objects and the `patch` tool, which temporarily replaces an object during a test. `patch` can be used as a decorator or a context manager. ```python from unittest.mock import Mock, patch # The code under test calls an external service import requests def get_user_name(user_id): response = requests.get(f"https://api.example.com/users/{user_id}") return response.json()["name"] # Test WITHOUT hitting the real network — patch requests.get @patch("requests.get") def test_get_user_name(mock_get): # Configure the fake response mock_get.return_value.json.return_value = {"name": "Priya"} result = get_user_name(42) assert result == "Priya" # Verify the call was made as expected mock_get.assert_called_once_with("https://api.example.com/users/42") # patch as a context manager (scoped to the with block) def test_with_context_manager(): with patch("requests.get") as mock_get: mock_get.return_value.json.return_value = {"name": "Arjun"} assert get_user_name(7) == "Arjun" # A standalone Mock records how it was used mock = Mock() mock.process(1, 2, key="value") mock.process.assert_called_once_with(1, 2, key="value") print(mock.process.call_count) # 1 ``` > Patch where it is used, not where it is defined A common mistake: when you `patch`, you must target the name in the module that *uses* it, not where it is originally defined. If `mymodule` does `from requests import get`, you patch `"mymodule.get"`, not `"requests.get"`. The mock replaces the reference in the namespace where the lookup happens. ## Test organisation, coverage, and the testing pyramid The widely cited testing pyramid suggests many fast *unit* tests (testing one function in isolation), fewer *integration* tests (testing components together), and a small number of *end-to-end* tests. Code coverage tools such as `coverage.py` (often run as `pytest --cov`) measure which lines your tests exercise — useful as a guide, though high coverage alone does not guarantee good tests. The Codex covers the broader strategy in [CI/CD](/coding/devops/#cicd), where tests run automatically on every push. ```python # Install pytest and the coverage plugin pip install pytest pytest-cov # Discover and run every test in the project pytest # Show which lines are covered by tests pytest --cov=myapp # Standard library discovery (no install needed) python -m unittest discover # Conventional project layout pytest discovers automatically: # myproject/ # myapp/ # __init__.py # calculator.py # tests/ # test_calculator.py ``` > What makes a good test A good test is fast, isolated (does not depend on other tests or external state), deterministic (same result every run), and focused on one behaviour. Tests double as living documentation: a well-named test like `test_divide_by_zero_raises` tells the next reader exactly what the code guarantees. Write the test name to describe the behaviour, not the implementation. ✅ Expert tab complete - I can use @patch to replace a function or class with a Mock during a test - I know the "patch where it is used" rule — patch the name in the module that uses it, not where it's defined - I can use mock_obj.assert_called_once_with() to verify a mock was called correctly - I understand the testing pyramid: many unit tests, fewer integration tests, few E2E tests Continue to Stage 3: [os — Operating System Interface →](/coding/languages/python/standard-library/os/) ## SOURCES - Python Standard Library — unittest — Unit testing framework. docs.python.org/3/library/unittest.html. - Python Standard Library — unittest.mock — mock object library. docs.python.org/3/library/unittest.mock.html. - pytest documentation. docs.pytest.org. - coverage.py documentation. coverage.readthedocs.io. - Python Standard Library — unittest setUp/tearDown and test discovery. docs.python.org/3/library/unittest.html#organizing-test-code. **Primary source:** Python Standard Library · unittest *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python os — Operating System Interface module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/os/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Create files, list directories, read environment variables — everything the OS exposes, from Python.* ## CANONICAL DEFINITION The Python os module provides a portable interface to operating system functionality — file and directory operations, environment variables, process information, and POSIX system calls — abstracting platform differences between Unix and Windows. ## BEGINNER > One sentence The `os` module is Python's bridge to the operating system — it lets you create, read, move, and delete files and directories, read environment variables, and run system commands, all from Python code. ## Working with paths ```python import os # Current working directory cwd = os.getcwd() print(cwd) # /home/priya/projects # Build paths portably (uses / on Unix, \ on Windows) config_path = os.path.join(cwd, "config", "settings.json") print(config_path) # /home/priya/projects/config/settings.json # Path components path = "/home/priya/data/report.csv" print(os.path.dirname(path)) # /home/priya/data print(os.path.basename(path)) # report.csv print(os.path.splitext(path)) # ('/home/priya/data/report', '.csv') print(os.path.exists(path)) # True or False print(os.path.isfile(path)) # True if it's a file print(os.path.isdir(path)) # True if it's a directory print(os.path.getsize(path)) # size in bytes ``` ## Creating and listing directories ```python import os # Create a single directory os.mkdir("new_folder") # Create nested directories (like mkdir -p) os.makedirs("parent/child/grandchild", exist_ok=True) # exist_ok=True: no error if already exists # List directory contents entries = os.listdir(".") print(entries) # ['file1.txt', 'folder', 'script.py', ...] # Walk entire directory tree for dirpath, dirnames, filenames in os.walk("/home/priya/projects"): for filename in filenames: full_path = os.path.join(dirpath, filename) print(full_path) # Rename / move os.rename("old_name.txt", "new_name.txt") # Delete file os.remove("old_file.txt") # Delete empty directory os.rmdir("empty_folder") ``` ## Environment variables ```python import os # Read environment variable (None if not set) home = os.environ.get("HOME") db_url = os.environ.get("DATABASE_URL", "sqlite:///default.db") # Set environment variable (only in current process) os.environ["DEBUG"] = "true" # All environment variables for key, value in os.environ.items(): print(f"{key}={value}") # Run a shell command (os.system is legacy — use subprocess instead) exit_code = os.system("ls -la") # not recommended for production ``` ✅ Beginner tab complete — check your understanding - I can use os.path.join() to build file paths cross-platform - I can use os.listdir() or os.walk() to list files in a directory - I can read and write environment variables with os.environ - I know os.path.exists(), os.path.isfile(), os.path.isdir() Continue to [pathlib →](/coding/languages/python/standard-library/pathlib/) ## INTERMEDIATE ## os.path vs. pathlib `os.path` works with plain strings. `pathlib` (Python 3.4+) uses objects that are more readable and composable. In new code, prefer `pathlib`. `os` is still needed for environment variables, process management, file permissions, and low-level OS operations that pathlib does not cover. ## os.scandir — the efficient directory lister ```python import os # os.scandir() is faster than os.listdir() — returns DirEntry objects # DirEntry has .name, .path, .is_file(), .is_dir(), .stat() with os.scandir(".") as entries: for entry in entries: if entry.is_file(): stat = entry.stat() print(f"{entry.name}: {stat.st_size} bytes") # Find all Python files recursively def find_py_files(root): for entry in os.scandir(root): if entry.is_dir(follow_symlinks=False): yield from find_py_files(entry.path) elif entry.name.endswith(".py"): yield entry.path for py_file in find_py_files("."): print(py_file) ``` ## Process and system information ```python import os # Current process print(os.getpid()) # process ID print(os.getppid()) # parent process ID (Unix) # User and group (Unix) print(os.getuid()) # user ID print(os.getgid()) # group ID # CPU count print(os.cpu_count()) # number of CPUs # File permissions os.chmod("script.py", 0o755) # rwxr-xr-x # Symbolic links os.symlink("target.txt", "link.txt") # create symlink os.readlink("link.txt") # read symlink target # Temporary files — prefer tempfile module import tempfile with tempfile.NamedTemporaryFile(delete=False, suffix=".tmp") as f: f.write(b"temporary data") print(f.name) # /tmp/tmpXXXXXX.tmp ``` **Commonly confused:** - os.path.join() ignores everything before an absolute path component. os.path.join("/home/user", "/etc/config") returns "/etc/config" — the absolute second argument discards the first. This is a common source of path bugs. Validate that dynamic path components are relative before joining. - os.system() is not the right way to run commands. It gives you only the exit code, not the output. Use the subprocess module (subprocess.run(), subprocess.check_output()) for anything serious — it gives full control over stdin/stdout/stderr, timeouts, and error handling. How this connects Related [pathlib (modern paths)](/coding/languages/python/standard-library/pathlib/)[sys](/coding/languages/python/standard-library/sys/) Primary source [docs.python.org/3/library/os.html](https://docs.python.org/3/library/os.html) ✅ Intermediate tab complete — check your understanding - I can use os.walk to process every file in a directory tree recursively - I know that os.scandir() returns DirEntry objects with cached stat info — faster than os.listdir() Continue to [pathlib →](/coding/languages/python/standard-library/pathlib/) ## EXPERT ## POSIX compliance and platform differences The `os` module exposes POSIX-defined functions on Unix systems and Win32 equivalents on Windows. Where behaviour differs, `os.name` is `'posix'` on Unix/macOS and `'nt'` on Windows. `os.sep` is `'/'` on Unix and `'\\'` on Windows — which is why you should always use `os.path.join()` or `pathlib.Path` rather than string concatenation. The `os` module documentation explicitly notes which functions are only available on certain platforms. ## inode, st_mtime, and file metadata `os.stat(path)` returns a `os.stat_result` object. Key fields: `st_size` (bytes), `st_mtime` (modification time as Unix timestamp), `st_ctime` (creation time on Windows; inode change time on Unix — not creation time), `st_ino` (inode number), `st_mode` (permission bits — interpret with `stat` module), `st_nlink` (hard link count). Note: `st_ctime` is not "creation time" on Unix — this is a common misconception documented in the Python docs. ✅ Expert tab complete - I know what an inode is and how to get file metadata with os.stat() - I understand the key platform differences between Windows and POSIX systems Continue to [pathlib →](/coding/languages/python/standard-library/pathlib/) ## SOURCES - Python Standard Library — os. docs.python.org/3/library/os.html. - POSIX.1-2017. pubs.opengroup.org. — Underlying specification for os module on Unix. - Python Standard Library — os.path. docs.python.org/3/library/os.path.html. **Primary source:** docs.python.org/3/library/os.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python pathlib — Object-Oriented Filesystem Paths module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/pathlib/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Path objects that think — join with /, read files directly, glob recursively, no more string concatenation.* ## CANONICAL DEFINITION The Python pathlib module provides Path objects that represent filesystem paths as objects — supporting path joining with the / operator, direct file reading and writing, glob matching, and portable handling of platform path differences. ## BEGINNER > One sentence `pathlib` turns file paths from plain strings into smart objects — you can join them with `/`, ask them questions (`.exists()`, `.is_file()`), read and write files directly, and never worry about `\\` vs `/` again. ## Path basics ```python from pathlib import Path # Create a Path object p = Path("/home/priya/data/report.csv") home = Path.home() # current user's home directory cwd = Path.cwd() # current working directory # Join paths with / operator config = home / "projects" / "myapp" / "config.json" print(config) # /home/priya/projects/myapp/config.json # Path components print(p.name) # report.csv print(p.stem) # report print(p.suffix) # .csv print(p.parent) # /home/priya/data print(p.parts) # ('/', 'home', 'priya', 'data', 'report.csv') # Test what the path is print(p.exists()) # True/False print(p.is_file()) # True if file print(p.is_dir()) # True if directory ``` ## Reading and writing files ```python from pathlib import Path # Read entire file as string text = Path("readme.md").read_text(encoding="utf-8") # Read as bytes data = Path("image.png").read_bytes() # Write text (creates file, overwrites if exists) Path("output.txt").write_text("Hello, World!\n", encoding="utf-8") # Write bytes Path("data.bin").write_bytes(b"\x00\x01\x02") # For line-by-line or large files, use open() with Path("large.csv").open("r", encoding="utf-8") as f: for line in f: process(line.strip()) ``` ## Listing and searching ```python from pathlib import Path base = Path("/home/priya/projects") # List immediate children for item in base.iterdir(): print(item) # Glob — find files matching a pattern for py_file in base.glob("*.py"): # direct children only print(py_file) for py_file in base.rglob("*.py"): # recursive — all descendants print(py_file) # Find all CSV files modified recently import time cutoff = time.time() - 86400 # 24 hours ago recent = [p for p in base.rglob("*.csv") if p.stat().st_mtime > cutoff] # Create directories (base / "new_dir" / "subdir").mkdir(parents=True, exist_ok=True) # Rename and delete old = Path("old_name.txt") new = old.with_name("new_name.txt") # same directory, new name old.rename(new) Path("unwanted.txt").unlink(missing_ok=True) # delete file, no error if missing ``` ✅ Beginner tab complete — check your understanding - I can build paths with the / operator: Path("data") / "users" / "profile.json" - I can check existence with p.exists(), p.is_file(), p.is_dir() - I can read a file in one line: Path("file.txt").read_text(encoding="utf-8") - I can write a file in one line: Path("file.txt").write_text("content", encoding="utf-8") Continue to [json →](/coding/languages/python/standard-library/json/) ## INTERMEDIATE ## pathlib vs os.path Both do the same job. `pathlib` uses an OOP interface — a `Path` is an object with methods. `os.path` uses standalone functions on strings. Prefer `pathlib` for new code: it is more readable, composable (the `/` operator), and the Path object carries type information. Use `os` when you need environment variables, permissions, symlinks, or process-related calls that `pathlib` does not cover. `Path` objects can be passed to most functions that accept string paths — they implement `__fspath__()` (PEP 519). ```python # os.path style import os path = os.path.join(os.path.expanduser("~"), "projects", "app", "config.json") if os.path.exists(path) and os.path.isfile(path): with open(path) as f: data = f.read() # pathlib style — same thing from pathlib import Path path = Path.home() / "projects" / "app" / "config.json" if path.exists() and path.is_file(): data = path.read_text() # Converting between them import os p = Path("/home/priya/data.csv") as_string = str(p) # "/home/priya/data.csv" as_string = os.fspath(p) # same — uses __fspath__ # Path arithmetic p = Path("report.csv") json_version = p.with_suffix(".json") # report.json backup = p.with_name(f"backup_{p.name}") # backup_report.csv ``` ## Resolving and relative paths ```python from pathlib import Path # relative_to — compute relative path base = Path("/home/priya/projects") file = Path("/home/priya/projects/app/main.py") rel = file.relative_to(base) print(rel) # app/main.py # resolve — canonicalise (follows symlinks, resolves ..) p = Path("../data/../data/file.txt") print(p.resolve()) # absolute path with symlinks resolved # is_relative_to (Python 3.9+) print(file.is_relative_to(base)) # True # samefile — same file even through symlinks p1 = Path("/tmp/link") p2 = Path("/home/priya/original") print(p1.samefile(p2)) # True if they point to same inode ``` **Commonly confused:** - Path("~") does not expand the tilde. Path("~/documents") gives a path starting with literal ~. Use Path("~/documents").expanduser() or Path.home() / "documents" to get the actual home directory. - path.unlink() only removes files, not directories. To remove a directory, use path.rmdir() (empty only) or shutil.rmtree(path) (with contents). unlink() on a directory raises IsADirectoryError. How this connects Related [os](/coding/languages/python/standard-library/os/)[Error Handling](/coding/languages/python/error-handling/) Primary source [docs.python.org/3/library/pathlib.html](https://docs.python.org/3/library/pathlib.html) ✅ Intermediate tab complete — check your understanding - I can list all Python files recursively with list(Path(".").rglob("*.py")) - I can use glob() for non-recursive search with wildcards - I can use resolve() to get the absolute path and follow symlinks - I can compute a relative path with path.relative_to(base) Continue to [json →](/coding/languages/python/standard-library/json/) ## EXPERT ## PEP 428 and the Path hierarchy PEP 428 (Python 3.4, 2013) introduced `pathlib`. The class hierarchy: `PurePath` (no I/O, just path manipulation) → `PurePosixPath` / `PureWindowsPath`; `Path(PurePath)` (adds I/O) → `PosixPath` / `WindowsPath`. `Path()` automatically instantiates the platform-appropriate concrete class. `PurePath` objects are useful for path manipulation in cross-platform code (e.g. building Windows paths on a Linux machine) without requiring the path to exist. ## PEP 519 — os.fspath() and __fspath__ PEP 519 (Python 3.6, 2016) defined the "file system path protocol." Any object implementing `__fspath__()` that returns a `str` or `bytes` is a "path-like object." `os.fspath(p)` calls `p.__fspath__()`. All Python stdlib functions accepting file paths accept path-like objects. This is why `open(Path("file.txt"))` works. ✅ Expert tab complete - I know pathlib provides both PurePath (no I/O) and Path (with I/O) classes - I can use os.fspath() or str() to convert a Path to a string for APIs that don't accept Path objects Continue to [json →](/coding/languages/python/standard-library/json/) ## SOURCES - Python Standard Library — pathlib. docs.python.org/3/library/pathlib.html. - PEP 428 — The pathlib module. peps.python.org/pep-0428/. - PEP 519 — Adding a file system path protocol. peps.python.org/pep-0519/. **Primary source:** docs.python.org/3/library/pathlib.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python json — JSON Encoder and Decoder module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/json/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Python dicts to JSON and back — the universal data exchange format of the web.* ## CANONICAL DEFINITION The Python json module encodes Python objects (dict, list, str, int, float, bool, None) to JSON strings and decodes JSON strings back to Python objects, following the RFC 8259 JSON specification. ## BEGINNER > One sentence The `json` module converts between Python objects (dicts, lists, strings, numbers, booleans, None) and JSON strings — the universal data format of the web. ## Encoding Python → JSON ```python import json data = { "name": "Priya Sharma", "age": 28, "city": "Mumbai", "skills": ["Python", "SQL", "pandas"], "employed": True, "score": 9.5, "notes": None } # json.dumps() — Python object to JSON string compact = json.dumps(data) print(compact) # {"name": "Priya Sharma", "age": 28, ...} # Pretty-printed with indentation pretty = json.dumps(data, indent=2, ensure_ascii=False) print(pretty) # Write directly to a file with open("data.json", "w", encoding="utf-8") as f: json.dump(data, f, indent=2, ensure_ascii=False) # ensure_ascii=False: preserve non-ASCII (e.g. ₹, हिंदी) ``` ## Decoding JSON → Python ```python import json json_string = ''' { "name": "Priya", "age": 28, "skills": ["Python", "SQL"], "active": true } ''' # json.loads() — JSON string to Python object data = json.loads(json_string) print(data["name"]) # Priya print(type(data)) # print(type(data["age"])) # # Read from a file with open("data.json", "r", encoding="utf-8") as f: loaded = json.load(f) # JSON type mapping # JSON null → Python None # JSON true → Python True # JSON false → Python False # JSON number → Python int or float # JSON string → Python str # JSON array → Python list # JSON object → Python dict ``` ## Custom serialisation ```python import json from datetime import datetime, date from decimal import Decimal # json.dumps raises TypeError for non-JSON-serialisable types # datetime, Decimal, custom objects — need custom encoder class PrecisionEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, datetime): return obj.isoformat() # "2026-06-01T14:30:00" if isinstance(obj, date): return obj.isoformat() # "2026-06-01" if isinstance(obj, Decimal): return float(obj) # precision loss — be careful return super().default(obj) # raises TypeError for unknown types data = { "timestamp": datetime.now(), "price": Decimal("99.99"), } print(json.dumps(data, cls=PrecisionEncoder, indent=2)) # Simpler: use default= parameter for a single conversion function def json_default(obj): if isinstance(obj, (datetime, date)): return obj.isoformat() raise TypeError(f"Object of type {type(obj)} is not JSON serialisable") json.dumps(data, default=json_default) ``` ✅ Beginner tab complete — check your understanding - I know: json.load() reads from a file, json.loads() parses a string - I know: json.dump() writes to a file, json.dumps() returns a string - I know the Python ↔ JSON type mapping (dict↔object, list↔array, None↔null) - I can pretty-print JSON with indent=2 Continue to [re — Regular Expressions →](/coding/languages/python/standard-library/re/) ## INTERMEDIATE ## Performance and alternatives Python's built-in `json` module is written in Python (with a C accelerator). For high-performance JSON, consider `orjson` (Rust-based, 3–10× faster, handles datetime/numpy natively) or `ujson`. `orjson` returns bytes, not str. For config files, consider `tomllib` (Python 3.11+ stdlib for TOML) or PyYAML for YAML. ```python import json # Sorting keys for deterministic output data = {"b": 2, "a": 1, "c": 3} print(json.dumps(data, sort_keys=True)) # {"a": 1, "b": 2, "c": 3} # Separators — compact output without spaces compact = json.dumps(data, separators=(",", ":")) print(compact) # {"b":2,"a":1,"c":3} # object_hook — convert dict to custom class on decode class Config: def __init__(self, d): self.__dict__ = d cfg = json.loads('{"host":"localhost","port":5432}', object_hook=Config) print(cfg.host) # localhost print(cfg.port) # 5432 # parse_float / parse_int — control number parsing from decimal import Decimal precise = json.loads('{"price": 9.99}', parse_float=Decimal) print(precise["price"]) # Decimal('9.99') — no float precision loss print(type(precise["price"])) # ``` **Commonly confused:** - JSON null maps to Python None, not the string "null". When you decode {"value": null}, you get {"value": None} in Python. When you encode {"value": None}, you get {"value": null} in JSON. They are equivalent. - JSON has no datetime type. Datetimes must be serialised as strings (ISO 8601: "2026-06-01T14:30:00" is the convention). The json module does not do this automatically — you need a custom encoder or a library like orjson that handles it. How this connects Related [pathlib (file I/O)](/coding/languages/python/standard-library/pathlib/)[Error Handling](/coding/languages/python/error-handling/) Primary source [docs.python.org/3/library/json.html](https://docs.python.org/3/library/json.html) ✅ Intermediate tab complete — check your understanding - I can use json.dumps(data, default=str) to handle non-serialisable types simply - I can write a custom default= function for precise control over serialisation - I handle json.JSONDecodeError when loading data from external sources Continue to [re — Regular Expressions →](/coding/languages/python/standard-library/re/) ## EXPERT ## RFC 8259 and JSON specification JSON is specified by RFC 8259 (December 2017, superseding RFC 7159 and RFC 4627). RFC 8259 specifies: JSON is text encoded as UTF-8, UTF-16, or UTF-32 — Python's `json` module uses UTF-8 by default. A JSON value is one of: object, array, number, string, true, false, null. Numbers have no size limit in the spec; Python's parser maps integers to `int` (arbitrary precision) and floats to IEEE 754 double. The `ensure_ascii=True` default escapes all non-ASCII characters as `\uXXXX`; set `ensure_ascii=False` to emit raw Unicode. ✅ Expert tab complete - I know what RFC 8259 says about JSON keys (must be strings), NaN/Infinity (not valid JSON), and encoding - I know orjson is ~5-10x faster than the standard json module for large payloads Continue to [re — Regular Expressions →](/coding/languages/python/standard-library/re/) ## SOURCES - Python Standard Library — json. docs.python.org/3/library/json.html. - RFC 8259 — The JavaScript Object Notation (JSON) Data Interchange Format. datatracker.ietf.org/doc/html/rfc8259. - ECMA-404 — The JSON Data Interchange Standard. ecma-international.org. **Primary source:** docs.python.org/3/library/json.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python re — Regular Expressions module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/re/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Search text for patterns — emails, phone numbers, dates, anything describable as a regular expression.* ## CANONICAL DEFINITION The Python re module provides regular expression matching operations — pattern-based search, extraction, and substitution on strings — using an NFA-based regex engine supporting groups, lookaheads, lookbehinds, and Unicode. ## BEGINNER > One sentence The `re` module lets you search text for patterns — not just fixed strings, but flexible descriptions like "a word starting with a capital letter" or "a phone number in any format" — using regular expressions. ## Basic pattern matching ```python import re text = "Priya Sharma, priya@example.com, +91-98765-43210, Mumbai 400001" # re.search() — find first match anywhere in string match = re.search(r"\d{6}", text) # 6-digit number if match: print(match.group()) # 400001 print(match.start()) # position where match starts print(match.span()) # (start, end) tuple # re.match() — match at the START of string only m = re.match(r"Priya", text) # matches (starts with Priya) m2 = re.match(r"Mumbai", text) # None (not at start) # re.findall() — all non-overlapping matches as a list emails = re.findall(r"[\w.]+@[\w.]+\.[a-z]{2,}", text) print(emails) # ['priya@example.com'] # re.fullmatch() — entire string must match valid = re.fullmatch(r"\d{6}", "400001") # match invalid = re.fullmatch(r"\d{6}", "40000") # None — 5 digits ``` ## Pattern syntax | Pattern | Matches | Example | | --- | --- | --- | | `.` | Any character except newline | `a.c` → "abc", "axc" | | `\w` | Word character: [a-zA-Z0-9_] | `\w+` → "hello_world" | | `\d` | Digit: [0-9] | `\d{4}` → "2026" | | `\s` | Whitespace: space, tab, newline | `\s+` → " " | | `^` | Start of string (or line) | `^Hello` | | `$` | End of string (or line) | `world$` | | `*` | 0 or more of previous | `ab*` → "a", "ab", "abb" | | `+` | 1 or more of previous | `ab+` → "ab", "abb" | | `?` | 0 or 1 of previous | `colou?r` → "color", "colour" | | `{n,m}` | Between n and m repetitions | `\d{2,4}` | | `[abc]` | Character class | `[aeiou]` → any vowel | | `(group)` | Capture group | `(\d{4})-(\d{2})` | | `(?:group)` | Non-capturing group | `(?:http|https)://` | | `a|b` | a or b | `cat|dog` | ## Groups and substitution ```python import re # Capturing groups date_pattern = r"(\d{4})-(\d{2})-(\d{2})" match = re.search(date_pattern, "Today is 2026-06-01, meeting at 14:00") if match: year, month, day = match.groups() print(year, month, day) # 2026 06 01 print(match.group(1)) # 2026 (group 1) print(match.group(0)) # 2026-06-01 (entire match) # Named groups — (?Ppattern) pattern = r"(?P\d{4})-(?P\d{2})-(?P\d{2})" m = re.search(pattern, "2026-06-01") print(m.group("year")) # 2026 # re.sub() — replace matches text = "Hello World Python" cleaned = re.sub(r"\s+", " ", text) # collapse whitespace print(cleaned) # Hello World Python # Use function as replacement def upper_first(m): return m.group().upper() result = re.sub(r"\b[a-z]", upper_first, "hello world") print(result) # Hello World ``` ✅ Beginner tab complete — check your understanding - I know the difference between re.search() (anywhere in string) and re.match() (only at start) - I can write a basic pattern using ., *, +, ?, [], ^, and $ - I can use re.findall() to get all matches as a list - I know to use raw strings (r"pattern") to avoid double-backslash issues Continue to [datetime →](/coding/languages/python/standard-library/datetime/) ## INTERMEDIATE ## Compiled patterns and flags ```python import re # Compile a pattern for reuse — faster in loops email_re = re.compile(r"[\w.]+@[\w.]+\.[a-z]{2,}", re.IGNORECASE) emails = ["priya@example.com", "RAHUL@COMPANY.COM", "not-an-email"] for e in emails: if email_re.fullmatch(e): print(f"Valid: {e}") # Common flags re.IGNORECASE # or re.I — case-insensitive matching re.MULTILINE # or re.M — ^ and $ match start/end of each line re.DOTALL # or re.S — . matches newline too re.VERBOSE # or re.X — ignore whitespace and comments in pattern # re.VERBOSE — write readable patterns with comments phone_re = re.compile(r""" \+? # optional country code + (\d{1,3}) # country code (1-3 digits) [-\s]? # optional separator (\d{5}) # first part of number [-\s]? # optional separator (\d{5}) # second part of number """, re.VERBOSE) # finditer — return match objects, not just strings text = "call +91-98765-43210 or +91-87654-32109" for match in phone_re.finditer(text): print(match.group(), "→", match.groups()) ``` ## Lookaheads, lookbehinds, and non-greedy ```python import re # Greedy vs non-greedy text = "bold and italic" greedy = re.findall(r"", text) # ['bold and italic'] non_greedy = re.findall(r"", text) # ['', '', '', ''] # Lookahead (?=...) — assert what follows, without consuming # Match "Python" only when followed by " 3" matches = re.findall(r"Python(?= 3)", "Python 3 and Python 2") print(matches) # ['Python'] — only the one before " 3" # Negative lookahead (?!...) matches = re.findall(r"Python(?! 2)", "Python 3 and Python 2") print(matches) # ['Python'] — only the one NOT before " 2" # Lookbehind (?<=...) — assert what precedes prices = re.findall(r"(?<=₹)\d+", "Items: ₹500 and ₹1200") print(prices) # ['500', '1200'] ``` **Commonly confused:** - Raw strings (r"...") and regular expressions are separate concepts. A raw string just means backslashes are not escape sequences — r"\n" is a two-character string (\ and n), not a newline. You use raw strings for regex patterns because regex also uses backslashes — \d in a raw string stays \d and is passed literally to the regex engine. - re.match() only matches at the start of the string. To match anywhere, use re.search(). To require the entire string to match, use re.fullmatch(). A common mistake is using re.match() and being surprised it doesn't find a pattern in the middle of a string. How this connects Related [pathlib (file text)](/coding/languages/python/standard-library/pathlib/)[Control Flow](/coding/languages/python/control-flow/) Primary source [docs.python.org/3/library/re.html](https://docs.python.org/3/library/re.html) ✅ Intermediate tab complete — check your understanding - I can use () to capture groups and access them with match.group(1) - I can use re.sub(pattern, replacement, string) for pattern-based replacement - I know when to use re.compile() for patterns used more than once - I know re.IGNORECASE, re.MULTILINE, and re.DOTALL flags Continue to [datetime →](/coding/languages/python/standard-library/datetime/) ## EXPERT ## The regex engine: NFA-based matching Python's `re` module uses a backtracking NFA (non-deterministic finite automaton) engine. The engine tries alternatives at each branch point and backtracks when a path fails. This gives it features (backreferences, lookaheads) that DFA-based engines cannot support, but it is vulnerable to catastrophic backtracking — a regex like `(a+)+` on a non-matching string of `n` a's takes O(2ⁿ) time. Python 3.11+ includes the `re`-written regex module as an alternative. The third-party `regex` package supports PCRE2 features like recursive patterns, possessive quantifiers (preventing backtracking: `a++`), and atomic groups, all of which prevent ReDoS (Regular Expression Denial of Service) vulnerabilities. ✅ Expert tab complete - I can write a positive lookahead (?=...) to match based on what follows - I know the difference between greedy .* and non-greedy .*? - I can recognise a pattern that might cause catastrophic backtracking Continue to [datetime →](/coding/languages/python/standard-library/datetime/) ## SOURCES - Python Standard Library — re. docs.python.org/3/library/re.html. - Python HowTo: Regular Expression HOWTO. docs.python.org/3/howto/regex.html. - POSIX.1-2017 §9 — Regular Expressions. pubs.opengroup.org. **Primary source:** docs.python.org/3/library/re.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python datetime — Date and Time Handling module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/datetime/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Dates, times, durations, and timezones — the complete toolkit for working with time in Python.* ## CANONICAL DEFINITION The Python datetime module provides classes for representing and manipulating dates and times — date, time, datetime, timedelta — with formatting (strftime), parsing (strptime), and timezone-aware arithmetic. ## BEGINNER > One sentence The `datetime` module provides objects for representing and manipulating dates, times, and time intervals — the tools for "what time is it now", "how many days until the deadline", and "format this timestamp for display". ## datetime objects ```python from datetime import datetime, date, time, timedelta # date — year, month, day only today = date.today() print(today) # 2026-06-01 print(today.year) # 2026 print(today.month) # 6 print(today.weekday()) # 0=Mon, 6=Sun # time — hours, minutes, seconds, microseconds t = time(14, 30, 45) print(t) # 14:30:45 # datetime — date + time combined now = datetime.now() # local time utc = datetime.utcnow() # UTC (naive — no timezone info) print(now) # 2026-06-01 14:30:45.123456 # Create specific datetime deadline = datetime(2026, 12, 31, 23, 59, 59) # timedelta — represent a duration one_week = timedelta(weeks=1) ten_days = timedelta(days=10, hours=3) print(today + ten_days) # date 10 days from now # Arithmetic diff = deadline - now print(diff.days) # days until deadline print(diff.total_seconds()) # total seconds ``` ## Formatting and parsing ```python from datetime import datetime now = datetime.now() # strftime — format datetime as string print(now.strftime("%Y-%m-%d")) # 2026-06-01 print(now.strftime("%d/%m/%Y %H:%M")) # 01/06/2026 14:30 print(now.strftime("%A, %B %d, %Y")) # Monday, June 01, 2026 print(now.strftime("%I:%M %p")) # 02:30 PM # strptime — parse string to datetime date_str = "01/06/2026 14:30" parsed = datetime.strptime(date_str, "%d/%m/%Y %H:%M") print(parsed) # 2026-06-01 14:30:00 # ISO 8601 format iso = now.isoformat() # "2026-06-01T14:30:45.123456" back = datetime.fromisoformat(iso) # parse back (Python 3.7+) # Common format codes # %Y = 4-digit year %m = month (01-12) %d = day (01-31) # %H = hour (00-23) %M = minute %S = second # %A = weekday name %B = month name %p = AM/PM ``` ## Timezones ```python from datetime import datetime, timezone, timedelta # Timezone-aware datetime utc_now = datetime.now(timezone.utc) print(utc_now) # 2026-06-01 09:00:45.123456+00:00 # Fixed offset timezone ist = timezone(timedelta(hours=5, minutes=30)) # India Standard Time ist_now = datetime.now(ist) print(ist_now) # 2026-06-01 14:30:45.123456+05:30 # Convert between timezones utc_time = datetime(2026, 6, 1, 9, 0, tzinfo=timezone.utc) ist_time = utc_time.astimezone(ist) print(ist_time) # 2026-06-01 14:30:00+05:30 # For full timezone database (all cities, DST), use zoneinfo (3.9+) from zoneinfo import ZoneInfo mumbai = ZoneInfo("Asia/Kolkata") mumbai_now = datetime.now(mumbai) ``` ✅ Beginner tab complete — check your understanding - I can get the current date/time with datetime.datetime.now() - I can format a datetime as a string with strftime() - I can parse a date string with strptime() - I can add days or hours to a date with timedelta Continue to [collections →](/coding/languages/python/standard-library/collections/) ## INTERMEDIATE ## Naive vs. aware datetimes A **naive** datetime has no timezone information — Python doesn't know if it's UTC, IST, or PST. An **aware** datetime has a `tzinfo` object attached. You cannot compare or subtract naive and aware datetimes — Python raises `TypeError`. The rule for production code: always store and transmit timestamps in UTC (aware), convert to local time only for display. Never use `datetime.now()` without a timezone for anything that will be stored or compared across systems. ```python from datetime import datetime, timezone from zoneinfo import ZoneInfo # Python 3.9+ # CORRECT: always use UTC for storage and computation def record_event(): return datetime.now(timezone.utc) # aware UTC datetime # CORRECT: convert to local only for display def display_time(utc_dt, tz_name="Asia/Kolkata"): local_tz = ZoneInfo(tz_name) local = utc_dt.astimezone(local_tz) return local.strftime("%d %B %Y, %I:%M %p %Z") event_time = record_event() print(display_time(event_time)) # 01 June 2026, 02:30 PM IST # Compare datetimes t1 = datetime(2026, 6, 1, 9, 0, tzinfo=timezone.utc) t2 = datetime(2026, 6, 1, 14, 30, tzinfo=ZoneInfo("Asia/Kolkata")) print(t1 == t2) # True — same moment in time # Epoch timestamp (Unix time) now = datetime.now(timezone.utc) timestamp = now.timestamp() # float seconds since 1970-01-01 UTC back = datetime.fromtimestamp(timestamp, tz=timezone.utc) ``` **Commonly confused:** - datetime.utcnow() is deprecated (Python 3.12+) and returns a naive datetime. It looks like UTC but carries no timezone info — it can be mistakenly compared with local times. Use datetime.now(timezone.utc) instead, which returns a timezone-aware UTC datetime. - Naive datetimes assume local time, not UTC. datetime.now() (without timezone) returns the local time as a naive datetime. datetime.now(timezone.utc) returns UTC as an aware datetime. Mixing them causes bugs. How this connects Related [json (ISO 8601 serialisation)](/coding/languages/python/standard-library/json/) Primary source [docs.python.org/3/library/datetime.html](https://docs.python.org/3/library/datetime.html) ✅ Intermediate tab complete — check your understanding - I know the difference between a naive datetime (no timezone) and an aware datetime - I can create a timezone-aware datetime using datetime.timezone.utc or ZoneInfo - I know never to compare naive and aware datetimes (it raises TypeError) Continue to [collections →](/coding/languages/python/standard-library/collections/) ## EXPERT ## Epoch, TAI, and the complexity of time The Unix epoch is 1970-01-01 00:00:00 UTC. `datetime.timestamp()` returns seconds since the epoch as a float (IEEE 754 double, accurate to microseconds until ~2255). UTC is not perfectly uniform — it is occasionally adjusted by leap seconds to stay within 0.9 seconds of UT1 (based on Earth's rotation). Python's `datetime` ignores leap seconds (follows POSIX time). The `zoneinfo` module (Python 3.9, PEP 615) uses the IANA time zone database — the authoritative source for all historical and current time zone rules including DST transitions. ✅ Expert tab complete - I know what Unix epoch time is and its 2038 problem (for 32-bit systems) - I understand why daylight saving time makes time arithmetic non-trivial Continue to [collections →](/coding/languages/python/standard-library/collections/) ## SOURCES - Python Standard Library — datetime. docs.python.org/3/library/datetime.html. - Python Standard Library — zoneinfo. docs.python.org/3/library/zoneinfo.html. - PEP 615 — Support for the IANA Time Zone Database. peps.python.org/pep-0615/. - IANA Time Zone Database. iana.org/time-zones. **Primary source:** docs.python.org/3/library/datetime.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python collections — Specialised Container Datatypes module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/collections/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Counter, defaultdict, namedtuple, deque — the specialised data structures that solve common problems with less code.* ## CANONICAL DEFINITION The Python collections module provides specialised container datatypes beyond the built-in list, dict, and set — including Counter for tallying, defaultdict for auto-initialised dicts, namedtuple for named tuples, and deque for O(1) double-ended queues. ## BEGINNER > One sentence The `collections` module contains specialised data structures that go beyond Python's built-in list, dict, and set — each solving a common problem with less code and better performance. ## Counter ```python from collections import Counter # Count occurrences of elements words = ["apple", "banana", "apple", "cherry", "banana", "apple"] counts = Counter(words) print(counts) # Counter({'apple': 3, 'banana': 2, 'cherry': 1}) print(counts["apple"]) # 3 print(counts["unknown"]) # 0 — missing keys return 0, not KeyError # most_common(n) — top n elements print(counts.most_common(2)) # [('apple', 3), ('banana', 2)] # Count characters in a string char_counts = Counter("mississippi") print(char_counts.most_common(3)) # [('s', 4), ('i', 4), ('p', 2)] # Arithmetic c1 = Counter(a=3, b=2) c2 = Counter(a=1, b=4) print(c1 + c2) # Counter({'b': 6, 'a': 4}) print(c1 - c2) # Counter({'a': 2}) — only positive counts ``` ## defaultdict ```python from collections import defaultdict # defaultdict(factory) — auto-creates missing keys word_positions = defaultdict(list) # missing key → empty list for i, word in enumerate("the quick brown fox jumps".split()): word_positions[word].append(i) # no KeyError for new words print(dict(word_positions)) # {'the': [0], 'quick': [1], 'brown': [2], 'fox': [3], 'jumps': [4]} # Grouping students = [("A", "Priya"), ("B", "Rahul"), ("A", "Anita"), ("B", "Zara")] by_grade = defaultdict(list) for grade, name in students: by_grade[grade].append(name) print(dict(by_grade)) # {'A': ['Priya', 'Anita'], 'B': ['Rahul', 'Zara']} # Counter: count with defaultdict(int) counts = defaultdict(int) for word in "the fox the fox fox".split(): counts[word] += 1 # no KeyError — missing key → 0 ``` ## namedtuple and OrderedDict ```python from collections import namedtuple, OrderedDict, deque # namedtuple — tuple with named fields Point = namedtuple("Point", ["x", "y"]) p = Point(10, 20) print(p.x, p.y) # 10 20 print(p[0]) # 10 — index access still works print(p._asdict()) # {'x': 10, 'y': 20} x, y = p # unpacking still works Employee = namedtuple("Employee", "name department salary") emp = Employee("Priya", "Engineering", 75000) print(emp.name) # Priya # OrderedDict — dict that remembers insertion order # Note: Regular dict is ordered since Python 3.7 # OrderedDict still has move_to_end() and popitem(last=) od = OrderedDict() od["a"] = 1 od["b"] = 2 od.move_to_end("a") # move to end print(list(od)) # ['b', 'a'] # deque — O(1) insert/delete at both ends (unlike list which is O(n) at front) q = deque([1, 2, 3]) q.appendleft(0) # O(1) — add to front q.popleft() # O(1) — remove from front q.append(4) # O(1) — add to back q.rotate(1) # rotate right by 1: [4, 1, 2, 3] # maxlen — sliding window recent = deque(maxlen=3) # keeps last 3 items for x in range(6): recent.append(x) print(list(recent)) # [3, 4, 5] ``` ✅ Beginner tab complete — check your understanding - I can use Counter(list) to count frequencies in one line - I can use Counter.most_common(n) to get the n most frequent items - I can use defaultdict(list) to avoid KeyError when the key doesn't exist yet - I can use defaultdict(int) as a counter dict (auto-initialises to 0) Continue to [itertools →](/coding/languages/python/standard-library/itertools/) ## INTERMEDIATE ## ChainMap `ChainMap` links multiple dicts into a single view. Lookups search each map in order — the first one containing the key wins. Writes go to the first map. This is the mechanism behind Python's LEGB scope lookup — local vars, then enclosing, then global, then built-ins, all as separate dicts. ```python from collections import ChainMap defaults = {"color": "blue", "size": "medium", "style": "flat"} overrides = {"color": "green", "size": "large"} config = ChainMap(overrides, defaults) print(config["color"]) # green — from overrides print(config["style"]) # flat — from defaults print(dict(config)) # {"color":"green","size":"large","style":"flat"} # ChainMap is how Python implements nested scopes # Updating the first map only config["color"] = "red" print(overrides) # {'color': 'red', 'size': 'large'} print(defaults) # unchanged ``` ## UserDict, UserList, UserString These are wrapper classes that make it easy to subclass `dict`, `list`, and `str` without running into the pitfalls of subclassing built-ins directly (built-in methods sometimes bypass overridden methods). Subclassing `UserDict` instead of `dict` ensures all dict operations go through your overridden methods. **Commonly confused:** - deque is not a list with O(1) front operations. deque does not support O(1) random access by index. deque[n] is O(n) (traversal from nearest end). Use deque when you need O(1) insertions/deletions at both ends (queues, sliding windows). Use list when you need O(1) random access. - namedtuple is still a tuple. It is immutable. You cannot set p.x = 5. For mutable named-field data classes, use @dataclass (Python 3.7+) or namedtuple._replace(x=5) to create a new copy. How this connects Related [Built-in Types](/coding/languages/python/variables-and-types/)[itertools](/coding/languages/python/standard-library/itertools/) Primary source [docs.python.org/3/library/collections.html](https://docs.python.org/3/library/collections.html) ✅ Intermediate tab complete — check your understanding - I know why deque is faster than list for appendleft/popleft operations - I can create a namedtuple and access fields by name instead of index - I know that regular dict maintains insertion order in Python 3.7+ (so OrderedDict is rarely needed) Continue to [itertools →](/coding/languages/python/standard-library/itertools/) ## EXPERT ## Abstract Base Classes in collections.abc The `collections.abc` module (part of `collections`) defines Abstract Base Classes (ABCs) for containers: `Iterable`, `Iterator`, `Sequence`, `MutableSequence`, `Mapping`, `MutableMapping`, `Set`, `MutableSet`. These enable `isinstance()` checks for protocol compliance and serve as base classes. A class implementing `__iter__` and `__next__` is automatically an `Iterator` (via virtual subclass registration). This is how `isinstance([], Sequence)` returns `True` even though `list` does not explicitly inherit from `Sequence`. ✅ Expert tab complete - I can use ChainMap for layered lookups (e.g., defaults overridden by user settings) - I know to subclass UserDict instead of dict when creating custom mapping types Continue to [itertools →](/coding/languages/python/standard-library/itertools/) ## SOURCES - Python Standard Library — collections. docs.python.org/3/library/collections.html. - Python Standard Library — collections.abc. docs.python.org/3/library/collections.abc.html. - PEP 3119 — Introducing Abstract Base Classes. peps.python.org/pep-3119/. **Primary source:** docs.python.org/3/library/collections.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python itertools — Iterator Building Blocks module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/itertools/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *chain, product, combinations, groupby, accumulate — the iterator toolkit that makes sequence operations both elegant and memory-efficient.* ## CANONICAL DEFINITION The Python itertools module provides a collection of fast, memory-efficient iterator building blocks — functions for combining, filtering, grouping, and generating sequences lazily, following the iterator algebra model. ## BEGINNER > One sentence `itertools` is a toolkit of building blocks for working with sequences — it gives you efficient, memory-friendly tools for chaining, grouping, filtering, combining, and repeating iterables. ## Combining iterables ```python import itertools # chain — flatten multiple iterables into one a = [1, 2, 3] b = [4, 5, 6] c = [7, 8, 9] combined = list(itertools.chain(a, b, c)) print(combined) # [1, 2, 3, 4, 5, 6, 7, 8, 9] # chain.from_iterable — flatten one level nested = [[1, 2], [3, 4], [5, 6]] flat = list(itertools.chain.from_iterable(nested)) print(flat) # [1, 2, 3, 4, 5, 6] # zip_longest — zip, filling missing values with fillvalue names = ["Priya", "Rahul"] scores = [95, 88, 72] # longer for n, s in itertools.zip_longest(names, scores, fillvalue="N/A"): print(f"{n}: {s}") # Priya: 95 | Rahul: 88 | N/A: 72 ``` ## Permutations and combinations ```python import itertools items = ["A", "B", "C"] # product — cartesian product (nested for loops) for p in itertools.product([0, 1], repeat=3): print(p) # all 8 binary combinations of length 3 # permutations — all orderings for p in itertools.permutations(items, 2): print(p) # ('A','B'), ('A','C'), ('B','A'), ... — 6 total # combinations — unique subsets (order doesn't matter) for c in itertools.combinations(items, 2): print(c) # ('A','B'), ('A','C'), ('B','C') — 3 total # combinations_with_replacement — can repeat elements for c in itertools.combinations_with_replacement("ABC", 2): print(c) # ('A','A'), ('A','B'), ('A','C'), ('B','B'), ... ``` ## Infinite iterators and slicing ```python import itertools # count — count from start by step for n in itertools.count(10, 2): # 10, 12, 14, ... if n > 20: break print(n) # cycle — repeat a sequence forever colours = itertools.cycle(["red", "green", "blue"]) print([next(colours) for _ in range(7)]) # ['red', 'green', 'blue', 'red', 'green', 'blue', 'red'] # repeat — repeat a value (n times, or forever) ones = list(itertools.repeat(1, 5)) # [1, 1, 1, 1, 1] # islice — slice any iterator (not just sequences) gen = (x**2 for x in range(1000)) first_ten = list(itertools.islice(gen, 10)) # [0,1,4,9,16,25,36,49,64,81] # groupby — group consecutive same-key elements data = [("A", 1), ("A", 2), ("B", 3), ("B", 4), ("A", 5)] for key, group in itertools.groupby(data, key=lambda x: x[0]): print(key, list(group)) # A [('A', 1), ('A', 2)] | B [('B', 3), ('B', 4)] | A [('A', 5)] ``` ✅ Beginner tab complete — check your understanding - I can use itertools.chain() to flatten a list of lists - I can use itertools.islice() to take the first N items from any iterator - I can use itertools.product() to generate all combinations of multiple sequences Continue to [sys →](/coding/languages/python/standard-library/sys/) ## INTERMEDIATE ## Functional itertools: takewhile, dropwhile, filterfalse, accumulate ```python import itertools import operator # takewhile — yield items while condition is True nums = [1, 2, 3, 4, 5, 4, 3, 2, 1] ascending = list(itertools.takewhile(lambda x: x < 5, nums)) print(ascending) # [1, 2, 3, 4] # dropwhile — skip items while condition is True, then yield rest rest = list(itertools.dropwhile(lambda x: x < 5, nums)) print(rest) # [5, 4, 3, 2, 1] # filterfalse — yield items where predicate is False evens = list(itertools.filterfalse(lambda x: x % 2, range(10))) print(evens) # [0, 2, 4, 6, 8] # accumulate — running accumulation (default: sum) running_sum = list(itertools.accumulate([1, 2, 3, 4, 5])) print(running_sum) # [1, 3, 6, 10, 15] # Running max running_max = list(itertools.accumulate([3,1,4,1,5,9,2], func=max)) print(running_max) # [3, 3, 4, 4, 5, 9, 9] # Running product factorial = list(itertools.accumulate(range(1,6), func=operator.mul)) print(factorial) # [1, 2, 6, 24, 120] ``` ## Recipes from itertools docs ```python import itertools # pairwise (Python 3.10+) — overlapping pairs from itertools import pairwise pairs = list(pairwise([1, 2, 3, 4])) print(pairs) # [(1,2), (2,3), (3,4)] # batched (Python 3.12+) — batch into n-size chunks from itertools import batched pages = list(batched(range(10), 3)) print(pages) # [(0,1,2), (3,4,5), (6,7,8), (9,)] # Classic recipe: sliding window def sliding_window(iterable, n): it = iter(iterable) window = tuple(itertools.islice(it, n)) if len(window) == n: yield window for x in it: window = window[1:] + (x,) yield window print(list(sliding_window([1,2,3,4,5], 3))) # [(1,2,3), (2,3,4), (3,4,5)] ``` **Commonly confused:** - itertools.groupby() only groups consecutive equal elements — it does not group all elements with the same key. If the input is [A, A, B, A], you get three groups, not two. Sort the input by key first if you want all same-key elements together. - All itertools functions return lazy iterators — they do not materialise results. itertools.chain(a, b) returns an iterator; nothing is computed until you iterate. Wrap in list() to materialise, but only when you need all values — lazy is memory-efficient for large sequences. How this connects Related [Generators](/coding/languages/python/functions-and-scope/)[collections](/coding/languages/python/standard-library/collections/) Primary source [docs.python.org/3/library/itertools.html](https://docs.python.org/3/library/itertools.html) ✅ Intermediate tab complete — check your understanding - I know the difference between combinations (order doesn't matter) and permutations (order matters) - I know that groupby only groups consecutive equal elements — data must be sorted first - I can use accumulate() to create a running total without writing a loop Continue to [sys →](/coding/languages/python/standard-library/sys/) ## EXPERT ## itertools and the iterator algebra The `itertools` documentation describes the module as "a set of fast, memory efficient tools that are useful by themselves or in combination" and calls the patterns from the module "iterator algebra." Every `itertools` function returns a lazy iterator that consumes its inputs one element at a time — composing them builds pipelines that never materialise large intermediate lists. This model is equivalent to Haskell's lazy list combinators and inspired by Clojure's transducers. The Python docs include a recipes section (docs.python.org/3/library/itertools.html#itertools-recipes) with implementations of `pairwise`, `batched`, `sliding_window`, and ~20 other common patterns. ✅ Expert tab complete - I can use itertools.cycle() with islice() to create a repeating sequence of finite length - I understand iterator algebra: composing simple iterators avoids intermediate lists Continue to [sys →](/coding/languages/python/standard-library/sys/) ## SOURCES - Python Standard Library — itertools. docs.python.org/3/library/itertools.html. - Python itertools recipes. docs.python.org/3/library/itertools.html#itertools-recipes. **Primary source:** docs.python.org/3/library/itertools.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python sys — System-specific Parameters and Functions module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/sys/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *argv, stdout, sys.path, sys.modules — direct access to the Python interpreter's own internals.* ## CANONICAL DEFINITION The Python sys module provides access to variables and functions used by the Python interpreter itself — command-line arguments, standard streams, the module import system, recursion limits, and interpreter version information. ## BEGINNER > One sentence The `sys` module is your window into the Python interpreter itself — it lets you read command-line arguments, control stdin/stdout/stderr, inspect the import system, and exit the program. ## Command-line arguments ```python import sys # sys.argv[0] is the script name # sys.argv[1:] are the arguments passed # python3 script.py hello world --verbose print(sys.argv) # ['script.py', 'hello', 'world', '--verbose'] print(sys.argv[0]) # script.py print(sys.argv[1:]) # ['hello', 'world', '--verbose'] # Simple argument parsing if len(sys.argv) ", file=sys.stderr) sys.exit(1) # exit with error code name = sys.argv[1] print(f"Hello, {name}!") # For complex argument parsing, use argparse (stdlib) instead ``` ## stdin, stdout, stderr ```python import sys # Write to stdout (same as print but explicit) sys.stdout.write("Hello\n") # Write to stderr — for error messages and diagnostics sys.stderr.write("Error: something went wrong\n") print("Error:", exc_info, file=sys.stderr) # print to stderr # Read from stdin line = sys.stdin.readline() for line in sys.stdin: # iterate all lines process(line.strip()) # Redirect stdout import io buffer = io.StringIO() sys.stdout = buffer print("captured") sys.stdout = sys.__stdout__ # restore output = buffer.getvalue() # "captured\n" ``` ## Python interpreter information ```python import sys print(sys.version) # '3.13.0 (main, Oct 2024, ...)' — full version string print(sys.version_info) # sys.version_info(major=3, minor=13, micro=0, ...) print(sys.platform) # 'linux', 'darwin', 'win32' print(sys.executable) # '/usr/bin/python3' — path to interpreter print(sys.prefix) # installation directory # sys.path — where Python looks for modules to import print(sys.path) sys.path.insert(0, "/my/custom/modules") # add custom path # Check Python version if sys.version_info < (3, 10): raise RuntimeError("Python 3.10+ required") ``` ✅ Beginner tab complete — check your understanding - I know sys.argv[0] is the script name; sys.argv[1:] are the arguments - I can use sys.exit(0) for success and sys.exit(1) for failure - I know sys.path is the list of directories Python searches for modules Continue to [asyncio →](/coding/languages/python/standard-library/asyncio/) ## INTERMEDIATE ## The import system and sys.modules `sys.modules` is a dictionary mapping module names to already-imported module objects. When you `import foo`, Python checks `sys.modules` first — if it's there, it returns the cached module. This is why importing the same module twice is cheap. You can inspect it, add mock modules, or even remove a module to force reimport. ```python import sys import json # Already imported print("json" in sys.modules) # True print(sys.modules["json"] is json) # True — same object # Force reimport (rarely needed) del sys.modules["json"] import json # fresh import # Mock a module for testing import types mock_requests = types.ModuleType("requests") mock_requests.get = lambda url: type("Response", (), {"status_code": 200})() sys.modules["requests"] = mock_requests import requests # returns our mock ``` ## sys.getrecursionlimit and sys.setrecursionlimit ```python import sys # Default is 1000 print(sys.getrecursionlimit()) # 1000 # Increase if needed (with care) sys.setrecursionlimit(10000) # Check recursion depth def depth_check(): frame = sys._getframe() d = 0 while frame: frame = frame.f_back d += 1 return d # sys.getsizeof — memory usage of an object import sys print(sys.getsizeof([])) # 56 bytes (empty list) print(sys.getsizeof([1,2,3])) # 88 bytes (3 ints + list overhead) print(sys.getsizeof("hello")) # 54 bytes # Note: getsizeof does NOT include objects referenced by the object ``` **Commonly confused:** - sys.exit() raises SystemExit, it does not immediately terminate. sys.exit() raises a SystemExit exception. This means finally blocks and atexit handlers still run. If caught by a bare except:, the program will not exit. Use os._exit() only if you need immediate termination without cleanup. - sys.getsizeof() does not measure total memory of nested objects. It returns only the memory used by the object itself, not by any objects it references. A list of 1000 strings: sys.getsizeof(lst) gives the list's own memory (pointers). The strings' memory is not included. Use tracemalloc for full memory profiling. How this connects Related [os](/coding/languages/python/standard-library/os/)[Functions & Scope](/coding/languages/python/functions-and-scope/) Primary source [docs.python.org/3/library/sys.html](https://docs.python.org/3/library/sys.html) ✅ Intermediate tab complete — check your understanding - I know the difference between stdout (normal output) and stderr (error output) - I can check if a module is already imported with "name" in sys.modules - I know sys.getrecursionlimit() and can increase it with sys.setrecursionlimit() Continue to [asyncio →](/coding/languages/python/standard-library/asyncio/) ## EXPERT ## CPython frame objects and the frame stack `sys._getframe(depth)` returns the call frame at the given depth. A frame object contains: `f_code` (the code object), `f_locals` (local variables dict), `f_globals` (global variables dict), `f_back` (the calling frame), `f_lineno` (current line). The leading underscore signals this is a CPython implementation detail — not guaranteed to exist in other Python implementations (PyPy, Jython). This is the mechanism behind tracebacks, debuggers (pdb), and profilers (cProfile). ## sys.audit hooks (Python 3.8+) PEP 578 added `sys.audit(event, *args)` and `sys.addaudithook(hook)` — a security mechanism for monitoring or restricting potentially dangerous operations. When `sys.audit("open", file, mode)` is called internally by CPython (on file open, exec, import, etc.), all registered audit hooks are called. This enables security frameworks to monitor or veto operations. Audit hooks cannot be removed once added — only new hooks can be added. This is used by the CPython restricted execution proposals. ✅ Expert tab complete - I know sys._getframe() returns the current execution frame object - I understand what sys.addaudithook() does and when it's used Continue to [asyncio →](/coding/languages/python/standard-library/asyncio/) ## SOURCES - Python Standard Library — sys. docs.python.org/3/library/sys.html. - PEP 578 — Python Runtime Audit Hooks. peps.python.org/pep-0578/. - CPython source — Python/ceval.c. github.com/python/cpython. **Primary source:** docs.python.org/3/library/sys.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python asyncio module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/asyncio/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Concurrent I/O on a single thread — write code that waits for many things at once without creating a thread per task.* ## CANONICAL DEFINITION The Python asyncio module provides an event-loop-based framework for writing concurrent I/O-bound code using async/await syntax — running many coroutines on a single thread by suspending them when waiting for I/O. ## BEGINNER > One sentence `asyncio` lets you write code that does multiple things at once — downloading files, querying databases, waiting for network responses — without creating threads, by pausing tasks when they're waiting and resuming them when ready. ## async / await basics ```python import asyncio # async def defines a coroutine function # Calling it returns a coroutine object — nothing runs yet async def greet(name, delay): await asyncio.sleep(delay) # pause THIS coroutine, yield control print(f"Hello, {name}!") return f"greeted {name}" # asyncio.run() — entry point; creates and runs the event loop async def main(): # Sequential — total 3 seconds await greet("Priya", 1) await greet("Rahul", 2) asyncio.run(main()) # Run concurrently with asyncio.gather async def main_concurrent(): # Both start immediately, total ~2 seconds (not 3) results = await asyncio.gather( greet("Priya", 1), greet("Rahul", 2), ) print(results) # ['greeted Priya', 'greeted Rahul'] asyncio.run(main_concurrent()) ``` ## Tasks ```python import asyncio async def fetch_data(url, delay): print(f"Fetching {url}...") await asyncio.sleep(delay) # simulate network I/O return f"data from {url}" async def main(): # Create tasks — schedule coroutines to run concurrently task1 = asyncio.create_task(fetch_data("api.example.com/users", 1)) task2 = asyncio.create_task(fetch_data("api.example.com/posts", 2)) task3 = asyncio.create_task(fetch_data("api.example.com/comments", 0.5)) # Tasks run concurrently from here; await to get results result1 = await task1 result2 = await task2 result3 = await task3 print(result1, result2, result3) # Timeout — cancel if too slow try: result = await asyncio.wait_for(fetch_data("slow.com", 10), timeout=2.0) except asyncio.TimeoutError: print("Request timed out") asyncio.run(main()) ``` ## Real I/O: aiohttp pattern ```python # asyncio itself doesn't do HTTP — use aiohttp (pip install aiohttp) # This shows the correct pattern for concurrent HTTP requests import asyncio async def fetch(session, url): # async with: async context manager async with session.get(url) as response: return await response.json() async def fetch_all(urls): import aiohttp async with aiohttp.ClientSession() as session: tasks = [fetch(session, url) for url in urls] return await asyncio.gather(*tasks) # all concurrent urls = [ "https://jsonplaceholder.typicode.com/posts/1", "https://jsonplaceholder.typicode.com/posts/2", "https://jsonplaceholder.typicode.com/posts/3", ] results = asyncio.run(fetch_all(urls)) print(len(results)) # 3 responses fetched concurrently ``` ✅ Beginner tab complete — check your understanding - I can define a coroutine with "async def" and await another coroutine with "await" - I can run a coroutine as the entry point with asyncio.run() - I can run multiple coroutines concurrently with await asyncio.gather() - I understand that await yields control — it does NOT block the event loop Continue to [logging →](/coding/languages/python/standard-library/logging/) ## INTERMEDIATE ## Event loop, coroutines, and the execution model asyncio runs on a single thread. The **event loop** maintains a queue of coroutines. When a coroutine hits an `await` that blocks (network I/O, sleep), the event loop suspends it and runs another. When the I/O completes (signalled by the OS via epoll/kqueue/IOCP), the event loop resumes the waiting coroutine. This is cooperative multitasking — coroutines must voluntarily yield with `await`. CPU-bound code never yields and starves other coroutines. ```python import asyncio # asyncio.TaskGroup (Python 3.11+) — structured concurrency async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro()) task2 = tg.create_task(another_coro()) # Both tasks complete (or raise) before continuing # If either raises, the other is cancelled # Semaphore — limit concurrent operations async def limited_fetch(semaphore, url): async with semaphore: # max 5 concurrent fetches return await fetch(url) async def main_limited(): sem = asyncio.Semaphore(5) tasks = [limited_fetch(sem, url) for url in large_url_list] return await asyncio.gather(*tasks) # Queue — producer/consumer pattern async def producer(queue): for i in range(10): await queue.put(i) await asyncio.sleep(0.1) await queue.put(None) # sentinel async def consumer(queue): while True: item = await queue.get() if item is None: break print(f"Processing {item}") async def main_queue(): q = asyncio.Queue() await asyncio.gather(producer(q), consumer(q)) ``` ## Running sync code in executor ```python import asyncio from concurrent.futures import ThreadPoolExecutor # CPU-bound or blocking sync code must run in an executor # to avoid blocking the event loop async def main(): loop = asyncio.get_event_loop() # Run blocking I/O in thread pool def blocking_read(): with open("large_file.txt") as f: return f.read() content = await loop.run_in_executor(None, blocking_read) # Run CPU-bound work in process pool from concurrent.futures import ProcessPoolExecutor with ProcessPoolExecutor() as pool: result = await loop.run_in_executor(pool, cpu_heavy_function, data) ``` **Commonly confused:** - asyncio does not make CPU-bound code faster. asyncio is for I/O-bound concurrency — tasks that spend time waiting for network, disk, or timers. If your code is CPU-bound (number crunching, image processing), use multiprocessing or concurrent.futures.ProcessPoolExecutor instead. Running CPU-bound code in a single asyncio event loop starves all other coroutines. - A coroutine function is not a coroutine. async def greet() is a coroutine function. Calling greet() returns a coroutine object — nothing runs. The coroutine only runs when awaited or scheduled as a task. Forgetting to await a coroutine is a common bug that produces no error — the coroutine just never executes. How this connects Requires first [Generators & Coroutines](/coding/languages/python/functions-and-scope/)[Error Handling](/coding/languages/python/error-handling/) Enables [stdlib hub](/coding/languages/python/standard-library//) Primary source [docs.python.org/3/library/asyncio.html](https://docs.python.org/3/library/asyncio.html) ✅ Intermediate tab complete — check your understanding - I know the difference between asyncio.gather() and asyncio.create_task() - I can use asyncio.wait_for(coro, timeout=5) to cancel if it takes too long - I can use loop.run_in_executor(None, blocking_func, arg) to run synchronous code Continue to [logging →](/coding/languages/python/standard-library/logging/) ## EXPERT ## PEP 492, PEP 3156, and the async/await history Python's async history: PEP 342 (Python 2.5, 2005) gave generators `send()` — enabling coroutines. PEP 3156 (Python 3.4, 2014) introduced `asyncio` as the standard event loop and the `@asyncio.coroutine` / `yield from` syntax. PEP 492 (Python 3.5, 2015) introduced `async def`, `await`, `async for`, and `async with` as first-class syntax, replacing the decorator-based approach. PEP 525 (Python 3.6) added asynchronous generators. PEP 654 (Python 3.11) introduced `TaskGroup` and `ExceptionGroup` for structured concurrency. ## The event loop implementation CPython's asyncio event loop uses the OS I/O polling mechanism: `select.epoll()` on Linux, `select.kqueue()` on macOS, and IOCP on Windows (via the `ProactorEventLoop`). The loop maintains a *ready* queue (callbacks to call immediately) and a *scheduled* queue (callbacks with a deadline). Each iteration: run all ready callbacks, then poll the OS for I/O events with a timeout matching the next scheduled deadline, then process I/O callbacks. Third-party loops (uvloop) replace the event loop with a libuv-based implementation, achieving 2–4× higher throughput for I/O-bound servers. ✅ Expert tab complete - I can explain what the event loop's selector mechanism does - I know that PEP 492 (Python 3.5) introduced native async/await syntax - I know when to use asyncio.shield() to prevent a coroutine from being cancelled Continue to [logging →](/coding/languages/python/standard-library/logging/) ## SOURCES - Python Standard Library — asyncio. docs.python.org/3/library/asyncio.html. - PEP 492 — Coroutines with async and await syntax. peps.python.org/pep-0492/. - PEP 3156 — Asynchronous IO Support Rebooted. peps.python.org/pep-3156/. - PEP 654 — Exception Groups and except*. peps.python.org/pep-0654/. **Primary source:** docs.python.org/3/library/asyncio.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python logging module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/logging/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Replace print() with a proper logging system — levels, file rotation, structured output, and zero-cost debug messages.* ## CANONICAL DEFINITION The Python logging module provides a flexible, hierarchical logging system — named loggers, five severity levels (DEBUG through CRITICAL), pluggable handlers (console, file, rotating), formatters, and centralised configuration via dictConfig. ## BEGINNER > One sentence `logging` is the right way to record what your program is doing — it beats `print()` because you can control how much detail you see, where messages go, and how they are formatted, all without changing the code that generates them. ## Basic usage ```python import logging # Configure the root logger (do this once at program startup) logging.basicConfig( level=logging.DEBUG, format="%(asctime)s — %(name)s — %(levelname)s — %(message)s", datefmt="%Y-%m-%d %H:%M:%S", ) # Five severity levels (lowest to highest) logging.debug("Detailed diagnostic info — usually too verbose for normal use") logging.info("Normal operation confirmation: server started on port 8080") logging.warning("Something unexpected but not an error: disk 80% full") logging.error("Something failed: could not connect to database") logging.critical("Severe failure: application cannot continue") # Output: # 2026-06-01 14:30:00 — root — DEBUG — Detailed diagnostic... # 2026-06-01 14:30:00 — root — INFO — Normal operation... ``` ## Named loggers ```python import logging # Best practice: use __name__ as logger name # This gives each module its own logger: "myapp.database", "myapp.server" logger = logging.getLogger(__name__) def connect_to_db(host, port): logger.info("Connecting to %s:%d", host, port) # lazy string formatting try: # ... connection logic ... logger.debug("Connection established, pool size: %d", 10) except Exception as e: logger.error("Connection failed: %s", e, exc_info=True) # include traceback raise # In libraries: add NullHandler to avoid "No handler found" warning # (let the application configure handlers) logger.addHandler(logging.NullHandler()) ``` ## Handlers and formatters ```python import logging from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler # Create logger logger = logging.getLogger("myapp") logger.setLevel(logging.DEBUG) # Console handler — INFO and above console = logging.StreamHandler() console.setLevel(logging.INFO) console.setFormatter(logging.Formatter("%(levelname)s — %(message)s")) # File handler — DEBUG and above, rotates at 10MB, keeps 5 backups file_handler = RotatingFileHandler( "app.log", maxBytes=10*1024*1024, backupCount=5 ) file_handler.setLevel(logging.DEBUG) file_handler.setFormatter(logging.Formatter( "%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:%(lineno)d — %(message)s" )) logger.addHandler(console) logger.addHandler(file_handler) logger.info("Server started") # → console + file logger.debug("Config loaded") # → file only (below console level) # TimedRotatingFileHandler — rotate daily at midnight daily = TimedRotatingFileHandler("daily.log", when="midnight", backupCount=30) ``` ✅ Beginner tab complete — check your understanding - I know the five log levels in order: DEBUG < INFO < WARNING < ERROR < CRITICAL - I can set up basic logging with logging.basicConfig(level=logging.DEBUG) - I use logging.debug() for development details, logging.info() for normal operations, logging.warning() for unexpected-but-recoverable situations, logging.error() for real errors - I know that the default level is WARNING — DEBUG and INFO messages are silenced unless you change it Continue to [pathlib →](/coding/languages/python/standard-library/pathlib/) ## INTERMEDIATE ## Logging hierarchy and propagation Loggers form a hierarchy based on their names — dot-separated like Python modules. `"myapp.database"` is a child of `"myapp"`, which is a child of the root logger. When a logger decides a message should be emitted, it passes it to its handlers, then propagates it up to the parent logger (unless `propagate = False`). The root logger is the ancestor of all loggers. This means you can set up one handler on the root logger and all child loggers will use it — or configure specific loggers to handle their own output differently. ```python import logging # Root logger — ancestor of all root = logging.getLogger() # Module-level loggers db_logger = logging.getLogger("myapp.database") api_logger = logging.getLogger("myapp.api") # Only configure the root — children propagate up to it logging.basicConfig(level=logging.WARNING) # Override level for specific module db_logger.setLevel(logging.DEBUG) # database logs more verbosely # Prevent propagation for a noisy library noisy_lib = logging.getLogger("noisy_library") noisy_lib.propagate = False # stop messages going to root noisy_lib.addHandler(logging.NullHandler()) # swallow all ``` ## Structured logging and dictConfig ```python import logging import logging.config import json # Configure via dict (better than basicConfig for production) LOGGING_CONFIG = { "version": 1, "disable_existing_loggers": False, "formatters": { "standard": {"format": "%(asctime)s %(levelname)s %(name)s: %(message)s"}, "json": {"()": "pythonjsonlogger.jsonlogger.JsonFormatter"}, # pip install }, "handlers": { "console": { "class": "logging.StreamHandler", "formatter": "standard", "level": "INFO", }, "file": { "class": "logging.handlers.RotatingFileHandler", "filename": "app.log", "maxBytes": 10485760, "backupCount": 5, "formatter": "standard", "level": "DEBUG", }, }, "root": {"level": "DEBUG", "handlers": ["console", "file"]}, } logging.config.dictConfig(LOGGING_CONFIG) # Log with extra context logger = logging.getLogger(__name__) logger.info("User login", extra={"user_id": 42, "ip": "192.168.1.1"}) ``` **Commonly confused:** - Do not use print() for diagnostic output in production code. print() always goes to stdout, cannot be filtered by level, has no timestamps or module names, cannot be redirected to files or external systems, and must be removed or commented out to silence. logging.debug() costs nothing if the level is INFO or above — the string is never even formatted. - Use %s formatting in log calls, not f-strings. logger.debug("User: %s", user) is lazy — the string is only formatted if the message will actually be emitted. logger.debug(f"User: {user}") always formats the string, even if the DEBUG level is disabled. Use % formatting in logging calls. How this connects Requires first [Functions](/coding/languages/python/functions-and-scope/) Primary source [docs.python.org/3/library/logging.html](https://docs.python.org/3/library/logging.html) ✅ Intermediate tab complete — check your understanding - I always use logging.getLogger(__name__) instead of the root logger - I can add a FileHandler to write logs to a file - I can set a Formatter to control the timestamp, level, and message format - I understand logger hierarchy: "myapp.module" is a child of "myapp" Continue to [pathlib →](/coding/languages/python/standard-library/pathlib/) ## EXPERT ## Logging architecture: LogRecord, Filter, Handler, Formatter When you call `logger.info("msg")`, the logging system creates a `LogRecord` object containing: name, levelno, pathname, lineno, msg, args, exc_info, func, and created (timestamp). The logger checks if the record's level meets its own `effectiveLevel` (walking up the hierarchy until a level is found). If it passes, each handler checks its own level filter, then applies its `Filter` chain, then its `Formatter` to produce the final string, then emits. The `Formatter` uses `%`-style formatting with LogRecord attributes as the dictionary. Custom `Formatter` subclasses can produce structured output (JSON, etc.). ## logging.config and fileConfig vs dictConfig The older `logging.config.fileConfig()` reads INI-format config files. The modern `logging.config.dictConfig()` (Python 3.2+) accepts a dictionary — easily sourced from JSON, YAML, or in-code dicts. The `disable_existing_loggers` key (default `True` in fileConfig, should usually be `False` in dictConfig) controls whether loggers created before `dictConfig()` is called are disabled — a common source of "my library's logger suddenly stopped working" bugs. ✅ Expert tab complete - I can configure logging with a dictConfig dictionary for clean, maintainable setup - I can output structured JSON logs for log aggregation systems - I understand the LogRecord pipeline: Logger → Handler → Formatter → output Continue to [pathlib →](/coding/languages/python/standard-library/pathlib/) ## SOURCES - Python Standard Library — logging. docs.python.org/3/library/logging.html. - Python logging HOWTO. docs.python.org/3/howto/logging.html. - Python logging Cookbook. docs.python.org/3/howto/logging-cookbook.html. - Python Standard Library — logging.handlers. docs.python.org/3/library/logging.handlers.html. **Primary source:** docs.python.org/3/library/logging.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python unittest module ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/unittest/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *Automated tests that verify your code works — and keep working — as you change it.* ## CANONICAL DEFINITION The Python unittest module is the built-in testing framework — TestCase subclasses with assertion methods, setUp/tearDown lifecycle, unittest.mock for replacing dependencies, and a test runner for automated test discovery and execution. ## BEGINNER > One sentence `unittest` is Python's built-in testing framework — it lets you write automated tests that verify your code does what you expect, and run them all with one command to catch regressions before they reach production. ## Writing your first tests ```python import unittest # The code we're testing def add(a, b): return a + b def divide(a, b): if b == 0: raise ZeroDivisionError("Cannot divide by zero") return a / b # Tests live in a class that inherits from unittest.TestCase class TestMathFunctions(unittest.TestCase): def test_add_positive_numbers(self): self.assertEqual(add(2, 3), 5) def test_add_negative_numbers(self): self.assertEqual(add(-1, -2), -3) def test_add_zero(self): self.assertEqual(add(5, 0), 5) def test_divide_normal(self): self.assertAlmostEqual(divide(10, 3), 3.333, places=3) def test_divide_by_zero(self): with self.assertRaises(ZeroDivisionError): divide(10, 0) def test_divide_by_zero_message(self): with self.assertRaises(ZeroDivisionError) as ctx: divide(10, 0) self.assertIn("Cannot divide", str(ctx.exception)) # Run tests from command line: python -m unittest test_basics.py # Or discover all tests: python -m unittest discover if __name__ == "__main__": unittest.main() ``` ## Key assertion methods | Method | Tests that | | --- | --- | | `assertEqual(a, b)` | a == b | | `assertNotEqual(a, b)` | a != b | | `assertTrue(x)` | bool(x) is True | | `assertFalse(x)` | bool(x) is False | | `assertIs(a, b)` | a is b (identity) | | `assertIsNone(x)` | x is None | | `assertIn(a, b)` | a in b | | `assertIsInstance(a, T)` | isinstance(a, T) | | `assertRaises(Exc)` | block raises Exc | | `assertAlmostEqual(a, b)` | round(a-b, 7) == 0 | | `assertGreater(a, b)` | a > b | | `assertRegex(s, r)` | re.search(r, s) | ## setUp and tearDown ```python import unittest import tempfile import os class TestWithSetup(unittest.TestCase): def setUp(self): # Runs BEFORE each test method self.temp_dir = tempfile.mkdtemp() self.test_file = os.path.join(self.temp_dir, "test.txt") with open(self.test_file, "w") as f: f.write("test content") def tearDown(self): # Runs AFTER each test method — even if test fails import shutil shutil.rmtree(self.temp_dir) def test_file_exists(self): self.assertTrue(os.path.exists(self.test_file)) def test_file_content(self): with open(self.test_file) as f: content = f.read() self.assertEqual(content, "test content") @classmethod def setUpClass(cls): # Runs once BEFORE all tests in this class cls.db = connect_to_test_database() @classmethod def tearDownClass(cls): # Runs once AFTER all tests in this class cls.db.close() ``` ✅ Beginner tab complete — check your understanding - I know at least 8 assertion methods beyond assertEqual - I know setUp runs before EACH test method, not once before the class - I can run tests with python -m unittest discover to find all test files automatically Stage 3 complete. Continue to [Stage 4: Concepts →](/coding/courses/python/#stage-4) ## INTERMEDIATE ## Mocking with unittest.mock Tests should be isolated — they should not make real network requests, write to real databases, or depend on external services. `unittest.mock` lets you replace these dependencies with controllable fakes during tests. ```python from unittest import TestCase from unittest.mock import patch, MagicMock, call def get_user_name(user_id): import requests response = requests.get(f"https://api.example.com/users/{user_id}") return response.json()["name"] class TestGetUserName(TestCase): @patch("requests.get") # replace requests.get with a Mock def test_get_user_name(self, mock_get): # Configure what the mock returns mock_response = MagicMock() mock_response.json.return_value = {"name": "Priya", "id": 1} mock_get.return_value = mock_response result = get_user_name(1) # Assert return value self.assertEqual(result, "Priya") # Assert mock was called correctly mock_get.assert_called_once_with("https://api.example.com/users/1") @patch("requests.get", side_effect=ConnectionError("No internet")) def test_handles_network_error(self, mock_get): with self.assertRaises(ConnectionError): get_user_name(1) def test_mock_multiple_calls(self): mock = MagicMock() mock.side_effect = [10, 20, 30] # returns these in order self.assertEqual(mock(), 10) self.assertEqual(mock(), 20) self.assertEqual(mock(), 30) ``` ## Parametrised tests and subTest ```python import unittest def is_prime(n): if n < 2: return False for i in range(2, int(n**0.5) + 1): if n % i == 0: return False return True class TestIsPrime(unittest.TestCase): def test_primes(self): # subTest — each case shown separately on failure prime_cases = [2, 3, 5, 7, 11, 13, 17, 19, 23] for n in prime_cases: with self.subTest(n=n): self.assertTrue(is_prime(n), f"{n} should be prime") def test_non_primes(self): non_prime_cases = [0, 1, 4, 6, 8, 9, 10, 15, 25] for n in non_prime_cases: with self.subTest(n=n): self.assertFalse(is_prime(n), f"{n} should not be prime") @unittest.skip("not implemented yet") def test_large_primes(self): pass @unittest.skipIf(True, "always skipped") def test_conditional(self): pass ``` **Commonly confused:** - Test method names must start with test. The test runner discovers methods by looking for names beginning with test. A method named check_something() will not run. This is a common source of tests that appear to pass but never actually ran. - assertEqual is not the same as assertTrue(a == b). assertEqual produces a better error message — it shows you the actual and expected values. assertTrue(a == b) just says "False is not True." Always use the most specific assertion method available. How this connects Requires first [Functions](/coding/languages/python/functions-and-scope/)[Error Handling](/coding/languages/python/error-handling/) Primary source [docs.python.org/3/library/unittest.html](https://docs.python.org/3/library/unittest.html) ✅ Intermediate tab complete — check your understanding - I know the difference between Mock (basic) and MagicMock (also mocks dunder methods) - I can use @patch as a decorator to replace a dependency for the duration of a test - I can inspect mock.call_args_list to see every call that was made - I remember: patch where it is used, not where it is defined Stage 3 complete. Continue to [Stage 4: Concepts →](/coding/courses/python/#stage-4) ## EXPERT ## unittest architecture: TestCase, TestSuite, TestRunner The unittest framework has four core components. `TestCase`: the base class for individual test classes; defines assertions and lifecycle methods. `TestSuite`: a collection of test cases and suites — allows grouping and ordering tests. `TestLoader`: discovers and loads tests from modules, classes, or methods — `unittest.TestLoader().loadTestsFromModule(module)`. `TestRunner`: executes the suite and reports results — the default is `TextTestRunner`; CI systems use custom runners (JUnit XML, HTML reports). The `TestResult` object accumulates results — successes, failures, errors, skips — and is passed to runners. ## Test isolation and the test doubles taxonomy Gerard Meszaros's *xUnit Test Patterns* (2007) defines the taxonomy: **Dummy** — passed but never used (placeholder). **Stub** — returns hardcoded values. **Spy** — records calls made to it. **Mock** — a spy with expectations; fails if not used as expected. **Fake** — a working implementation simplified for testing (e.g., in-memory database). Python's `MagicMock` is a spy/mock hybrid — it records calls and can assert on them, but does not automatically fail if assertions are not made (unlike Java Mockito). ✅ Expert tab complete - I know the difference between a stub (returns canned data), a mock (verifies calls), and a fake (working implementation) - I can build a custom TestSuite to run specific tests in a specific order Stage 3 complete. Continue to [Stage 4: Concepts →](/coding/courses/python/#stage-4) ## SOURCES - Python Standard Library — unittest. docs.python.org/3/library/unittest.html. - Python Standard Library — unittest.mock. docs.python.org/3/library/unittest.mock.html. - Meszaros, G. (2007). xUnit Test Patterns. Addison-Wesley. **Primary source:** docs.python.org/3/library/unittest.html *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # functools — Higher-Order Functions ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/functools/ **Type:** standard-library | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The functools module provides tools for working with functions: caching with lru_cache and cache, partial application with partial, reducing with reduce, and decorator-writing helpers like wraps.* ## CANONICAL DEFINITION functools is a standard library module for higher-order functions — functions that act on or return other functions. Its most-used members are lru_cache/cache (memoisation decorators), partial (partial application), reduce (fold a sequence to a single value), and wraps (preserve metadata when writing decorators). ## BEGINNER > The idea functools is a toolbox for working with functions themselves. The two tools you'll reach for most: `@lru_cache` to make slow functions fast by remembering results, and `partial` to create a new function with some arguments pre-filled. ## What functools provides functools is part of the standard library — no installation needed. It contains higher-order functions: tools that take functions as arguments or return new functions. | Tool | What it does | | --- | --- | | `@lru_cache` | Cache a function's results (memoisation) | | `@cache` | Simpler unbounded cache (Python 3.9+) | | `partial()` | Create a new function with some arguments fixed | | `reduce()` | Combine a sequence into a single value | | `@wraps` | Preserve metadata when writing decorators | | `@singledispatch` | Function overloading based on argument type | | `cmp_to_key()` | Convert old-style comparison functions to keys | ## Caching with lru_cache and cache The single most impactful functools tool: `@lru_cache` remembers the results of function calls. When the function is called again with the same arguments, the cached result is returned instantly instead of recomputing. ```python from functools import lru_cache, cache # Without caching — fibonacci is exponentially slow def fib_slow(n): if n < 2: return n return fib_slow(n-1) + fib_slow(n-2) # fib_slow(35) makes ~30 million calls — takes seconds # With caching — each value computed once @lru_cache(maxsize=None) # None = unbounded cache def fib_fast(n): if n < 2: return n return fib_fast(n-1) + fib_fast(n-2) # fib_fast(35) makes 36 calls — instant print(fib_fast(100)) # 354224848179261915075 — instant # @cache (Python 3.9+) is shorthand for @lru_cache(maxsize=None) @cache def expensive_lookup(key): print(f"Computing for {key}...") return key ** 2 print(expensive_lookup(5)) # Computing for 5... → 25 print(expensive_lookup(5)) # 25 (no "Computing" — cached) # Inspect cache performance print(fib_fast.cache_info()) # CacheInfo(hits=98, misses=101, maxsize=None, currsize=101) # Clear the cache if needed fib_fast.cache_clear() ``` > Cached functions need hashable arguments `@lru_cache` stores results in a dict keyed by the arguments — so all arguments must be hashable. You can cache `f(1, "x")` but not `f([1,2,3])` because lists aren't hashable. Pass tuples instead of lists for cacheable functions. ✅ Beginner tab complete - I can apply @lru_cache or @cache to memoise an expensive function - I know cached functions need hashable arguments (tuples, not lists) - I can inspect cache performance with func.cache_info() - I know @cache (3.9+) is shorthand for @lru_cache(maxsize=None) Continue to [csv →](/coding/languages/python/standard-library/csv/) ## INTERMEDIATE ## partial application and reduce `partial` creates a new function with some arguments already filled in. `reduce` combines all elements of a sequence into a single value by repeatedly applying a function. ```python from functools import partial, reduce # partial — fix some arguments to create a specialised function def power(base, exponent): return base ** exponent square = partial(power, exponent=2) # exponent is now fixed at 2 cube = partial(power, exponent=3) print(square(5)) # 25 print(cube(3)) # 27 # Real-world: partial for callbacks with preset config def log(level, message): print(f"[{level}] {message}") info = partial(log, "INFO") error = partial(log, "ERROR") info("Server started") # [INFO] Server started error("Connection lost") # [ERROR] Connection lost # reduce — fold a sequence into one value numbers = [1, 2, 3, 4, 5] total = reduce(lambda acc, x: acc + x, numbers) # ((((1+2)+3)+4)+5) print(total) # 15 product = reduce(lambda acc, x: acc * x, numbers) # 5! = 120 print(product) # 120 # reduce with an initial value total = reduce(lambda acc, x: acc + x, numbers, 100) # starts at 100 print(total) # 115 # Note: for sum/product, prefer the built-ins — they're clearer print(sum(numbers)) # 15 — clearer than reduce import math print(math.prod(numbers)) # 120 — clearer than reduce ``` ## singledispatch — function overloading by type `@singledispatch` lets you write a function that behaves differently based on the type of its first argument — Python's version of function overloading. ```python from functools import singledispatch @singledispatch def describe(value): """Default implementation.""" return f"Unknown type: {value}" @describe.register def _(value: int): return f"Integer: {value} (binary: {value:b})" @describe.register def _(value: str): return f"String of length {len(value)}: {value!r}" @describe.register def _(value: list): return f"List with {len(value)} items" print(describe(42)) # Integer: 42 (binary: 101010) print(describe("hello")) # String of length 5: 'hello' print(describe([1, 2, 3])) # List with 3 items print(describe(3.14)) # Unknown type: 3.14 (default) ``` **Commonly confused:** - lru_cache on methods can leak memory. Applying @lru_cache to an instance method keeps a reference to self in the cache, preventing the instance from being garbage collected. For methods, consider functools.cached_property (for properties) or a different caching strategy. ✅ Intermediate tab complete - I can use partial() to fix some arguments and create a specialised function - I can use reduce() to combine a sequence into a single value - I know to prefer sum()/math.prod() over reduce() when they apply - I can use @singledispatch for type-based overloading Continue to [csv →](/coding/languages/python/standard-library/csv/) ## EXPERT ## cached_property, total_ordering, and the wraps protocol functools includes several class-oriented tools. `@cached_property` computes a property once and caches it on the instance. `@total_ordering` generates all comparison methods from just `__eq__` and one other. `@wraps` (covered fully on the Decorators page) copies the wrapped function's metadata. ```python from functools import cached_property, total_ordering class Dataset: def __init__(self, data): self.data = data @cached_property def statistics(self): """Computed once, then cached on the instance.""" print("Computing statistics...") import statistics return { "mean": statistics.mean(self.data), "stdev": statistics.stdev(self.data), } ds = Dataset([1, 2, 3, 4, 5]) print(ds.statistics) # Computing statistics... → {...} print(ds.statistics) # {...} — no recomputation, cached on instance # @total_ordering — define __eq__ and one comparison, get the rest free @total_ordering class Version: def __init__(self, major, minor): self.major, self.minor = major, minor def __eq__(self, other): return (self.major, self.minor) == (other.major, other.minor) def __lt__(self, other): return (self.major, self.minor) = v2) # False — generated from __eq__ and __lt__ print(v1 <= v2) # True — generated ``` The implementation of `lru_cache` uses a circular doubly-linked list plus a dict to implement the Least-Recently-Used eviction policy in O(1) per access. When the cache reaches `maxsize`, the least recently used entry is evicted. With `maxsize=None` (or `@cache`), no eviction occurs — the cache grows without bound, so use it only when the set of distinct argument combinations is finite. ✅ Expert tab complete - I can use @cached_property for a compute-once instance attribute - I can use @total_ordering to generate comparison methods from __eq__ and __lt__ - I know lru_cache uses a doubly-linked list + dict for O(1) LRU eviction Continue to [csv →](/coding/languages/python/standard-library/csv/) ## SOURCES - Python Standard Library — functools. docs.python.org/3/library/functools.html. - PEP 443 — Single-dispatch generic functions. peps.python.org/pep-0443/. - Python Standard Library — functools.lru_cache and cache. docs.python.org/3/library/functools.html#functools.lru_cache. - Python Standard Library — functools.cached_property. docs.python.org/3/library/functools.html#functools.cached_property. **Primary source:** docs.python.org functools *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # csv — Reading and Writing CSV Files ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/csv/ **Type:** standard-library | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The csv module reads and writes comma-separated values files correctly — handling quoting, escaping, different delimiters, and the edge cases that break naive string splitting.* ## CANONICAL DEFINITION csv is a standard library module for reading and writing CSV (Comma-Separated Values) files. It correctly handles quoted fields, embedded commas and newlines, custom delimiters, and dialects. The DictReader and DictWriter classes map rows to and from dictionaries using the header row. ## BEGINNER > The idea CSV files look simple — values separated by commas — but they have tricky edge cases: a field might contain a comma inside quotes, or a newline. The csv module handles all of this correctly so you don't have to. Never parse CSV by splitting on commas yourself. ## Reading a CSV file The `csv.reader` object reads a file row by row, giving each row as a list of strings. Always open CSV files with `newline=""` — this is required for the csv module to handle line endings correctly across platforms. ```python import csv # A CSV file: data.csv # name,age,city # Priya,30,Mumbai # Arjun,28,"New Delhi, India" <- note the comma inside quotes with open("data.csv", newline="", encoding="utf-8") as f: reader = csv.reader(f) for row in reader: print(row) # ['name', 'age', 'city'] # ['Priya', '30', 'Mumbai'] # ['Arjun', '28', 'New Delhi, India'] <- comma handled correctly # Skip the header row with open("data.csv", newline="", encoding="utf-8") as f: reader = csv.reader(f) header = next(reader) # read the first row separately for row in reader: name, age, city = row print(f"{name} is {age}") ``` > Always use newline="" when opening CSV files The csv module handles its own line-ending translation. If you open without `newline=""`, Python's universal newline translation interferes, causing blank rows to appear on Windows. This is the single most common csv bug. `open("file.csv", newline="", encoding="utf-8")` — every time. ## Writing a CSV file The `csv.writer` object writes rows to a file, automatically quoting fields that contain commas, quotes, or newlines. ```python import csv rows = [ ["name", "age", "city"], ["Priya", 30, "Mumbai"], ["Arjun", 28, "New Delhi, India"], # comma will be auto-quoted ] with open("output.csv", "w", newline="", encoding="utf-8") as f: writer = csv.writer(f) writer.writerows(rows) # write all rows at once # or write one at a time: # for row in rows: # writer.writerow(row) # output.csv contains: # name,age,city # Priya,30,Mumbai # Arjun,28,"New Delhi, India" <- automatically quoted ``` ✅ Beginner tab complete - I always open CSV files with newline="" and encoding="utf-8" - I never parse CSV by splitting on commas myself - I can read rows with csv.reader and skip the header with next() - I can write rows with csv.writer and writerows() Continue to [subprocess →](/coding/languages/python/standard-library/subprocess/) ## INTERMEDIATE ## DictReader and DictWriter The most convenient way to work with CSV files that have a header row. `DictReader` gives each row as a dictionary keyed by the column names; `DictWriter` writes dictionaries back to CSV. ```python import csv # DictReader — each row is a dict keyed by the header with open("data.csv", newline="", encoding="utf-8") as f: reader = csv.DictReader(f) for row in reader: # row is {'name': 'Priya', 'age': '30', 'city': 'Mumbai'} print(f"{row['name']} lives in {row['city']}") # Priya lives in Mumbai # Arjun lives in New Delhi, India # DictWriter — write dicts to CSV people = [ {"name": "Priya", "age": 30, "city": "Mumbai"}, {"name": "Arjun", "age": 28, "city": "Delhi"}, ] with open("output.csv", "w", newline="", encoding="utf-8") as f: fieldnames = ["name", "age", "city"] writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() # write the header row writer.writerows(people) # write all the dicts # Note: all values are read as STRINGS — convert types yourself with open("data.csv", newline="", encoding="utf-8") as f: reader = csv.DictReader(f) total_age = sum(int(row["age"]) for row in reader) # int() needed print(f"Total age: {total_age}") ``` ## Dialects and custom delimiters Not all "CSV" files use commas. Tab-separated (TSV), semicolon-separated (common in European locales), and pipe-separated files are handled by specifying the delimiter or a dialect. ```python import csv # Tab-separated values with open("data.tsv", newline="", encoding="utf-8") as f: reader = csv.reader(f, delimiter="\t") for row in reader: print(row) # Semicolon-separated (common in European Excel exports) with open("data_eu.csv", newline="", encoding="utf-8") as f: reader = csv.reader(f, delimiter=";") for row in reader: print(row) # Control quoting behaviour with open("output.csv", "w", newline="", encoding="utf-8") as f: # QUOTE_ALL — quote every field writer = csv.writer(f, quoting=csv.QUOTE_ALL) writer.writerow(["a", "b", "c"]) # "a","b","c" # Register a custom dialect for reuse csv.register_dialect("pipes", delimiter="|", quoting=csv.QUOTE_MINIMAL) with open("data.txt", newline="", encoding="utf-8") as f: reader = csv.reader(f, dialect="pipes") for row in reader: print(row) # Sniffer — auto-detect the dialect with open("unknown.csv", newline="", encoding="utf-8") as f: sample = f.read(1024) f.seek(0) dialect = csv.Sniffer().sniff(sample) reader = csv.reader(f, dialect) ``` **Commonly confused:** - CSV values are always strings. csv.reader and DictReader return every field as a string — including numbers. row["age"] is "30", not 30. You must convert: int(row["age"]). The csv module does not infer types. ✅ Intermediate tab complete - I can use DictReader to access fields by column name: row["name"] - I can use DictWriter with writeheader() and writerows() - I remember that all CSV values are strings — I convert types myself with int(), float() - I can handle TSV and semicolon-delimited files with the delimiter parameter Continue to [subprocess →](/coding/languages/python/standard-library/subprocess/) ## EXPERT ## The CSV format problem and RFC 4180 CSV is deceptively complex because it was never formally standardised until RFC 4180 (2005) — and even that is only a guideline that many tools ignore. The csv module exists precisely because correct CSV parsing requires a state machine, not a string split. Consider the edge cases: a field containing a comma must be quoted; a field containing a quote must escape it by doubling; a field can contain a newline if quoted. ```python import csv, io # These edge cases break naive line.split(",") parsing. # A real CSV file on disk would contain (between the dashes): # --- # name,quote,note # Priya,"She said ""hello""","Line 1 # Line 2" # Arjun,"Comma, inside",normal # --- # Field 2 has escaped quotes; field 3 spans two physical lines; # Arjun's quote field contains a comma. The csv module handles all three. with open("tricky.csv", newline="", encoding="utf-8") as f: reader = csv.reader(f) for row in reader: print(row) # ['name', 'quote', 'note'] # ['Priya', 'She said "hello"', 'Line 1\nLine 2'] <- embedded newline # ['Arjun', 'Comma, inside', 'normal'] <- embedded comma# ['Priya', 'She said "hello"', 'Line 1\nLine 2'] <- embedded newline! # ['Arjun', 'Comma, inside', 'normal'] <- embedded comma! # Why newline="" matters — the csv module needs to see raw \r\n # to distinguish line endings WITHIN quoted fields from row separators. # Without newline="", Python translates \r\n to \n before csv sees it, # corrupting quoted fields that legitimately contain \r\n. # Performance: csv.reader is implemented in C (_csv module) # It processes millions of rows per second — far faster than # any pure-Python parser. For a 1GB CSV, stream it row by row: def process_large_csv(path): with open(path, newline="", encoding="utf-8") as f: reader = csv.DictReader(f) for row in reader: # one row at a time, constant memory yield process(row) ``` The csv module's reader and writer are implemented in C (the `_csv` extension module) for performance. The Python-level `csv` module wraps this with the convenient DictReader/DictWriter classes and dialect management. This is why csv.reader can process millions of rows per second while a hand-written Python parser would be orders of magnitude slower. ✅ Expert tab complete - I know CSV fields can contain commas, quotes, and newlines when quoted - I understand why newline="" is required (csv handles line endings itself) - I know csv.reader is implemented in C and can process millions of rows per second Continue to [subprocess →](/coding/languages/python/standard-library/subprocess/) ## SOURCES - Python Standard Library — csv. docs.python.org/3/library/csv.html. - PEP 305 — CSV File API. peps.python.org/pep-0305/. - RFC 4180 — Common Format and MIME Type for CSV Files. rfc-editor.org/rfc/rfc4180. - Python Standard Library — csv.DictReader and csv.DictWriter. docs.python.org/3/library/csv.html#csv.DictReader. **Primary source:** docs.python.org csv *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # subprocess — Running External Programs ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/subprocess/ **Type:** standard-library | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The subprocess module runs external commands and programs from Python — capturing their output, sending input, checking exit codes, and managing pipelines safely.* ## CANONICAL DEFINITION subprocess is a standard library module for spawning new processes, connecting to their input/output/error pipes, and obtaining their return codes. subprocess.run() is the recommended high-level API for most use cases. It replaces older functions like os.system() and os.popen(). ## BEGINNER > The idea subprocess lets your Python program run other programs — like running a terminal command from inside your code. Want to run `git status`, call `ffmpeg`, or execute a shell script? subprocess is how. `subprocess.run()` is the one function to learn first. ## subprocess.run() — the main API The `subprocess.run()` function (Python 3.5+) is the recommended way to run external commands. Pass the command as a list of strings — the program name first, then each argument as a separate list item. ```python import subprocess # Run a command — arguments as a list result = subprocess.run(["echo", "Hello from subprocess"]) # Hello from subprocess (printed to terminal) # Check the exit code result = subprocess.run(["ls", "/tmp"]) print(result.returncode) # 0 means success, non-zero means error # Run git status (in a git repository) result = subprocess.run(["git", "status"]) # IMPORTANT: pass arguments as separate list items, NOT one string # RIGHT: subprocess.run(["ls", "-l", "/home"]) # WRONG (unless shell=True): # subprocess.run("ls -l /home") <- this looks for a program # literally named "ls -l /home" # Raise an exception if the command fails (check=True) try: subprocess.run(["false"], check=True) # 'false' exits with code 1 except subprocess.CalledProcessError as e: print(f"Command failed with code {e.returncode}") ``` ## Capturing output By default, a subprocess's output goes to the terminal. To capture it in your Python program, pass `capture_output=True` and `text=True`. ```python import subprocess # Capture stdout as text result = subprocess.run( ["git", "branch", "--show-current"], capture_output=True, # capture stdout and stderr text=True, # decode bytes to str (instead of raw bytes) ) print(f"Current branch: {result.stdout.strip()}") # Capture and process output result = subprocess.run( ["ls", "-1", "/usr/bin"], capture_output=True, text=True, ) programs = result.stdout.splitlines() print(f"Found {len(programs)} programs in /usr/bin") # stdout and stderr are separate result = subprocess.run( ["python3", "-c", "import sys; print('out'); print('err', file=sys.stderr)"], capture_output=True, text=True, ) print(f"stdout: {result.stdout}") # out print(f"stderr: {result.stderr}") # err # Get output as a simple string (convenience for capture + check) output = subprocess.check_output(["date"], text=True) print(output.strip()) ``` ✅ Beginner tab complete - I can run a command with subprocess.run(["program", "arg1", "arg2"]) - I pass arguments as separate list items, NOT as one string - I can capture output with capture_output=True, text=True - I can check success with result.returncode or check=True Stage 3 stdlib expansion complete for this batch. Continue the [Python Course →](/coding/courses/python/) ## INTERMEDIATE ## Sending input and building pipelines You can send data to a program's standard input via the `input` parameter, set a timeout, and even chain programs together as a pipeline. ```python import subprocess # Send input to a program's stdin result = subprocess.run( ["grep", "error"], input="line 1\nerror line\nline 3\n", # fed to stdin capture_output=True, text=True, ) print(result.stdout) # error line # Set a timeout — kill the process if it runs too long try: subprocess.run(["sleep", "10"], timeout=2) except subprocess.TimeoutExpired: print("Command timed out after 2 seconds") # Set the working directory and environment result = subprocess.run( ["pwd"], cwd="/tmp", # run in this directory capture_output=True, text=True, ) print(result.stdout.strip()) # /tmp import os custom_env = {**os.environ, "MY_VAR": "value"} subprocess.run(["printenv", "MY_VAR"], env=custom_env) # Pipelines — connect one program's output to another's input # Equivalent to: ls /usr/bin | grep python ls = subprocess.run(["ls", "/usr/bin"], capture_output=True, text=True) grep = subprocess.run( ["grep", "python"], input=ls.stdout, # feed ls output into grep capture_output=True, text=True, ) print(grep.stdout) # For true streaming pipelines, use Popen p1 = subprocess.Popen(["ls", "/usr/bin"], stdout=subprocess.PIPE) p2 = subprocess.Popen(["grep", "python"], stdin=p1.stdout, stdout=subprocess.PIPE, text=True) p1.stdout.close() # allow p1 to receive SIGPIPE if p2 exits output, _ = p2.communicate() print(output) ``` ## Security: shell=True and command injection The most important subprocess safety rule: avoid `shell=True` with untrusted input. It opens your program to command injection attacks. ```python import subprocess # DANGEROUS — command injection vulnerability filename = input("Enter filename: ") # user types: x.txt; rm -rf / subprocess.run(f"cat {filename}", shell=True) # NEVER DO THIS # The shell interprets ; rm -rf / as a second command! # SAFE — arguments as a list, no shell filename = input("Enter filename: ") subprocess.run(["cat", filename]) # filename is a single argument # Even "x.txt; rm -rf /" is treated as one (invalid) filename — safe # If you MUST use shell features (pipes, globbing, redirection in the # shell), sanitise input with shlex.quote — but prefer the list form: import shlex safe_arg = shlex.quote(filename) # The list form also handles spaces and special characters correctly: subprocess.run(["cat", "file with spaces.txt"]) # works perfectly # vs shell=True where you'd need careful quoting ``` **Commonly confused:** - text=True vs the default bytes. Without text=True, stdout and stderr are bytes objects (b"output"), not strings. Pass text=True (or its alias universal_newlines=True) to get decoded strings. This also enables universal newline handling. ✅ Intermediate tab complete - I can send input to a process with the input= parameter - I can set a timeout= to kill a hung process - I NEVER use shell=True with untrusted input (command injection risk) - I use the list form ["cmd", arg] which is injection-safe by default Continue the [Python Course →](/coding/courses/python/) ## EXPERT ## Popen, process lifecycle, and async subprocess Under the hood, `subprocess.run()` is a convenience wrapper around the `Popen` class, which provides fine-grained control over process creation and the process lifecycle. Use Popen directly when you need to interact with a long-running process, stream output as it's produced, or manage multiple processes concurrently. ```python import subprocess # Popen — start a process and interact with it while it runs proc = subprocess.Popen( ["ping", "-c", "5", "example.com"], stdout=subprocess.PIPE, text=True, ) # Stream output line by line as the process produces it for line in proc.stdout: print(f"Got: {line.strip()}") proc.wait() # wait for the process to finish print(f"Exit code: {proc.returncode}") # poll() — check if the process is done without blocking import time proc = subprocess.Popen(["sleep", "3"]) while proc.poll() is None: # None means still running print("Still running...") time.sleep(1) print("Done") # Async subprocess with asyncio import asyncio async def run_command(cmd): proc = await asyncio.create_subprocess_exec( *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) stdout, stderr = await proc.communicate() return stdout.decode() async def main(): # Run multiple commands concurrently results = await asyncio.gather( run_command(["echo", "first"]), run_command(["echo", "second"]), run_command(["echo", "third"]), ) print(results) asyncio.run(main()) ``` The process creation mechanism differs by platform: on POSIX systems, subprocess uses `fork()` followed by `exec()` (or the more efficient `posix_spawn()` where available); on Windows it uses `CreateProcess()`. The `communicate()` method is the safe way to interact with a process's pipes — it reads stdout and stderr concurrently, avoiding the deadlock that occurs if you naively read one pipe while the process blocks writing to the other. ✅ Expert tab complete - I can use Popen to start a process and stream its output while it runs - I know communicate() reads stdout and stderr concurrently to avoid deadlock - I can run subprocesses concurrently with asyncio.create_subprocess_exec Continue the [Python Course →](/coding/courses/python/) ## SOURCES - Python Standard Library — subprocess. docs.python.org/3/library/subprocess.html. - PEP 324 — subprocess - New process module. peps.python.org/pep-0324/. - Python Standard Library — subprocess.run and subprocess.Popen. docs.python.org/3/library/subprocess.html#subprocess.run. - Python Standard Library — Security Considerations. docs.python.org/3/library/subprocess.html#security-considerations. **Primary source:** docs.python.org subprocess *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # argparse — Command-Line Argument Parsing ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/argparse/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *argparse turns command-line arguments into a clean, validated Python interface — with automatic help text, type conversion, and error messages.* ## CANONICAL DEFINITION argparse is the standard library module for parsing command-line arguments. You declare the arguments your program accepts; argparse parses sys.argv, converts and validates values, generates --help output, and reports usage errors automatically. ## BEGINNER > The idea When you run a program like `python backup.py /data --verbose`, argparse is what reads `/data` and `--verbose` and hands them to your code as clean Python values — while writing the help screen and error messages for you. ## Your first parser Three steps: create a parser, add the arguments you accept, then parse. argparse handles the rest — including `-h/--help` automatically. ```python import argparse parser = argparse.ArgumentParser(description="Greet a user by name.") parser.add_argument("name", help="the name to greet") parser.add_argument("--shout", action="store_true", help="use uppercase") args = parser.parse_args() greeting = f"Hello, {args.name}!" if args.shout: greeting = greeting.upper() print(greeting) ``` ```python $ python greet.py Priya Hello, Priya! $ python greet.py Priya --shout HELLO, PRIYA! $ python greet.py # missing required argument usage: greet.py [-h] [--shout] name greet.py: error: the following arguments are required: name $ python greet.py --help # generated automatically usage: greet.py [-h] [--shout] name Greet a user by name. positional arguments: name the name to greet options: -h, --help show this help message and exit --shout use uppercase ``` ## Positional vs optional arguments ```python import argparse parser = argparse.ArgumentParser() # Positional — required, identified by position parser.add_argument("source") parser.add_argument("destination") # Optional — identified by name (starts with - or --), not required by default parser.add_argument("-v", "--verbose", action="store_true") parser.add_argument("--retries", default=3) # default if not given args = parser.parse_args(["in.txt", "out.txt", "--verbose"]) print(args.source) # in.txt print(args.destination) # out.txt print(args.verbose) # True print(args.retries) # 3 (the default) # Note: --retries becomes args.retries (dashes -> underscores) ``` > Dashes become underscores An option named `--dry-run` is accessed as `args.dry_run`. argparse converts dashes to underscores so the attribute name is a valid Python identifier. ✅ Beginner tab complete - I can create an ArgumentParser and add an argument - I know positional args are required; optional args (--name) are not by default - I know action="store_true" makes a flag - I know dashes in --dry-run become args.dry_run Continue to [math →](/coding/languages/python/standard-library/math/) ## INTERMEDIATE ## Type conversion, choices, and nargs argparse can convert and validate values for you: convert to `int` or `float`, restrict to a set of `choices`, or collect multiple values with `nargs`. ```python import argparse parser = argparse.ArgumentParser() # type= converts and validates — bad input gives a clean error parser.add_argument("--count", type=int, default=1) parser.add_argument("--ratio", type=float) # choices= restricts allowed values parser.add_argument("--mode", choices=["fast", "safe", "debug"], default="safe") # nargs collects multiple values into a list parser.add_argument("--files", nargs="+") # one or more -> list parser.add_argument("--coords", nargs=2, type=float) # exactly two # required=True forces an optional argument to be given parser.add_argument("--output", required=True) args = parser.parse_args( ["--count", "5", "--mode", "fast", "--files", "a.txt", "b.txt", "--output", "result.txt"]) print(args.count) # 5 (an int, not "5") print(args.mode) # fast print(args.files) # ['a.txt', 'b.txt'] # Invalid type gives an automatic error: # $ prog --count abc # error: argument --count: invalid int value: 'abc' # Invalid choice: # $ prog --mode turbo # error: argument --mode: invalid choice: 'turbo' (choose from 'fast', 'safe', 'debug') ``` ## Subcommands (like git commit, git push) ```python import argparse parser = argparse.ArgumentParser(prog="tool") subparsers = parser.add_subparsers(dest="command", required=True) # "tool add " add = subparsers.add_parser("add", help="add an item") add.add_argument("item") # "tool remove --force" remove = subparsers.add_parser("remove", help="remove an item") remove.add_argument("item") remove.add_argument("--force", action="store_true") args = parser.parse_args(["add", "milk"]) print(args.command) # add print(args.item) # milk # Dispatch on the chosen subcommand if args.command == "add": print(f"Adding {args.item}") elif args.command == "remove": print(f"Removing {args.item}" + (" (forced)" if args.force else "")) ``` **Commonly confused:** - action="store_true" for flags. A flag like --verbose that is either present or absent uses action="store_true" — it becomes True when given, False otherwise. Without it, argparse expects a value after the flag. ✅ Intermediate tab complete - I can use type=int to convert and validate - I can restrict values with choices=[...] - I can collect multiple values with nargs - I can build subcommands with add_subparsers Continue to [math →](/coding/languages/python/standard-library/math/) ## EXPERT ## Custom types, argument groups, and parser composition Because `type=` accepts any callable that takes a string and returns a value (raising `ValueError` or `argparse.ArgumentTypeError` on failure), you can plug in arbitrary validation and conversion. Mutually exclusive groups enforce that only one of a set of options is given, and parent parsers let you share common arguments across multiple subcommands. ```python import argparse from pathlib import Path # Custom type — a callable str -> value def existing_file(s): p = Path(s) if not p.is_file(): raise argparse.ArgumentTypeError(f"{s} is not a file") return p parser = argparse.ArgumentParser() parser.add_argument("--config", type=existing_file) # validated Path # Mutually exclusive group — at most one may be given group = parser.add_mutually_exclusive_group() group.add_argument("--verbose", action="store_true") group.add_argument("--quiet", action="store_true") # Giving BOTH --verbose and --quiet is an automatic error # Parent parser — share common args across subcommands common = argparse.ArgumentParser(add_help=False) common.add_argument("--log-level", default="INFO") main = argparse.ArgumentParser() subs = main.add_subparsers(dest="cmd") subs.add_parser("build", parents=[common]) # inherits --log-level subs.add_parser("deploy", parents=[common]) # inherits --log-level # Built-in actions beyond store_true: parser.add_argument("--tag", action="append") # repeatable -> list parser.add_argument("-v", action="count", default=0) # -vvv -> 3 parser.add_argument("--version", action="version", version="1.0") ``` For very large CLIs, the standard pattern is to attach a handler function to each subparser with `subparser.set_defaults(func=handler)`, then call `args.func(args)` after parsing — a clean dispatch mechanism that keeps each command's logic in its own function. While third-party libraries like Click and Typer offer more concise decorator-based APIs, argparse is in the standard library, has zero dependencies, and is sufficient for the large majority of command-line tools. ✅ Expert tab complete - I can write a custom type= callable that validates input - I can use a mutually exclusive group - I can share arguments via a parent parser - I know the set_defaults(func=handler) dispatch pattern Continue to [math →](/coding/languages/python/standard-library/math/) ## SOURCES - Python Standard Library — argparse. docs.python.org/3/library/argparse.html. - Python HOWTO — Argparse Tutorial. docs.python.org/3/howto/argparse.html. - Python Standard Library — sys.argv. docs.python.org/3/library/sys.html#sys.argv. **Primary source:** docs.python.org argparse *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # math — Mathematical Functions ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/math/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The math module provides the standard mathematical functions and constants — trigonometry, logarithms, factorials, and reliable tools for floating-point work.* ## CANONICAL DEFINITION The math module provides access to the mathematical functions defined by the C standard. It operates on floats (for complex numbers, use cmath), and includes constants like pi and e, plus functions for rounding, powers, logarithms, trigonometry, and combinatorics. ## BEGINNER > The idea The math module is Python's calculator — square roots, powers, logarithms, trigonometry, and constants like pi. It works on regular numbers (floats) and is part of the standard library, so there is nothing to install. ## Constants and basic functions ```python import math # Constants print(math.pi) # 3.141592653589793 print(math.e) # 2.718281828459045 print(math.inf) # infinity print(math.nan) # not-a-number # Square root and powers print(math.sqrt(144)) # 12.0 print(math.pow(2, 10)) # 1024.0 (always returns a float) print(2 ** 10) # 1024 (** keeps int type — often better) # Absolute value and sign print(math.fabs(-5)) # 5.0 (always float; abs(-5) gives int 5) print(math.copysign(3, -1)) # -3.0 (magnitude of 3, sign of -1) ``` > math.pow vs the ** operator `math.pow(2, 10)` always returns a float (1024.0). The `**` operator preserves int types (`2 ** 10` gives int 1024). For integer powers, prefer `**`; it is faster and keeps the type. Use `math.pow` only when you specifically want a float result. ## Rounding and powers ```python import math # floor rounds DOWN, ceil rounds UP — both return int print(math.floor(4.7)) # 4 print(math.ceil(4.1)) # 5 print(math.floor(-4.1)) # -5 (down means toward negative infinity) print(math.ceil(-4.7)) # -4 # trunc chops toward zero (drops the fractional part) print(math.trunc(4.9)) # 4 print(math.trunc(-4.9)) # -4 (toward zero, not down) # The built-in round() uses banker's rounding (round half to even) print(round(2.5)) # 2 (not 3!) — rounds to even print(round(3.5)) # 4 — also rounds to even print(round(3.14159, 2)) # 3.14 — round to 2 decimal places # Factorials and combinatorics print(math.factorial(5)) # 120 print(math.comb(10, 3)) # 120 — combinations "10 choose 3" print(math.perm(10, 3)) # 720 — permutations print(math.gcd(48, 36)) # 12 — greatest common divisor print(math.lcm(4, 6)) # 12 — least common multiple (3.9+) ``` ✅ Beginner tab complete - I can use math.pi, math.e, math.sqrt - I know math.pow returns a float but ** keeps int type - I know floor rounds down, ceil rounds up, trunc chops toward zero - I can use factorial, comb, perm, gcd, lcm Continue to [random →](/coding/languages/python/standard-library/random/) ## INTERMEDIATE ## Trigonometry and logarithms Trig functions work in *radians*, not degrees — a frequent source of bugs. Use `math.radians()` and `math.degrees()` to convert. ```python import math # Trig functions take RADIANS, not degrees print(math.sin(math.pi / 2)) # 1.0 (sin of 90 degrees) print(math.cos(0)) # 1.0 # Convert between degrees and radians print(math.radians(180)) # 3.14159... (180 degrees in radians) print(math.degrees(math.pi)) # 180.0 # To take sin of 30 DEGREES, convert first: print(math.sin(math.radians(30))) # 0.5 # Logarithms print(math.log(math.e)) # 1.0 (natural log, base e) print(math.log(8, 2)) # 3.0 (log base 2) print(math.log10(1000)) # 3.0 (base 10) print(math.log2(1024)) # 10.0 (base 2, most accurate) # Exponential print(math.exp(1)) # 2.718... (e^1) # Hypotenuse — distance from origin (handles 2D and N-D) print(math.hypot(3, 4)) # 5.0 print(math.dist((0, 0), (3, 4))) # 5.0 (distance between points, 3.8+) ``` ## Float safety functions Floating-point arithmetic is imprecise. The math module provides tools to compare and check floats safely. ```python import math # The classic float problem print(0.1 + 0.2) # 0.30000000000000004 print(0.1 + 0.2 == 0.3) # False! # isclose — the RIGHT way to compare floats print(math.isclose(0.1 + 0.2, 0.3)) # True # Check for special values print(math.isnan(float("nan"))) # True print(math.isinf(float("inf"))) # True print(math.isfinite(42.0)) # True # fsum — accurate summation of floats (avoids accumulated error) values = [0.1] * 10 print(sum(values)) # 0.9999999999999999 print(math.fsum(values)) # 1.0 — accurate # isqrt — integer square root, exact, no float error (3.8+) print(math.isqrt(99)) # 9 (floor of the exact square root) ``` **Commonly confused:** - Trig functions use radians. math.sin(90) does NOT give the sine of 90 degrees — it gives the sine of 90 radians. Convert first: math.sin(math.radians(90)). ✅ Intermediate tab complete - I know trig functions take radians, and convert with math.radians() - I can use log, log2, log10 - I never compare floats with == — I use math.isclose - I can use fsum for accurate summation and isqrt for exact integer roots Continue to [random →](/coding/languages/python/standard-library/random/) ## EXPERT ## IEEE 754, the float spec, and when to use decimal Python floats are IEEE 754 double-precision (64-bit) binary floating-point numbers. Because they store values in binary, decimal fractions like 0.1 cannot be represented exactly — 0.1 is actually stored as the nearest representable binary value, which is slightly more than 0.1. This is not a Python flaw; it is inherent to binary floating-point and behaves identically in C, Java, and JavaScript. ```python import math from decimal import Decimal from fractions import Fraction # See the true stored value of 0.1 print(f"{0.1:.20f}") # 0.10000000000000000555 # math.isclose with explicit tolerances print(math.isclose(1000.0, 1000.1, rel_tol=1e-3)) # True (0.1% rel) print(math.isclose(0.0, 1e-10, abs_tol=1e-9)) # True (abs for near-zero) # rel_tol is relative (good for large numbers); # abs_tol is absolute (needed when comparing against zero) # For EXACT decimal arithmetic (money!), use Decimal print(Decimal("0.1") + Decimal("0.2")) # Decimal('0.3') — exact print(Decimal("0.1") + Decimal("0.2") == Decimal("0.3")) # True # For EXACT rational arithmetic, use Fraction print(Fraction(1, 3) + Fraction(1, 6)) # Fraction(1, 2) — exact # math.frexp / math.ldexp — decompose a float into mantissa & exponent m, e = math.frexp(8.0) print(m, e) # 0.5 4 -> 0.5 * 2**4 == 8.0 print(math.ldexp(m, e)) # 8.0 — reconstruct # math.nextafter (3.9+) — the next representable float print(math.nextafter(1.0, 2.0)) # 1.0000000000000002 (smallest step up) ``` The practical rule for choosing a numeric type: use `float` (and `math`) for scientific and engineering work where tiny relative errors are acceptable; use `decimal.Decimal` for money and any domain requiring exact decimal representation; use `fractions.Fraction` for exact rational arithmetic. The `math.isclose` function with appropriately chosen `rel_tol` and `abs_tol` is the correct tool for comparing floats — relative tolerance for general magnitudes, absolute tolerance when one value may be zero. ✅ Expert tab complete - I know Python floats are IEEE 754 double-precision binary - I know why 0.1 is not stored exactly - I can choose rel_tol vs abs_tol in math.isclose - I use Decimal for money and Fraction for exact ratios Continue to [random →](/coding/languages/python/standard-library/random/) ## SOURCES - Python Standard Library — math. docs.python.org/3/library/math.html. - Python Tutorial — Floating-Point Arithmetic: Issues and Limitations. docs.python.org/3/tutorial/floatingpoint.html. - Python Standard Library — decimal (exact decimal arithmetic). docs.python.org/3/library/decimal.html. - Python Standard Library — math.isclose. docs.python.org/3/library/math.html#math.isclose. **Primary source:** docs.python.org math *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # random — Generate Pseudo-Random Numbers ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/random/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The random module produces pseudo-random numbers for simulations, games, sampling, and shuffling — with a clear warning: it is not safe for security.* ## CANONICAL DEFINITION The random module implements pseudo-random number generators for various distributions. It uses the Mersenne Twister algorithm as its core generator. For security-sensitive randomness (tokens, passwords, keys), the secrets module must be used instead. ## BEGINNER > The idea The random module gives you dice rolls, coin flips, shuffled decks, and random picks. The numbers are not truly random — they come from a formula — but they are random enough for games, simulations, and sampling. Just never use them for passwords or security tokens. ## The core functions ```python import random # A float in [0.0, 1.0) print(random.random()) # e.g. 0.6394267984 # A float in a range print(random.uniform(1, 10)) # e.g. 7.31... # An integer in [a, b] — BOTH endpoints included print(random.randint(1, 6)) # a dice roll: 1 to 6 inclusive # An integer from a range [start, stop) — stop excluded, like range() print(random.randrange(0, 100, 2)) # a random even number 0..98 # Pick one item from a sequence colors = ["red", "green", "blue"] print(random.choice(colors)) # e.g. "green" ``` > randint includes both ends; randrange excludes the stop `random.randint(1, 6)` can return 1, 2, 3, 4, 5, or 6 — both ends included. `random.randrange(1, 6)` returns 1 to 5 — the stop is excluded, matching how `range()` works. Mixing these up is the most common random bug. ## Working with sequences ```python import random deck = ["A", "K", "Q", "J", "10", "9"] # Shuffle IN PLACE (modifies the list, returns None) random.shuffle(deck) print(deck) # e.g. ['Q', '9', 'A', 'K', '10', 'J'] # Pick k UNIQUE items (no repeats) — sampling without replacement hand = random.sample(deck, 3) print(hand) # e.g. ['K', 'A', '9'] # Pick k items WITH possible repeats — sampling with replacement rolls = random.choices(deck, k=5) print(rolls) # e.g. ['A', 'A', 'Q', 'J', 'A'] # Weighted choices — 'A' three times as likely as others weighted = random.choices(["A", "B", "C"], weights=[3, 1, 1], k=10) print(weighted) # mostly A's ``` > shuffle returns None `random.shuffle(deck)` shuffles the list in place and returns `None`. Writing `deck = random.shuffle(deck)` sets deck to None — a classic mistake. Just call `random.shuffle(deck)` on its own line. ✅ Beginner tab complete - I can generate random floats with random() and uniform() - I know randint(1,6) includes both 1 and 6 - I can pick with choice() and shuffle a list in place - I know random is NOT safe for passwords or tokens Continue to [argparse →](/coding/languages/python/standard-library/argparse/) ## INTERMEDIATE ## Statistical distributions Beyond uniform randomness, the module provides draws from common probability distributions — useful for simulations and modelling. ```python import random # Normal (Gaussian) distribution — mean=0, std dev=1 print(random.gauss(0, 1)) # e.g. -0.47 print(random.normalvariate(100, 15)) # IQ-like: mean 100, sd 15 # Other distributions print(random.expovariate(1.5)) # exponential print(random.triangular(0, 10, 3)) # triangular: low, high, mode print(random.betavariate(2, 5)) # beta distribution # Simulate 1000 dice rolls and check the distribution from collections import Counter rolls = [random.randint(1, 6) for _ in range(1000)] print(Counter(rolls)) # roughly even: each face ~166 times ``` ## Seeding for reproducibility Seeding the generator makes the "random" sequence repeatable — essential for tests, debugging, and reproducible simulations. ```python import random # Seed -> same sequence every run (reproducible) random.seed(42) print(random.randint(1, 100)) # always 82 with seed 42 print(random.random()) # always the same float random.seed(42) # reset to the same seed print(random.randint(1, 100)) # 82 again — identical sequence # Independent generators — don't share global state rng1 = random.Random(42) # a private generator instance rng2 = random.Random(99) print(rng1.randint(1, 10)) # independent of the global random print(rng2.randint(1, 10)) # and independent of rng1 # Use a dedicated Random() instance in libraries so you don't # disturb the application's global random state. ``` **Commonly confused:** - sample vs choices. random.sample picks unique items (no repeats, like dealing cards). random.choices picks with replacement (repeats allowed, like rolling dice) and supports weights. Cards → sample; dice → choices. ✅ Intermediate tab complete - I can draw from gauss/normalvariate distributions - I can seed for reproducible sequences - I know sample picks unique items; choices allows repeats and weights - I can create independent Random() instances Continue to [argparse →](/coding/languages/python/standard-library/argparse/) ## EXPERT ## The Mersenne Twister, state, and why it is not cryptographic The `random` module's core generator is the Mersenne Twister (MT19937), a pseudo-random number generator with an extremely long period of 219937−1. It is fast, well-distributed, and excellent for simulations — but it is deterministic and its entire future output can be reconstructed from observing 624 consecutive 32-bit outputs, which is exactly why it must never be used for security. ```python import random # The full generator state can be captured and restored state = random.getstate() a = random.random() random.setstate(state) # rewind b = random.random() print(a == b) # True — restored to the exact same point # SystemRandom — uses os.urandom (the OS entropy source) instead of MT # This IS suitable for security, though 'secrets' is the preferred API secure_rng = random.SystemRandom() print(secure_rng.randint(1, 100)) # not reproducible, not seedable # secure_rng.seed() does nothing — it draws from OS entropy # For security, prefer the secrets module: import secrets token = secrets.token_hex(16) # 32-char secure hex token pick = secrets.choice(["a", "b"]) # secure choice num = secrets.randbelow(100) # secure int in [0, 100) # random.randbytes (3.9+) — random bytes from the MT generator print(random.randbytes(4)) # NOT secure — for simulations only ``` The distinction in implementation is concrete: `random.random()` draws from the seeded, reproducible Mersenne Twister, while `random.SystemRandom` and the entire `secrets` module draw from `os.urandom()`, which pulls from the operating system's cryptographically secure entropy pool. The reproducibility that makes `random` perfect for testing — same seed, same sequence — is precisely the property that makes it unsafe for secrets, because reproducible means predictable. ✅ Expert tab complete - I know random uses the Mersenne Twister (MT19937) - I know its output is predictable from observed state - I know SystemRandom and secrets draw from os.urandom - I always use secrets for tokens, passwords, and keys Continue to [argparse →](/coding/languages/python/standard-library/argparse/) ## SOURCES - Python Standard Library — random. docs.python.org/3/library/random.html. - Python Standard Library — secrets (for security). docs.python.org/3/library/secrets.html. - Python Standard Library — random.SystemRandom. docs.python.org/3/library/random.html#random.SystemRandom. **Primary source:** docs.python.org random *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # contextlib — Utilities for with-Statements ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/contextlib/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *contextlib provides tools for creating and combining context managers — most importantly @contextmanager, which turns a simple generator into a full context manager.* ## CANONICAL DEFINITION contextlib is the standard library module providing utilities for working with the with statement. Its centrepiece is the @contextmanager decorator, which builds a context manager from a generator function, plus helpers like suppress, closing, redirect_stdout, and ExitStack. ## BEGINNER > The idea Writing a context manager (something usable with `with`) normally means a class with `__enter__` and `__exit__`. contextlib's `@contextmanager` lets you write one as a simple generator instead — setup code, then `yield`, then cleanup code. Much less boilerplate. ## @contextmanager — the easy way Decorate a generator with `@contextmanager`. Everything before `yield` is the setup (the `__enter__` part); the yielded value is bound to the `as` variable; everything after `yield` is the cleanup (the `__exit__` part). ```python from contextlib import contextmanager import time @contextmanager def timer(label): start = time.perf_counter() # setup (like __enter__) try: yield # the 'with' block runs here finally: elapsed = time.perf_counter() - start # cleanup (like __exit__) print(f"{label}: {elapsed:.4f}s") with timer("processing"): total = sum(range(10_000_000)) # processing: 0.31s # Yielding a value — it becomes the 'as' variable @contextmanager def open_upper(path): f = open(path, encoding="utf-8") try: yield (line.upper() for line in f) # yielded -> 'as' target finally: f.close() # with open_upper("data.txt") as upper_lines: # for line in upper_lines: # print(line) ``` > Always wrap yield in try/finally If the code inside the `with` block raises an exception, it propagates out through the `yield`. Wrapping `yield` in `try/finally` ensures your cleanup runs even when an exception occurs — exactly what a context manager is supposed to guarantee. ✅ Beginner tab complete - I can write a context manager with @contextmanager and a generator - I know code before yield is setup, after yield is cleanup - I wrap yield in try/finally for guaranteed cleanup - I know the generator must yield exactly once Continue to [functools →](/coding/languages/python/standard-library/functools/) ## INTERMEDIATE ## suppress and closing Two small but constantly useful helpers: `suppress` silences specific exceptions cleanly, and `closing` ensures an object's `close()` method is called. ```python from contextlib import suppress, closing import os # suppress — cleaner than try/except/pass for expected exceptions with suppress(FileNotFoundError): os.remove("temp.txt") # no error if the file is absent # Equivalent to: # try: # os.remove("temp.txt") # except FileNotFoundError: # pass # Suppress multiple exception types with suppress(FileNotFoundError, PermissionError): os.remove("locked.txt") # closing — call .close() on an object that lacks context-manager support from urllib.request import urlopen with closing(urlopen("https://example.com")) as page: data = page.read() # page.close() is called automatically on exit ``` ## Redirecting stdout and stderr ```python from contextlib import redirect_stdout, redirect_stderr import io # Capture print() output into a string buffer = io.StringIO() with redirect_stdout(buffer): print("This goes into the buffer") print("So does this") captured = buffer.getvalue() print(f"Captured: {captured!r}") # Captured: 'This goes into the buffer\nSo does this\n' # Useful for capturing output of a function you don't control def noisy_function(): print("debug spam") return 42 buffer = io.StringIO() with redirect_stdout(buffer): result = noisy_function() # its prints are captured, not shown print(f"Result was {result}, suppressed its output") # Redirect to a file with open("log.txt", "w") as f: with redirect_stdout(f): print("This line goes to log.txt") ``` **Commonly confused:** - suppress vs try/except/pass. They do the same thing, but suppress is clearer about intent and more concise. Use suppress when you genuinely expect and want to ignore a specific exception — not as a way to hide bugs. ✅ Intermediate tab complete - I can use suppress(FileNotFoundError) instead of try/except/pass - I can use closing() to ensure close() is called - I can capture print output with redirect_stdout and io.StringIO - I know suppress can take multiple exception types Continue to [functools →](/coding/languages/python/standard-library/functools/) ## EXPERT ## ExitStack and dynamic context management `ExitStack` manages an arbitrary, runtime-determined number of context managers — solving the problem that a `with` statement needs to know its context managers at write time. It guarantees that every entered context is exited in reverse order, even if an exception occurs partway through setup. ```python from contextlib import ExitStack, contextmanager, nullcontext # Open a runtime-determined number of files, all cleaned up safely def merge_files(filenames, output): with ExitStack() as stack: files = [stack.enter_context(open(f)) for f in filenames] with open(output, "w") as out: for f in files: out.write(f.read()) # every file is closed here, in reverse order, even on error # nullcontext — a do-nothing context manager for conditional use def process(data, lock=None): # Use the lock if provided, otherwise a no-op context with (lock if lock is not None else nullcontext()): return transform(data) # ExitStack as a dynamic callback registry with ExitStack() as stack: stack.callback(print, "cleanup 3") # runs last stack.callback(print, "cleanup 2") stack.callback(print, "cleanup 1") # runs first (LIFO) print("doing work") # doing work / cleanup 1 / cleanup 2 / cleanup 3 # Defer cleanup conditionally — pop_all transfers ownership def open_database(url): with ExitStack() as stack: conn = stack.enter_context(connect(url)) configure(conn) # if this raises, conn closes # success — transfer cleanup responsibility to the caller stack.pop_all() # conn will NOT be closed here return conn ``` contextlib also provides class-based building blocks: `ContextDecorator` lets a context manager double as a decorator, and `AbstractContextManager`/`AbstractAsyncContextManager` are the ABCs behind the protocol. For async code, `@asynccontextmanager` and `AsyncExitStack` mirror their synchronous counterparts for use with `async with`. The `ExitStack.pop_all()` pattern shown above is the idiomatic way to build an object that acquires several resources and hands cleanup responsibility to its caller — acquiring everything safely, and only disarming the automatic cleanup once construction fully succeeds. ✅ Expert tab complete - I can use ExitStack to manage a dynamic number of context managers - I can use nullcontext for conditional context management - I understand the pop_all() pattern for transferring cleanup ownership - I know @asynccontextmanager and AsyncExitStack exist for async Continue to [functools →](/coding/languages/python/standard-library/functools/) ## SOURCES - Python Standard Library — contextlib. docs.python.org/3/library/contextlib.html. - Python Standard Library — contextlib.contextmanager. docs.python.org/3/library/contextlib.html#contextlib.contextmanager. - Python Standard Library — contextlib.ExitStack. docs.python.org/3/library/contextlib.html#contextlib.ExitStack. - PEP 343 — The "with" Statement. peps.python.org/pep-0343/. **Primary source:** docs.python.org contextlib *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # secrets — Cryptographically Secure Random ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/secrets/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The secrets module generates cryptographically strong random values for passwords, security tokens, and anything where predictability would be a vulnerability.* ## CANONICAL DEFINITION The secrets module, added in Python 3.6 (PEP 506), generates cryptographically secure random numbers suitable for managing secrets such as account authentication, tokens, and passwords. It draws from the operating system’s most secure source of randomness, unlike the random module. ## BEGINNER > The idea When you need randomness that an attacker must never be able to guess — a password reset token, an API key, a session ID — you use `secrets`, not `random`. The `random` module is predictable; `secrets` is not. ## Why secrets exists The `random` module is fast and reproducible, which is perfect for simulations and games — and a disaster for security, because its output can be predicted. `secrets` draws from the operating system's cryptographically secure randomness source, making its output genuinely unpredictable. ```python import secrets # Secure random integer in [0, n) print(secrets.randbelow(100)) # e.g. 73 — unpredictable # Secure random selection from a sequence options = ["red", "green", "blue"] print(secrets.choice(options)) # cryptographically secure choice # Secure random bits print(secrets.randbits(16)) # a secure 16-bit integer # Compare with random — NEVER use this for security: import random random.seed(42) print(random.randbelow if False else "") # random has no randbelow; # random.randint(0, 99) is predictable once the seed/state is known. # secrets has NO seed() — you cannot make it reproducible, by design. ``` > secrets has no seed() You cannot seed `secrets` — there is deliberately no way to make its output reproducible. Reproducibility is exactly what you do not want for a security token. If you need reproducible randomness for testing, that code is not security-sensitive and should use `random` instead. ## Generating tokens ```python import secrets # URL-safe token — for password resets, email confirmations, etc. print(secrets.token_urlsafe(32)) # e.g. 'Drmhze6EPcv0fN_81Bj-nA...' # 32 bytes of randomness, encoded URL-safe (~43 characters) # Hex token — for API keys, session IDs print(secrets.token_hex(16)) # e.g. 'f9bf78b9a18ce6d46a0cd2b0...' # 16 bytes -> 32 hex characters # Raw bytes — when you need binary print(secrets.token_bytes(16)) # e.g. b'\xf9\xbf...' (16 secure bytes) # How many bytes? A common guideline: 32 bytes (256 bits) is ample # for tokens that must resist brute force indefinitely. api_key = secrets.token_urlsafe(32) reset_token = secrets.token_urlsafe(32) ``` ✅ Beginner tab complete - I know secrets is for security; random is for simulations/games - I can generate a token with token_urlsafe and token_hex - I know secrets has no seed() — it cannot be made reproducible - I know to use secrets for passwords, tokens, keys, session IDs Continue to [hashlib →](/coding/languages/python/standard-library/hashlib/) ## INTERMEDIATE ## Generating passwords and PINs You can compose `secrets` primitives to build secure passwords, passphrases, and one-time codes. The key is to always use `secrets.choice`, never `random.choice`. ```python import secrets import string # A secure random password from letters, digits, and symbols alphabet = string.ascii_letters + string.digits + string.punctuation password = "".join(secrets.choice(alphabet) for _ in range(16)) print(password) # e.g. 'k$2Lm9#pQr!7Zx@N' # A password meeting complexity rules — regenerate until satisfied def strong_password(length=12): alphabet = string.ascii_letters + string.digits + string.punctuation while True: pw = "".join(secrets.choice(alphabet) for _ in range(length)) if (any(c.islower() for c in pw) and any(c.isupper() for c in pw) and any(c.isdigit() for c in pw) and any(c in string.punctuation for c in pw)): return pw print(strong_password()) # A memorable passphrase from a word list (XKCD-style) words = ["correct", "horse", "battery", "staple", "mountain", "river"] passphrase = "-".join(secrets.choice(words) for _ in range(4)) print(passphrase) # e.g. 'river-staple-horse-mountain' # A numeric one-time code (OTP) otp = "".join(secrets.choice(string.digits) for _ in range(6)) print(otp) # e.g. '042917' (leading zeros preserved — it's a string) ``` ## Constant-time comparison When comparing secrets (like checking a submitted token against the stored one), a normal `==` comparison can leak information through timing. `secrets.compare_digest` compares in constant time, defeating timing attacks. ```python import secrets stored_token = "a1b2c3d4e5f6" def check_token(submitted): # WRONG: submitted == stored_token # == returns as soon as it finds a mismatched character, so the # time taken leaks how many leading characters were correct. # RIGHT: constant-time comparison return secrets.compare_digest(submitted, stored_token) print(check_token("a1b2c3d4e5f6")) # True print(check_token("wrong")) # False # compare_digest always takes the same time regardless of where # (or whether) the strings differ — no timing information leaks. ``` **Commonly confused:** - secrets vs random — the rule is absolute. Anything a user should not be able to predict or forge — tokens, passwords, keys, session IDs, OTPs — uses secrets. Anything reproducible or for simulation/games uses random. When in doubt about whether something is security-sensitive, use secrets. ✅ Intermediate tab complete - I can build a secure password with secrets.choice over an alphabet - I can generate a numeric OTP as a string (preserving leading zeros) - I use secrets.compare_digest to compare tokens in constant time - I know == leaks timing information when comparing secrets Continue to [hashlib →](/coding/languages/python/standard-library/hashlib/) ## EXPERT ## os.urandom, entropy sources, and token sizing Under the hood, `secrets` uses `random.SystemRandom`, which draws from `os.urandom()` — the operating system's cryptographically secure pseudo-random number generator (CSPRNG). On Linux this is the `getrandom()` syscall backed by the kernel entropy pool; on Windows it is `BCryptGenRandom`. These sources are designed so that observing past output gives no usable information about future output, which is precisely the property the Mersenne Twister in `random` lacks. ```python import secrets, os # secrets is a thin, safe layer over os.urandom raw = os.urandom(16) # 16 cryptographically secure bytes print(secrets.token_hex(16) == secrets.token_hex(16)) # False (always) # Token sizing — the default nbytes # secrets.token_bytes() with no argument uses a reasonable default (32) print(len(secrets.token_bytes())) # 32 by default # Sizing guidance from PEP 506 / docs: # - 16 bytes (128 bits): minimum for most purposes # - 32 bytes (256 bits): recommended for long-lived secrets # token_urlsafe(n) produces ceil(n * 4 / 3) characters (base64url) print(len(secrets.token_urlsafe(32))) # 43 characters # secrets.choice uses SystemRandom under the hood sr = secrets.SystemRandom() # same generator secrets uses print(sr.randint(1, 100)) # secure, not seedable # Why constant-time matters — compare_digest works on str (ASCII) or bytes import hmac # secrets.compare_digest is the same primitive as hmac.compare_digest print(secrets.compare_digest(b"abc", b"abc")) # True # Both compare every byte regardless of early mismatches. ``` A subtle but important detail: `secrets.compare_digest` only guarantees constant time with respect to *matching* content — it still reveals the length of the strings through timing, so it should compare values of fixed or already-known length (such as fixed-size tokens or hash digests). For password verification specifically, you should not store or compare raw passwords at all — you store a salted hash from a slow password-hashing function (covered in the hashlib page) and compare digests. The `secrets` module handles secure generation; password storage is a separate concern requiring deliberately slow hashing. ✅ Expert tab complete - I know secrets draws from os.urandom (the OS CSPRNG) - I know 32 bytes/256 bits is recommended for long-lived secrets - I know compare_digest still reveals length, so compare fixed-size values - I know password storage needs slow hashing, not just secure generation Continue to [hashlib →](/coding/languages/python/standard-library/hashlib/) ## SOURCES - Python Standard Library — secrets. docs.python.org/3/library/secrets.html. - PEP 506 — Adding A Secrets Module to the Standard Library. peps.python.org/pep-0506/. - Python Standard Library — os.urandom. docs.python.org/3/library/os.html#os.urandom. - Python Standard Library — hmac.compare_digest. docs.python.org/3/library/hmac.html#hmac.compare_digest. **Primary source:** docs.python.org secrets *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # hashlib — Secure Hashes and Message Digests ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/hashlib/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *hashlib computes cryptographic hashes like SHA-256 — fixed-size fingerprints of data used for integrity checks, deduplication, and (with the right functions) password storage.* ## CANONICAL DEFINITION hashlib provides a common interface to many secure hash and message digest algorithms, including the SHA-2 family (sha256, sha512), SHA-3, and BLAKE2. It also provides scrypt and pbkdf2_hmac for password hashing. Hashes are one-way: you cannot recover the input from the digest. ## BEGINNER > The idea A hash function takes any data and produces a fixed-size "fingerprint" — the same input always gives the same fingerprint, but you cannot work backwards from the fingerprint to the input. Hashes verify that data has not changed, and (with special slow functions) store passwords safely. ## Computing a hash The core pattern: create a hash object, feed it bytes, then read the digest. Note that hashlib works on `bytes`, not `str` — you must encode text first. ```python import hashlib # SHA-256 — the most common general-purpose secure hash text = "The Codex" digest = hashlib.sha256(text.encode("utf-8")).hexdigest() print(digest) # 64 hex characters — always the same for this input # e.g. '6c3f...e91a' # The same input ALWAYS gives the same hash print(hashlib.sha256(b"hello").hexdigest() == hashlib.sha256(b"hello").hexdigest()) # True # A tiny change gives a completely different hash (avalanche effect) print(hashlib.sha256(b"hello").hexdigest()[:8]) # '2cf24dba' print(hashlib.sha256(b"Hello").hexdigest()[:8]) # ' 185f8db3' — totally different # You must pass bytes, not str # hashlib.sha256("hello") # TypeError: Unicode-objects must be encoded hashlib.sha256("hello".encode()) # correct — encode to bytes first ``` > hashlib needs bytes, not strings `hashlib.sha256("text")` raises TypeError. Encode the string first: `hashlib.sha256("text".encode("utf-8"))`. Hashes operate on raw bytes, so text must be converted to a byte representation, and the encoding affects the result. ## Hashing files for integrity ```python import hashlib # Hash a file in chunks — never load a large file fully into memory def hash_file(path, algorithm="sha256"): h = hashlib.new(algorithm) with open(path, "rb") as f: # binary mode! for chunk in iter(lambda: f.read(8192), b""): h.update(chunk) # feed data incrementally return h.hexdigest() # print(hash_file("video.mp4")) # verify a download matches its published hash # Python 3.11+ has a built-in helper for exactly this: # with open("file.bin", "rb") as f: # digest = hashlib.file_digest(f, "sha256").hexdigest() # update() can be called repeatedly — order matters h = hashlib.sha256() h.update(b"Hello, ") h.update(b"world!") print(h.hexdigest() == hashlib.sha256(b"Hello, world!").hexdigest()) # True ``` ✅ Beginner tab complete - I can compute a SHA-256 hash with hexdigest() - I know hashlib needs bytes, so I encode strings first - I can hash a file in chunks for integrity verification - I know a hash is one-way — you cannot reverse it Continue to [statistics →](/coding/languages/python/standard-library/statistics/) ## INTERMEDIATE ## Password hashing — the right way Storing passwords requires a *deliberately slow* hash with a unique salt per password. Never use plain SHA-256 for passwords — it is too fast, making brute-force attacks cheap. Use `pbkdf2_hmac` or `scrypt`. ```python import hashlib, secrets def hash_password(password): # A unique random salt per password defeats rainbow tables salt = secrets.token_bytes(16) # pbkdf2 with many iterations is deliberately slow key = hashlib.pbkdf2_hmac( "sha256", password.encode("utf-8"), salt, 600_000, # iteration count — higher = slower = safer ) return salt + key # store salt alongside the hash def verify_password(stored, password): salt = stored[:16] # the salt we prepended key = stored[16:] new_key = hashlib.pbkdf2_hmac("sha256", password.encode(), salt, 600_000) return secrets.compare_digest(key, new_key) # constant-time compare stored = hash_password("correct horse battery staple") print(verify_password(stored, "correct horse battery staple")) # True print(verify_password(stored, "wrong password")) # False # scrypt — memory-hard, even stronger against custom hardware attacks key = hashlib.scrypt( b"my password", salt=secrets.token_bytes(16), n=2**14, r=8, p=1, dklen=32) ``` > Never use plain sha256 for passwords SHA-256 is designed to be fast — billions of guesses per second on a GPU. For passwords you want the opposite: a slow, salted function like `pbkdf2_hmac` or `scrypt`. For production, a dedicated library implementing Argon2 or bcrypt is often preferable, but pbkdf2_hmac is solid and built in. ## Choosing an algorithm | Algorithm | Use for | Notes | | --- | --- | --- | | `sha256` / `sha512` | General integrity, fingerprints | Fast, secure, the default choice | | `blake2b` / `blake2s` | High-speed hashing | Faster than SHA-2, equally secure | | `sha3_256` | Integrity (different design) | SHA-3 family, distinct from SHA-2 | | `pbkdf2_hmac` / `scrypt` | Passwords | Deliberately slow + salted | | `md5` / `sha1` | Legacy / non-security checksums only | Broken — never use for security | **Commonly confused:** - Hashing is not encryption. Encryption is reversible with a key; hashing is one-way. You cannot "decrypt" a hash. This is exactly why hashes are used for passwords — even if the database leaks, the original passwords are not directly recoverable. ✅ Intermediate tab complete - I never use plain sha256 for passwords - I use pbkdf2_hmac or scrypt with a unique random salt per password - I store the salt alongside the hash and compare with compare_digest - I know md5 and sha1 are broken and only for non-security checksums Continue to [statistics →](/coding/languages/python/standard-library/statistics/) ## EXPERT ## Salts, peppers, HMAC, and BLAKE2 keyed hashing A salt is a unique random value stored alongside each password hash; it ensures two users with the same password get different hashes and defeats precomputed rainbow tables. A pepper is a secret value (not stored in the database, kept in application config) mixed into the hash, adding a layer that survives a database-only breach. For message authentication — verifying both integrity and authenticity — HMAC combines a hash with a secret key. ```python import hashlib, hmac, secrets # HMAC — authenticated hash, proves the data came from someone with the key key = secrets.token_bytes(32) message = b"transfer $100 to account 12345" signature = hmac.new(key, message, hashlib.sha256).hexdigest() # The recipient recomputes HMAC with the shared key and compares print(hmac.compare_digest( signature, hmac.new(key, message, hashlib.sha256).hexdigest())) # True # BLAKE2 supports keyed hashing natively (a built-in MAC) mac = hashlib.blake2b(message, key=key).hexdigest() # Also supports 'person' and 'salt' parameters for domain separation h = hashlib.blake2b(b"data", salt=b"16-byte-salt----", person=b"app-v1__________") # Available algorithms on this platform print(hashlib.algorithms_guaranteed) # always available everywhere print(hashlib.algorithms_available) # platform-dependent extras # usedforsecurity flag (3.9+) — mark a hash as non-security # Lets MD5 work in FIPS-restricted builds for checksums only h = hashlib.md5(b"data", usedforsecurity=False) # checksum, not security # Length matters: sha256 -> 32 bytes, sha512 -> 64 bytes print(hashlib.sha256().digest_size) # 32 print(hashlib.sha512().digest_size) # 64 ``` The choice between password-hashing functions involves a deliberate cost trade-off. `pbkdf2_hmac` is CPU-hard: its iteration count sets how many hash rounds an attacker must perform per guess. `scrypt` is additionally memory-hard, requiring large amounts of RAM per guess, which defeats the cheap parallelism of GPUs and custom ASICs. The modern recommendation for new systems is Argon2 (not in the standard library — available via the `argon2-cffi` package), which is memory-hard and won the Password Hashing Competition; but `scrypt` and a high-iteration `pbkdf2_hmac` remain strong, standards-based choices that ship with Python. ✅ Expert tab complete - I know a salt is per-password and stored; a pepper is secret and global - I can use hmac for authenticated message hashing - I know pbkdf2 is CPU-hard while scrypt is also memory-hard - I know Argon2 (third-party) is the modern recommendation Continue to [statistics →](/coding/languages/python/standard-library/statistics/) ## SOURCES - Python Standard Library — hashlib. docs.python.org/3/library/hashlib.html. - Python Standard Library — hashlib.pbkdf2_hmac. docs.python.org/3/library/hashlib.html#hashlib.pbkdf2_hmac. - Python Standard Library — hmac. docs.python.org/3/library/hmac.html. - Python Standard Library — hashlib.scrypt. docs.python.org/3/library/hashlib.html#hashlib.scrypt. **Primary source:** docs.python.org hashlib *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # statistics — Mathematical Statistics Functions ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/statistics/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The statistics module provides functions for calculating averages, spread, and distributions on real-valued data — the everyday statistics without needing NumPy.* ## CANONICAL DEFINITION The statistics module, added in Python 3.4, provides functions for calculating mathematical statistics of numeric data: measures of central tendency (mean, median, mode), measures of spread (variance, standard deviation), and tools for working with the normal distribution. ## BEGINNER > The idea The statistics module answers everyday questions about a list of numbers: what is the average? the middle value? the most common one? how spread out are they? It is built into Python — no need to install NumPy for basic statistics. ## Averages: mean, median, mode ```python import statistics scores = [85, 90, 78, 92, 88, 90, 76] # Mean — the arithmetic average print(statistics.mean(scores)) # 85.57... # Median — the middle value when sorted (robust against outliers) print(statistics.median(scores)) # 88 # Mode — the most common value print(statistics.mode(scores)) # 90 (appears twice) # Why median matters: outliers distort the mean but not the median incomes = [30000, 35000, 32000, 38000, 5000000] # one billionaire print(statistics.mean(incomes)) # 1027000 — misleading! print(statistics.median(incomes)) # 35000 — represents the typical value # fmean (3.8+) — faster, always returns float print(statistics.fmean(scores)) # 85.571... (float, faster than mean) ``` > Mean vs median: outliers The **mean** is pulled toward extreme values; one billionaire in a room makes the average net worth meaningless. The **median** (middle value) resists outliers and often better represents "typical." Choosing the right average is a judgement, not just a calculation. ## Variance and standard deviation ```python import statistics data = [2, 4, 4, 4, 5, 5, 7, 9] # Standard deviation — how spread out the data is, in original units print(statistics.stdev(data)) # 2.13... (sample std dev) print(statistics.pstdev(data)) # 2.0 (population std dev) # Variance — the square of standard deviation print(statistics.variance(data)) # 4.57... (sample variance) print(statistics.pvariance(data)) # 4.0 (population variance) # Sample vs population: # - Use stdev/variance (sample) when data is a SAMPLE of a larger group # - Use pstdev/pvariance (population) when data is the ENTIRE group # The sample versions divide by (n-1), correcting for sampling bias. ``` ✅ Beginner tab complete - I can compute mean, median, and mode - I know median resists outliers while mean does not - I can compute standard deviation and variance - I know sample (stdev) vs population (pstdev) versions Continue to [secrets →](/coding/languages/python/standard-library/secrets/) ## INTERMEDIATE ## The normal distribution Python 3.8+ includes a `NormalDist` class for working with normal (Gaussian) distributions — computing probabilities, percentiles, and modelling data. ```python from statistics import NormalDist # Model IQ scores: mean 100, standard deviation 15 iq = NormalDist(mu=100, sigma=15) # What fraction of people score below 130? (cumulative distribution) print(iq.cdf(130)) # 0.977... — about 97.7% # What IQ is the 90th percentile? (inverse CDF) print(iq.inv_cdf(0.90)) # 119.2... # Probability density at a point print(iq.pdf(100)) # 0.0266... (peak, at the mean) # Build a NormalDist directly from data data = [88, 92, 95, 100, 105, 108, 112] dist = NormalDist.from_samples(data) print(f"mean={dist.mean:.1f}, stdev={dist.stdev:.1f}") # Arithmetic on distributions (sum of independent normals) combined = NormalDist(100, 15) + NormalDist(50, 8) print(combined.mean, round(combined.stdev, 2)) # 150 17.0 # Overlap between two distributions (how similar they are) a = NormalDist(100, 15) b = NormalDist(110, 15) print(a.overlap(b)) # 0.737... — 73.7% overlap ``` ## Precision and data types ```python import statistics from fractions import Fraction from decimal import Decimal # statistics preserves the input type where it can print(statistics.mean([1, 2, 3])) # 2 (int average happens to be int) print(statistics.mean([1, 2])) # 1.5 (float when needed) print(type(statistics.mean([Decimal("1.1"), Decimal("2.2")]))) # Decimal print(statistics.mean([Fraction(1, 3), Fraction(2, 3)])) # Fraction(1, 2) # This matters for money — Decimal preserves exact precision prices = [Decimal("19.99"), Decimal("5.49"), Decimal("12.00")] print(statistics.mean(prices)) # Decimal('12.49333...') # quantiles — divide data into equal-probability intervals data = list(range(1, 101)) # 1 to 100 print(statistics.quantiles(data, n=4)) # quartiles: [25.75, 50.5, 75.25] print(statistics.quantiles(data, n=10)) # deciles # median variants for edge cases print(statistics.median_low([1, 2, 3, 4])) # 2 (lower of two middles) print(statistics.median_high([1, 2, 3, 4])) # 3 (higher of two middles) ``` **Commonly confused:** - Sample vs population statistics. Use stdev/variance when your data is a sample drawn from a larger population (the common case). Use pstdev/pvariance only when your data is the complete population. The sample versions divide by n−1 to correct for bias. ✅ Intermediate tab complete - I can use NormalDist for cdf, inv_cdf, and from_samples - I know statistics preserves Decimal and Fraction input types - I can compute quantiles (quartiles, deciles) - I know median_low and median_high for even-length data Continue to [secrets →](/coding/languages/python/standard-library/secrets/) ## EXPERT ## Numerical accuracy, mean implementations, and when to leave the stdlib A distinguishing feature of the `statistics` module is its emphasis on correctness over speed. `statistics.mean` avoids the floating-point error that naive summation accumulates, and it preserves the input's numeric type — returning `Fraction` for `Fraction` input and `Decimal` for `Decimal` input, which matters for exact financial or rational computation. The faster `fmean` always converts to float, trading that type preservation and some accuracy for speed. ```python import statistics # statistics.mean is accurate even for values that defeat naive summation data = [1e20, 1, -1e20] * 1000 print(sum(data) / len(data)) # may show float error print(statistics.mean(data)) # accurate # correlation, covariance, and linear regression (3.10+) xs = [1, 2, 3, 4, 5] ys = [2, 4, 5, 4, 6] print(statistics.correlation(xs, ys)) # Pearson correlation coefficient print(statistics.covariance(xs, ys)) # sample covariance # linear_regression returns slope and intercept (3.10+) slope, intercept = statistics.linear_regression(xs, ys) print(f"y = {slope:.2f}x + {intercept:.2f}") # Predict a new value predict = lambda x: slope * x + intercept print(predict(6)) # geometric_mean and harmonic_mean for the right kinds of data print(statistics.geometric_mean([1.1, 1.2, 1.5])) # for growth rates print(statistics.harmonic_mean([40, 60])) # for rates/speeds -> 48.0 # Multimodal data — mode returns first; multimode returns all print(statistics.multimode([1, 1, 2, 2, 3])) # [1, 2] ``` The module deliberately covers descriptive statistics and basic inferential tools (correlation, covariance, simple linear regression as of 3.10) but stops short of the full statistical toolkit. For hypothesis testing, multiple regression, ANOVA, time-series analysis, or anything involving large datasets and performance, the ecosystem standard is the scientific stack: NumPy for arrays, SciPy for statistical tests and distributions, pandas for tabular data, and statsmodels for advanced modelling. The standard library `statistics` module fills the gap where you need a correct mean, median, standard deviation, or a normal-distribution calculation without pulling in those dependencies. ✅ Expert tab complete - I know statistics.mean is accurate where naive summation fails - I can compute correlation, covariance, and linear_regression (3.10+) - I know geometric_mean and harmonic_mean for the right data - I know to use NumPy/SciPy/pandas for large data or advanced stats Continue to [secrets →](/coding/languages/python/standard-library/secrets/) ## SOURCES - Python Standard Library — statistics. docs.python.org/3/library/statistics.html. - Python Standard Library — statistics.NormalDist. docs.python.org/3/library/statistics.html#statistics.NormalDist. - Python Standard Library — statistics.linear_regression. docs.python.org/3/library/statistics.html#statistics.linear_regression. **Primary source:** docs.python.org statistics *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # pickle — Python Object Serialization ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/pickle/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *pickle converts almost any Python object into a byte stream and back — for caching, saving state, and inter-process communication. With one serious security caveat.* ## CANONICAL DEFINITION The pickle module implements binary protocols for serializing and de-serializing a Python object structure. "Pickling" converts an object hierarchy into a byte stream; "unpickling" reconstructs the object from the byte stream. Unlike JSON, pickle handles almost any Python type but is Python-specific and unsafe with untrusted data. ## BEGINNER > The idea pickle saves a Python object — a list, a dict, a custom class instance, almost anything — to a file or byte stream, so you can load it back exactly as it was later. Think of it as freezing a Python object to disk and thawing it on demand. ## Pickling and unpickling ```python import pickle data = { "name": "Priya", "scores": [85, 92, 78], "active": True, "nested": {"a": (1, 2, 3)}, } # dumps -> bytes (the "s" means string/bytes, like json) pickled = pickle.dumps(data) print(type(pickled)) # # loads -> reconstruct the object restored = pickle.loads(pickled) print(restored == data) # True — identical structure print(restored["nested"]["a"]) # (1, 2, 3) — even the tuple is preserved # pickle handles types JSON cannot: import datetime obj = { "when": datetime.datetime.now(), # datetime — JSON can't do this "coords": (1, 2), # tuple stays a tuple (JSON makes lists) "unique": {1, 2, 3}, # set — JSON can't do this } print(pickle.loads(pickle.dumps(obj))) # all types preserved exactly ``` > pickle preserves Python types exactly Unlike JSON, pickle keeps tuples as tuples (not lists), sets as sets, datetimes as datetimes, and can even serialize custom class instances. The trade-off: pickle files are readable only by Python, while JSON is universal across languages. ## Saving to and loading from files ```python import pickle cache = {"results": [1, 2, 3], "computed_at": "2026-06-17"} # dump -> write to a file (binary mode is required) with open("cache.pkl", "wb") as f: # 'wb' — write binary pickle.dump(cache, f) # load -> read from a file with open("cache.pkl", "rb") as f: # 'rb' — read binary loaded = pickle.load(f) print(loaded) # {'results': [1, 2, 3], 'computed_at': '2026-06-17'} # Pickle a custom class instance class Model: def __init__(self, weights): self.weights = weights m = Model([0.1, 0.5, 0.9]) with open("model.pkl", "wb") as f: pickle.dump(m, f) # Later — the class definition must be importable to unpickle with open("model.pkl", "rb") as f: restored = pickle.load(f) print(restored.weights) # [0.1, 0.5, 0.9] ``` > Always use binary mode Pickle data is binary — open files with `"wb"` and `"rb"`, never `"w"`/`"r"`. The conventional extension is `.pkl` or `.pickle`, though the extension is not required. ✅ Beginner tab complete - I can pickle.dumps an object to bytes and loads it back - I know pickle preserves tuples, sets, datetimes, custom classes - I use binary mode (wb/rb) for pickle files - I know pickle files are readable only by Python Continue to [json →](/coding/languages/python/standard-library/json/) ## INTERMEDIATE ## The security risk — read this carefully Unpickling data executes code embedded in the pickle. A malicious pickle can run arbitrary commands the moment you load it. **Never unpickle data from an untrusted source.** ```python import pickle # WHY pickle is dangerous: unpickling can execute arbitrary code. # A crafted pickle can run any command when loaded — no warning. # This is not a bug; it is inherent to how pickle reconstructs objects # (it can call __reduce__ which can invoke any callable). # THE RULE: # - NEVER pickle.load() data received over a network, from a user # upload, from a URL, or any source you do not fully control. # - Pickle is safe ONLY for data your own program created and stored # somewhere only your program can write. # If you must exchange data with the outside world, use JSON: import json safe = json.loads(untrusted_string) # JSON cannot execute code # If you need pickle's power but worry about tampering, sign the data: import hmac, hashlib, secrets key = secrets.token_bytes(32) def safe_dump(obj): payload = pickle.dumps(obj) sig = hmac.new(key, payload, hashlib.sha256).digest() return sig + payload def safe_load(blob): sig, payload = blob[:32], blob[32:] expected = hmac.new(key, payload, hashlib.sha256).digest() if not hmac.compare_digest(sig, expected): raise ValueError("Tampered or forged pickle — refusing to load") return pickle.loads(payload) # only after verifying the signature ``` ## pickle vs JSON — choosing | | pickle | JSON | | --- | --- | --- | | Readable by | Python only | Any language | | Types | Almost any Python object | dict, list, str, num, bool, null | | Human-readable | No (binary) | Yes (text) | | Safe with untrusted data | **No — never** | Yes | | Tuples / sets / datetime | Preserved | Lost / unsupported | | Use for | Internal caching, app state | APIs, config, data exchange | **Commonly confused:** - pickle is not for data exchange. Because it is Python-only and unsafe with untrusted input, never use pickle for APIs, file formats others will read, or anything crossing a trust boundary. Use JSON for those. pickle is for your own program's internal storage. ✅ Intermediate tab complete - I NEVER unpickle data from an untrusted source - I know unpickling can execute arbitrary code - I use JSON for data crossing a trust boundary or other languages - I can sign a pickle with hmac if I must verify integrity Continue to [json →](/coding/languages/python/standard-library/json/) ## EXPERT ## Protocols, __reduce__, and alternatives pickle supports several wire protocols, numbered 0 through 5. Higher protocols are more efficient and support more types; protocol 5 (PEP 574, Python 3.8+) adds out-of-band data for zero-copy serialization of large buffers. `pickle.DEFAULT_PROTOCOL` is the safe default, while `pickle.HIGHEST_PROTOCOL` gives the most efficient format your Python version supports. ```python import pickle print(pickle.HIGHEST_PROTOCOL) # 5 on modern Python print(pickle.DEFAULT_PROTOCOL) # 5 (3.8+) # Choose a protocol explicitly (e.g. for cross-version compatibility) data = {"x": 1} blob = pickle.dumps(data, protocol=4) # protocol 4 for older readers # __reduce__ / __getstate__ / __setstate__ control pickling of a class class Connection: def __init__(self, host): self.host = host self.socket = "live-socket-object" # can't be pickled def __getstate__(self): # Return what to pickle — exclude the unpicklable socket state = self.__dict__.copy() del state["socket"] return state def __setstate__(self, state): # Restore, then re-establish the excluded resource self.__dict__.update(state) self.socket = None # reconnect on demand c = pickle.loads(pickle.dumps(Connection("db.example.com"))) print(c.host, c.socket) # db.example.com None # What cannot be pickled: open files, sockets, lambdas, generators, # locks, and database connections — anything tied to runtime state. # pickle.dumps(lambda x: x) # PicklingError # Alternatives in the ecosystem: # - json: cross-language, safe, text # - marshal: internal CPython use only, NOT stable across versions # - dill: third-party, extends pickle (lambdas, more types) # - joblib: efficient for large NumPy arrays (ML model persistence) ``` For controlling serialization of complex objects, the pickling protocol offers a hierarchy of hooks: `__getstate__` and `__setstate__` customize what state is saved and how it is restored (the common case, ideal for excluding live resources like sockets or file handles), while `__reduce__` gives complete low-level control over reconstruction. The recommended practice for any object holding runtime-bound resources is to implement `__getstate__` to exclude them and `__setstate__` to re-establish or defer them. For the specific and common case of persisting machine-learning models with large NumPy arrays, `joblib` is more efficient than raw pickle; for cross-language or untrusted contexts, JSON remains the correct and safe choice. ✅ Expert tab complete - I know pickle has protocols 0-5; higher is more efficient - I can use __getstate__/__setstate__ to exclude unpicklable resources - I know open files, sockets, lambdas, and generators cannot be pickled - I know joblib for large arrays and dill for extended types Continue to [json →](/coding/languages/python/standard-library/json/) ## SOURCES - Python Standard Library — pickle. docs.python.org/3/library/pickle.html. - Python Standard Library — pickle protocols and Pickler. docs.python.org/3/library/pickle.html. - PEP 574 — Pickle protocol 5 with out-of-band data. peps.python.org/pep-0574/. - Python Standard Library — json (safe alternative). docs.python.org/3/library/json.html. **Primary source:** docs.python.org pickle *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # sqlite3 — Built-in SQL Database ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/sqlite3/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *sqlite3 gives Python a complete SQL database engine with zero setup — a real relational database living in a single file, perfect for apps, prototypes, and local storage.* ## CANONICAL DEFINITION The sqlite3 module provides a DB-API 2.0 interface to SQLite, a self-contained, serverless SQL database engine. The entire database is stored in a single file (or in memory). No separate server process or installation is required — SQLite ships with Python. ## BEGINNER > The idea sqlite3 is a complete SQL database built into Python — no server to install, no configuration. The whole database lives in one file on disk. It is the same engine that runs inside phones, browsers, and countless apps. For local storage and prototypes, it is ideal. ## Connecting and running a query The workflow: connect to a database file (created if it does not exist), get a cursor, execute SQL, fetch results, and commit changes. ```python import sqlite3 # Connect — creates the file if it doesn't exist conn = sqlite3.connect("app.db") cursor = conn.cursor() # Create a table cursor.execute(""" CREATE TABLE IF NOT EXISTS users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT UNIQUE, age INTEGER ) """) # Insert a row cursor.execute( "INSERT INTO users (name, email, age) VALUES (?, ?, ?)", ("Priya", "priya@example.com", 30) ) conn.commit() # save changes to disk — REQUIRED conn.close() # always close when done ``` > You must commit() to save Changes (INSERT, UPDATE, DELETE) live in a transaction until you call `conn.commit()`. If you close the connection or the program exits without committing, the changes are lost. This is a feature — it lets you roll back on error — but it catches beginners constantly. ## Reading data back ```python import sqlite3 conn = sqlite3.connect("app.db") cursor = conn.cursor() # Fetch ALL matching rows as a list of tuples cursor.execute("SELECT name, age FROM users") rows = cursor.fetchall() for name, age in rows: print(f"{name} is {age}") # Fetch ONE row (or None if no rows) cursor.execute("SELECT * FROM users WHERE name = ?", ("Priya",)) row = cursor.fetchone() print(row) # (1, 'Priya', 'priya@example.com', 30) # Iterate the cursor directly — memory-efficient for large results cursor.execute("SELECT name FROM users") for row in cursor: # fetches one row at a time print(row[0]) # Using the connection as a context manager auto-commits with sqlite3.connect("app.db") as conn: conn.execute("INSERT INTO users (name) VALUES (?)", ("Arjun",)) # commits automatically if no exception; rolls back on error conn.close() ``` ✅ Beginner tab complete - I can connect, get a cursor, execute SQL, and close - I know I must commit() to save changes to disk - I can fetchall() and fetchone() to read results - I know connect() creates the file if it does not exist Continue to [time →](/coding/languages/python/standard-library/time/) ## INTERMEDIATE ## Safe parameters — never use string formatting The single most important rule in sqlite3 (and all SQL): pass values as parameters with `?` placeholders. Never build SQL with f-strings or `+` — that opens an SQL injection vulnerability. ```python import sqlite3 conn = sqlite3.connect("app.db") cursor = conn.cursor() user_input = "Priya" # WRONG — SQL injection vulnerability. NEVER do this. # cursor.execute(f"SELECT * FROM users WHERE name = '{user_input}'") # If user_input were "'; DROP TABLE users; --" your table is gone. # RIGHT — parameterised query. The ? is a placeholder. cursor.execute("SELECT * FROM users WHERE name = ?", (user_input,)) # The driver safely escapes the value — injection is impossible. # Multiple parameters — order matches the ? positions cursor.execute( "SELECT * FROM users WHERE age > ? AND age < ?", (18, 65) ) # Named parameters — clearer for many values cursor.execute( "INSERT INTO users (name, email, age) VALUES (:name, :email, :age)", {"name": "Rohit", "email": "rohit@example.com", "age": 28} ) # executemany — insert many rows efficiently users = [("Sara", "sara@x.com", 25), ("Ken", "ken@x.com", 40)] cursor.executemany( "INSERT INTO users (name, email, age) VALUES (?, ?, ?)", users ) conn.commit() ``` > A single value still needs a tuple `cursor.execute("... WHERE name = ?", (name,))` — note the trailing comma making `(name,)` a tuple. `(name)` without the comma is just `name` in parentheses, not a tuple, and raises an error. This is the most common sqlite3 mistake. ## Row factories and transactions ```python import sqlite3 conn = sqlite3.connect("app.db") # Row factory — access columns by NAME instead of index conn.row_factory = sqlite3.Row cursor = conn.cursor() cursor.execute("SELECT * FROM users WHERE name = ?", ("Priya",)) row = cursor.fetchone() print(row["name"], row["email"]) # by column name — much clearer print(row.keys()) # ['id', 'name', 'email', 'age'] # Transactions — group operations so they all succeed or all fail try: with conn: # 'with conn' is a transaction conn.execute("UPDATE users SET age = age + 1 WHERE name = ?", ("Priya",)) conn.execute("INSERT INTO users (name) VALUES (?)", ("Maya",)) # Both committed together if no exception except sqlite3.IntegrityError as e: print(f"Transaction rolled back: {e}") # Neither change was applied — the database is unchanged # An in-memory database — fast, temporary, gone when closed mem = sqlite3.connect(":memory:") # great for tests and caching conn.close() ``` **Commonly confused:** - ? placeholders are not string formatting. Do not put quotes around ? in the SQL, and do not use f-strings. "WHERE name = ?" is correct; "WHERE name = '?'" and f"WHERE name = {x}" are both wrong (the first looks for a literal question mark, the second is an injection hole). ✅ Intermediate tab complete - I always use ? placeholders, never f-strings, for values - I remember a single value still needs a tuple: (value,) - I can use sqlite3.Row to access columns by name - I can use with conn: for automatic transaction commit/rollback Continue to [time →](/coding/languages/python/standard-library/time/) ## EXPERT ## Type adapters, isolation levels, and when to move to a server database SQLite stores values using a dynamic type system with five storage classes (NULL, INTEGER, REAL, TEXT, BLOB). Python's sqlite3 maps these to native types automatically, and you can register adapters and converters to store richer types like `datetime` or custom objects transparently. The `detect_types` parameter enables this conversion on the way back out. ```python import sqlite3, datetime # Register an adapter (Python -> SQLite) and converter (SQLite -> Python) def adapt_datetime(dt): return dt.isoformat() def convert_datetime(b): return datetime.datetime.fromisoformat(b.decode()) sqlite3.register_adapter(datetime.datetime, adapt_datetime) sqlite3.register_converter("datetime", convert_datetime) conn = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES) conn.execute("CREATE TABLE events (when_ datetime)") conn.execute("INSERT INTO events VALUES (?)", (datetime.datetime.now(),)) row = conn.execute("SELECT when_ FROM events").fetchone() print(type(row[0])) # — converted back # Isolation levels control automatic transaction behaviour # isolation_level=None gives autocommit mode (each statement commits) auto = sqlite3.connect(":memory:", isolation_level=None) # Explicit transaction control with autocommit auto.execute("BEGIN") # ... multiple statements ... auto.execute("COMMIT") # PRAGMA statements configure the database engine conn.execute("PRAGMA foreign_keys = ON") # enforce foreign keys (off by default!) conn.execute("PRAGMA journal_mode = WAL") # write-ahead logging — better concurrency # Backup API — copy a live database safely src = sqlite3.connect("app.db") dst = sqlite3.connect("backup.db") src.backup(dst) # atomic, consistent backup even of a live DB src.close(); dst.close() ``` A critical default to know: SQLite does **not** enforce foreign-key constraints unless you turn them on with `PRAGMA foreign_keys = ON` for each connection. SQLite excels for embedded use, single-writer workloads, local application storage, testing, and prototypes — its single-file simplicity is a genuine strength. It reaches its limits with high write concurrency (it permits only one writer at a time) and very large multi-user deployments, where a client-server database like PostgreSQL or MySQL becomes appropriate. Because Python's sqlite3 follows the DB-API 2.0 standard, the same cursor/execute/fetch patterns transfer almost unchanged to those databases through their respective drivers. ✅ Expert tab complete - I know foreign keys are OFF unless I enable PRAGMA foreign_keys = ON - I can register adapters/converters for custom types - I know SQLite suits single-writer/embedded use; Postgres/MySQL for high concurrency - I know sqlite3 follows DB-API 2.0, so patterns transfer to other databases Continue to [time →](/coding/languages/python/standard-library/time/) ## SOURCES - Python Standard Library — sqlite3. docs.python.org/3/library/sqlite3.html. - PEP 249 — Python Database API Specification v2.0. peps.python.org/pep-0249/. - Python Standard Library — sqlite3 placeholders and SQL injection. docs.python.org/3/library/sqlite3.html. - SQLite — Appropriate Uses For SQLite. sqlite.org/whentouse.html. **Primary source:** docs.python.org sqlite3 *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # time — Time Access and Conversions ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/time/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The time module provides low-level access to the system clock, sleeping, and performance measurement — the foundation beneath datetime.* ## CANONICAL DEFINITION The time module provides various time-related functions for accessing the system clock, pausing execution, and measuring elapsed time. It works primarily with timestamps (seconds since the Unix epoch) and struct_time tuples, sitting at a lower level than the datetime module. ## BEGINNER > The idea The time module is Python's connection to the system clock. It tells you the current time as a number, lets your program pause (`sleep`), and measures how long things take. For calendar dates and human-friendly times, use datetime — time is the lower-level layer underneath. ## Timestamps and sleeping ```python import time # The current time as a Unix timestamp (seconds since 1 Jan 1970 UTC) now = time.time() print(now) # e.g. 1781900000.123 — a float # Pause execution for a number of seconds print("Starting...") time.sleep(2) # wait 2 seconds (accepts floats: sleep(0.5)) print("...2 seconds later") # Convert a timestamp to a readable local time string print(time.ctime(now)) # e.g. 'Wed Jun 17 21:00:00 2026' # Get the local time as a struct (year, month, day, hour, ...) local = time.localtime(now) print(local.tm_year, local.tm_mon, local.tm_mday) # 2026 6 17 print(local.tm_hour, local.tm_min) # 21 0 ``` > time vs datetime Use **time** for timestamps, sleeping, and performance measurement. Use **datetime** for working with calendar dates, doing date arithmetic, and formatting dates for people. They interoperate: `datetime.fromtimestamp(time.time())` bridges the two. ## Measuring elapsed time ```python import time # perf_counter — the RIGHT tool for timing code (high resolution) start = time.perf_counter() total = sum(range(10_000_000)) elapsed = time.perf_counter() - start print(f"Took {elapsed:.4f} seconds") # Do NOT use time.time() for measuring durations: # it can jump backwards if the system clock is adjusted (NTP, DST). # perf_counter is monotonic — it never goes backwards. # A simple reusable timer pattern def time_it(func, *args): start = time.perf_counter() result = func(*args) return result, time.perf_counter() - start result, duration = time_it(sum, range(1_000_000)) print(f"Result {result} in {duration:.4f}s") # perf_counter_ns — nanosecond integer version (avoids float rounding) start = time.perf_counter_ns() x = 2 ** 1000 print(f"{time.perf_counter_ns() - start} nanoseconds") ``` ✅ Beginner tab complete - I can read the current timestamp with time.time() - I can pause execution with time.sleep() (accepts floats) - I know time is for timestamps/sleep/timing; datetime is for calendar dates - I can convert a timestamp with localtime() and ctime() Continue to [shutil →](/coding/languages/python/standard-library/shutil/) ## INTERMEDIATE ## Formatting and parsing time The time module can format struct_time into strings and parse strings back, using the same directives as datetime's strftime. ```python import time now = time.localtime() # strftime — struct_time -> formatted string print(time.strftime("%Y-%m-%d %H:%M:%S", now)) # '2026-06-17 21:00:00' print(time.strftime("%A, %B %d", now)) # 'Wednesday, June 17' # strptime — parse a string -> struct_time parsed = time.strptime("2026-06-17", "%Y-%m-%d") print(parsed.tm_year, parsed.tm_mon) # 2026 6 # gmtime — UTC instead of local time utc = time.gmtime() print(time.strftime("%Y-%m-%d %H:%M UTC", utc)) # mktime — struct_time (local) -> timestamp ts = time.mktime(now) print(ts) # back to a float timestamp # timezone info print(time.timezone) # seconds west of UTC (non-DST) print(time.tzname) # ('IST', 'IST') or local zone names ``` ## Choosing the right clock ```python import time # time.time() — wall-clock time, can jump (NTP sync, DST, manual change) # USE FOR: timestamps, "what time is it", logging when something happened print(time.time()) # time.monotonic() — never goes backwards, no fixed reference point # USE FOR: timeouts, rate limiting, "has 5 seconds passed?" start = time.monotonic() # ... do work ... if time.monotonic() - start > 5: print("timed out") # time.perf_counter() — highest resolution, for benchmarking # USE FOR: measuring how long code takes (most precise) print(time.perf_counter()) # time.process_time() — CPU time of THIS process only (excludes sleep) # USE FOR: measuring actual computation, ignoring I/O waits start = time.process_time() time.sleep(1) # process_time does NOT count sleep x = sum(range(1_000_000)) # this DOES count print(f"CPU time: {time.process_time() - start:.4f}s") # ~0.01, not ~1.0 ``` **Commonly confused:** - time.time() vs time.monotonic()/perf_counter(). Use time() for "what time is it" (timestamps). Use monotonic() or perf_counter() for "how much time has passed" (durations). time() can jump backwards when the system clock is adjusted, corrupting duration measurements. ✅ Intermediate tab complete - I use perf_counter() to measure how long code takes - I never measure durations with time.time() (it can jump backwards) - I can format and parse with strftime/strptime - I know monotonic for timeouts, process_time for CPU-only timing Continue to [shutil →](/coding/languages/python/standard-library/shutil/) ## EXPERT ## Clock characteristics, resolution, and get_clock_info Python exposes several distinct clocks because they answer different questions and have different guarantees. `time.get_clock_info()` reports the properties of each — whether it is monotonic, whether it is adjustable, and its resolution — letting you choose the right one programmatically and understand its precision on the current platform. ```python import time # Inspect the properties of each clock for name in ["time", "monotonic", "perf_counter", "process_time"]: info = time.get_clock_info(name) print(f"{name}: monotonic={info.monotonic}, " f"adjustable={info.adjustable}, " f"resolution={info.resolution}") # time: monotonic=False, adjustable=True (wall clock — can jump) # monotonic: monotonic=True, adjustable=False (never goes back) # perf_counter: monotonic=True, adjustable=False (highest resolution) # process_time: monotonic=True (CPU time only) # thread_time — CPU time of the current THREAD (3.7+) print(time.thread_time()) # The _ns variants return integer nanoseconds, avoiding float precision # loss for very short or very long durations print(time.time_ns()) # integer ns since epoch print(time.monotonic_ns()) # integer ns, monotonic print(time.perf_counter_ns()) # integer ns, highest resolution # clock_gettime / CLOCK constants — direct POSIX clock access (Unix) # import time # print(time.clock_gettime(time.CLOCK_MONOTONIC)) # Why _ns matters: a float has 53 bits of mantissa. For timestamps # that are billions of seconds, sub-microsecond precision is lost. # Integer nanoseconds preserve full precision. a = time.perf_counter_ns() b = time.perf_counter_ns() print(f"{b - a} ns elapsed") # exact integer difference ``` The relationship between `time` and `datetime` is layered: `datetime` is built on the same underlying system clock that `time.time()` reads, but adds calendar arithmetic, timezone handling, and human-readable formatting. The practical division is clear — reach for `time` when you need raw timestamps, to pause execution, or to measure performance (where `perf_counter` and the `_ns` variants are the correct tools); reach for `datetime` when you are working with dates as calendar concepts. For asynchronous code, neither `time.sleep` nor blocking measurement belongs in a coroutine — `asyncio` provides its own non-blocking `sleep` and event-loop clock. ✅ Expert tab complete - I can inspect clocks with time.get_clock_info() - I know the _ns variants avoid float precision loss - I know perf_counter and monotonic are non-adjustable; time() is adjustable - I know async code uses asyncio.sleep, not time.sleep Continue to [shutil →](/coding/languages/python/standard-library/shutil/) ## SOURCES - Python Standard Library — time. docs.python.org/3/library/time.html. - Python Standard Library — time.get_clock_info. docs.python.org/3/library/time.html#time.get_clock_info. - PEP 418 — Add monotonic time, performance counter, and process time functions. peps.python.org/pep-0418/. - PEP 564 — Add new time functions with nanosecond resolution. peps.python.org/pep-0564/. **Primary source:** docs.python.org time *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # shutil — High-Level File Operations ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/shutil/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *shutil handles whole-file and whole-directory operations — copying, moving, deleting trees, and creating archives — the high-level counterpart to os and pathlib.* ## CANONICAL DEFINITION The shutil module offers high-level operations on files and collections of files, especially functions that copy and remove entire directory trees, move files, and create or extract archives. It complements os and pathlib, which handle individual path and file operations. ## BEGINNER > The idea Where `pathlib` and `os` work with single files and paths, `shutil` does the big operations: copy a file with its metadata, copy or delete an entire folder tree, move things, and zip up directories. It is the module you reach for when "copy this whole folder" is the task. ## Copying files and directories ```python import shutil # Copy a single file (contents + permissions) shutil.copy("report.pdf", "backup/report.pdf") # copy2 also preserves metadata (timestamps) — usually what you want shutil.copy2("report.pdf", "backup/report.pdf") # Copy an ENTIRE directory tree shutil.copytree("project/", "project_backup/") # copytree with options shutil.copytree( "project/", "project_backup/", dirs_exist_ok=True, # don't fail if target exists (3.8+) ignore=shutil.ignore_patterns("*.pyc", "__pycache__", ".git") ) # Copy just the contents of a file object (low-level) with open("source.txt", "rb") as src, open("dest.txt", "wb") as dst: shutil.copyfileobj(src, dst) # efficient streaming copy ``` > copy vs copy2 vs copyfile `copyfile` copies only the bytes. `copy` copies bytes + permission bits. `copy2` copies bytes + permissions + timestamps and other metadata. Default to `copy2` when you want a faithful duplicate; use `copyfile` when you only care about contents. ## Moving and deleting ```python import shutil # Move a file or directory (also used for renaming) shutil.move("old_location/file.txt", "new_location/file.txt") shutil.move("temp.txt", "archive/") # into a directory # Delete an ENTIRE directory tree (files and subdirectories) shutil.rmtree("build/") # like 'rm -rf build/' # WARNING: rmtree is permanent and recursive — there is no undo. # Safely handle errors during rmtree shutil.rmtree("maybe_missing/", ignore_errors=True) # no error if absent # To remove a SINGLE file, use os or pathlib instead: import os os.remove("single_file.txt") # or Path("f.txt").unlink() ``` > rmtree is permanent `shutil.rmtree("folder/")` deletes the entire folder and everything inside it, recursively and irreversibly — the equivalent of `rm -rf`. There is no recycle bin. Double-check the path before calling it, especially if the path is built from variables. ✅ Beginner tab complete - I can copy a file with copy2 (preserving metadata) - I can copy a whole directory tree with copytree - I know copyfile (bytes) vs copy (bytes+perms) vs copy2 (bytes+perms+timestamps) - I can use dirs_exist_ok=True to merge into an existing directory Continue to [copy →](/coding/languages/python/standard-library/copy/) ## INTERMEDIATE ## Creating and extracting archives shutil can zip or tar an entire directory in one call, and extract archives just as easily — no need to handle the zipfile or tarfile modules directly for common cases. ```python import shutil # Create a .zip of an entire directory # make_archive(base_name, format, root_dir) shutil.make_archive("backup", "zip", "project/") # Creates backup.zip containing the contents of project/ # Other formats: 'tar', 'gztar' (.tar.gz), 'bztar', 'xztar' shutil.make_archive("backup", "gztar", "project/") # backup.tar.gz # See available formats on this system print([name for name, desc in shutil.get_archive_formats()]) # ['bztar', 'gztar', 'tar', 'xztar', 'zip'] # Extract an archive shutil.unpack_archive("backup.zip", "restored/") shutil.unpack_archive("data.tar.gz", "extracted/") # format auto-detected # A full backup-with-timestamp pattern from datetime import datetime stamp = datetime.now().strftime("%Y%m%d_%H%M%S") archive = shutil.make_archive(f"backup_{stamp}", "zip", "important_data/") print(f"Created {archive}") ``` ## Disk usage and finding executables ```python import shutil # Disk usage of the filesystem containing a path usage = shutil.disk_usage("/") print(f"Total: {usage.total // (2**30)} GB") print(f"Used: {usage.used // (2**30)} GB") print(f"Free: {usage.free // (2**30)} GB") # which — find an executable on PATH (like the Unix 'which' command) print(shutil.which("python3")) # /usr/bin/python3 (or None if not found) print(shutil.which("git")) # path to git, or None # Useful for checking whether a required tool is installed: if shutil.which("ffmpeg") is None: print("ffmpeg is not installed") # Get terminal size — for formatting CLI output size = shutil.get_terminal_size() print(f"Terminal: {size.columns} columns x {size.lines} lines") ``` **Commonly confused:** - shutil vs os vs pathlib. Use pathlib/os for single files and path manipulation (exists, rename one file, join paths). Use shutil for bulk operations: copying trees, deleting trees, moving, and archiving. shutil is the "do it to the whole folder" module. ✅ Intermediate tab complete - I can create a zip/tar of a directory with make_archive - I can extract an archive with unpack_archive - I can check free space with disk_usage - I can find an executable on PATH with shutil.which Continue to [copy →](/coding/languages/python/standard-library/copy/) ## EXPERT ## copy functions, metadata fidelity, and platform limits The family of copy functions in shutil forms a deliberate hierarchy of metadata fidelity, and understanding it matters when duplicates must be faithful. `copyfile` copies only data; `copymode` copies only permission bits; `copystat` copies permissions, timestamps, and flags but not data; `copy` combines copyfile + copymode; and `copy2` combines copyfile + copystat for the most complete duplicate. Even `copy2`, however, cannot always preserve every attribute — extended attributes, ACLs, and resource forks are platform-dependent and may be lost. ```python import shutil, os # copytree with a custom copy_function and ignore callable def log_copy(src, dst, *, follow_symlinks=True): print(f"copying {src} -> {dst}") return shutil.copy2(src, dst, follow_symlinks=follow_symlinks) shutil.copytree("src/", "dst/", copy_function=log_copy) # rmtree with an error handler (e.g. to fix read-only files on Windows) def handle_remove_readonly(func, path, exc_info): os.chmod(path, 0o700) # make writable func(path) # retry the removal shutil.rmtree("readonly_dir/", onexc=handle_remove_readonly) # 3.12+ onexc # Registering a custom archive format def make_custom(base_name, base_dir, **kwargs): ... # custom archiving logic # shutil.register_archive_format("myfmt", make_custom, [], "My format") # move semantics: if dst is on a different filesystem, move falls back # to copy2 + rmtree (not an atomic rename). On the SAME filesystem it # uses os.rename (atomic and instant). This matters for large files: # same fs -> instant metadata rename # cross fs -> full byte copy then delete (slow, not atomic) shutil.move("/tmp/bigfile", "/home/user/bigfile") # may be a full copy # COPY_BUFSIZE controls the streaming buffer size for copyfileobj # Larger buffers can speed up copies of very large files with open("huge.bin","rb") as s, open("out.bin","wb") as d: shutil.copyfileobj(s, d, length=16 * 1024 * 1024) # 16 MB buffer ``` An important performance and correctness subtlety lies in `shutil.move`: when source and destination are on the same filesystem, it performs an atomic `os.rename` — instantaneous regardless of file size, because only directory metadata changes. When they are on different filesystems (for example moving from `/tmp` to a home directory on a separate mount), it must fall back to copying every byte and then deleting the original, which is slow for large files and not atomic. For the common case of copying single large files, modern Python (3.8+) also uses platform-specific fast-copy syscalls like `sendfile` on Linux and `fcopyfile` on macOS under the hood, making `copyfile` and `copy2` substantially faster than a naive read/write loop. ✅ Expert tab complete - I know the copyfile/copymode/copystat/copy/copy2 fidelity hierarchy - I know move is atomic on the same filesystem but a full copy across filesystems - I know copy2/copyfile use fast-copy syscalls (sendfile/fcopyfile) on modern Python - I can pass a custom copy_function or error handler to copytree/rmtree Continue to [copy →](/coding/languages/python/standard-library/copy/) ## SOURCES - Python Standard Library — shutil. docs.python.org/3/library/shutil.html. - Python Standard Library — shutil.copytree. docs.python.org/3/library/shutil.html#shutil.copytree. - Python Standard Library — shutil archiving operations. docs.python.org/3/library/shutil.html#archiving-operations. **Primary source:** docs.python.org shutil *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # copy — Shallow and Deep Copy Operations ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/copy/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The copy module duplicates objects — but the difference between a shallow copy and a deep copy is the source of some of Python’s most common and confusing bugs.* ## CANONICAL DEFINITION The copy module provides copy() for shallow copies and deepcopy() for deep copies of arbitrary Python objects. A shallow copy duplicates the outer object but shares references to nested objects; a deep copy recursively duplicates everything, producing a fully independent object. ## BEGINNER > The idea Assigning `b = a` does not copy a list — it makes `b` another name for the same list, so changing one changes the other. The copy module makes real duplicates. But there are two kinds — shallow and deep — and knowing which you need prevents a whole class of baffling bugs. ## Why copying is tricky ```python # Assignment does NOT copy — both names point to the SAME object original = [1, 2, 3] alias = original # NOT a copy — same list alias.append(4) print(original) # [1, 2, 3, 4] — changed too! They're the same list. # Check identity: they ARE the same object print(original is alias) # True # To get a real copy, use the copy module (or list-specific methods) import copy real_copy = copy.copy(original) # a separate list real_copy.append(99) print(original) # [1, 2, 3, 4] — unaffected this time print(original is real_copy) # False — different objects # Lists also have shortcut copy methods (all shallow): shortcut1 = original[:] # slice copy shortcut2 = original.copy() # .copy() method shortcut3 = list(original) # constructor ``` > = is not a copy `b = a` creates a new *name* for the same object, not a new object. This is fine for immutable values (numbers, strings) where it never matters, but for mutable objects (lists, dicts, sets, custom classes) it means changes through one name are visible through the other. When you need independence, you must copy explicitly. ## Shallow copy — one level deep ```python import copy # A shallow copy duplicates the OUTER object but shares the INNER ones original = [[1, 2], [3, 4]] shallow = copy.copy(original) # The outer list is independent — appending to one doesn't affect the other shallow.append([5, 6]) print(original) # [[1, 2], [3, 4]] — outer unaffected # BUT the inner lists are SHARED — they're the same objects shallow[0].append(99) print(original) # [[1, 2, 99], [3, 4]] — inner list changed! print(original[0] is shallow[0]) # True — same inner list # This is the shallow-copy trap: one level of copying isn't enough # when your object contains other mutable objects. ``` ✅ Beginner tab complete - I know b = a creates a new name for the same object, not a copy - I can make a copy with copy.copy() or list/dict .copy() - I know copying only matters for mutable objects - I understand identity (is) vs equality (==) Continue to [pickle →](/coding/languages/python/standard-library/pickle/) ## INTERMEDIATE ## Deep copy — all the way down A deep copy recursively duplicates everything, so the copy shares nothing with the original. This is what you usually want for nested structures you intend to modify independently. ```python import copy original = [[1, 2], [3, 4]] deep = copy.deepcopy(original) # Now the inner lists are independent too deep[0].append(99) print(original) # [[1, 2], [3, 4]] — completely unaffected print(original[0] is deep[0]) # False — different inner lists # Real-world example: copying a nested configuration config = { "server": {"host": "localhost", "ports": [8000, 8001]}, "debug": True, } # Shallow copy shares the nested dict and list — dangerous shallow = copy.copy(config) shallow["server"]["host"] = "production" print(config["server"]["host"]) # 'production' — original changed! # Deep copy is safe for independent modification config2 = copy.deepcopy(config) config2["server"]["host"] = "staging" print(config["server"]["host"]) # 'production' — original safe # Deep copy handles cycles correctly (objects referencing themselves) a = [1, 2] a.append(a) # a contains itself! b = copy.deepcopy(a) # works — deepcopy tracks already-copied objects print(b[2] is b) # True — the cycle is preserved in the copy ``` > Deep copy can be slow `deepcopy` recursively duplicates every nested object, which costs time and memory for large structures. Use it when you genuinely need full independence. For flat structures (no nested mutable objects), a shallow copy is faster and sufficient. Do not reach for deepcopy reflexively. ## Controlling how objects are copied ```python import copy # Classes can control copying via __copy__ and __deepcopy__ class Document: def __init__(self, title, tags): self.title = title self.tags = tags self.cache = {} # expensive, shouldn't be copied def __copy__(self): # Custom shallow copy — share tags, fresh cache new = Document(self.title, self.tags) return new def __deepcopy__(self, memo): # memo prevents infinite loops on cyclic references new = Document( copy.deepcopy(self.title, memo), copy.deepcopy(self.tags, memo), ) return new # cache deliberately left empty doc = Document("Report", ["draft", "2026"]) clone = copy.deepcopy(doc) print(clone.title, clone.tags) # Report ['draft', '2026'] # The 'memo' dict maps id(original) -> copy, so shared/cyclic # references are copied once and reused — preserving structure. ``` **Commonly confused:** - Shallow vs deep — the one-sentence rule. A shallow copy duplicates the container but shares the contents; a deep copy duplicates everything. If your object contains other mutable objects you intend to modify, you need deepcopy. If it is flat, shallow is enough and faster. ✅ Intermediate tab complete - I know shallow copy shares nested objects; deep copy duplicates them - I use deepcopy when nested mutable objects must be independent - I know deepcopy handles cyclic references correctly - I can customise copying with __copy__ and __deepcopy__ Continue to [pickle →](/coding/languages/python/standard-library/pickle/) ## EXPERT ## The memo dictionary, __reduce__, and what gets shared The engine of `deepcopy` is the *memo* dictionary — a map from `id()` of each original object to its copy. Before copying any object, deepcopy checks the memo; if the object has already been copied, it reuses that copy rather than making a second one. This single mechanism accomplishes two critical things: it handles cyclic references without infinite recursion, and it preserves shared structure, so that if two parts of your object graph point to the same sub-object, the copy's two parts also point to a single shared copy rather than two separate duplicates. ```python import copy # Shared structure is preserved by the memo shared = [1, 2, 3] original = {"a": shared, "b": shared} # both keys point to ONE list print(original["a"] is original["b"]) # True deep = copy.deepcopy(original) print(deep["a"] is deep["b"]) # True — STILL shared in the copy! # deepcopy copied 'shared' once and reused it for both keys. # What deepcopy does NOT copy (treated as atomic / immutable): # int, float, str, bool, None, and tuples of immutables are not # duplicated — they're immutable, so sharing is safe. x = "immutable string" print(copy.deepcopy(x) is x) # True — no copy needed # How copy decides: it dispatches on type via copy._deepcopy_dispatch, # falls back to __deepcopy__, then __reduce_ex__ (the pickle protocol). # This is why most objects are deep-copyable without any custom code — # the same machinery that powers pickle powers deepcopy. # Objects that CANNOT be meaningfully copied (files, sockets, locks) # should define __copy__/__deepcopy__ or __reduce__ to handle it, # or raise. By default, copying them may share the underlying resource. # Manual memo use — pre-seed to control copying memo = {} original = [1, 2, 3] memo[id(original)] = original # tell deepcopy: reuse, don't copy this result = copy.deepcopy({"data": original}, memo) print(result["data"] is original) # True — we forced sharing ``` The fallback chain reveals why `copy` and `pickle` are deeply related: when an object defines no `__copy__` or `__deepcopy__`, the copy module dispatches to `__reduce_ex__` — the same protocol pickle uses to serialize objects. This is why the vast majority of objects are deep-copyable with no special code, and also why objects that cannot be pickled (open files, sockets, locks, generators) generally cannot be deep-copied either, and require an explicit `__deepcopy__` that decides how to handle the unpicklable resource. For controlling this, defining `__deepcopy__(self, memo)` and threading the `memo` through any nested `deepcopy` calls is essential — failing to pass `memo` breaks cycle detection and shared-structure preservation for everything beneath that object. ✅ Expert tab complete - I know the memo dict maps id(original) to copy, handling cycles - I know deepcopy preserves shared structure (one shared object stays shared) - I know copy falls back to __reduce_ex__ (the pickle protocol) - I know unpicklable objects (files, sockets) generally cannot be deep-copied Continue to [pickle →](/coding/languages/python/standard-library/pickle/) ## SOURCES - Python Standard Library — copy. docs.python.org/3/library/copy.html. - Python Standard Library — copy.deepcopy and the memo dictionary. docs.python.org/3/library/copy.html. - Python Standard Library — __reduce_ex__ (shared copy/pickle protocol). docs.python.org/3/library/pickle.html. **Primary source:** docs.python.org copy *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # concurrent.futures — High-Level Parallelism ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/concurrent-futures/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *concurrent.futures runs functions in parallel across threads or processes with one simple, uniform interface — the easiest way to speed up I/O-bound and CPU-bound work.* ## CANONICAL DEFINITION The concurrent.futures module provides a high-level interface for asynchronously executing callables using pools of threads (ThreadPoolExecutor) or processes (ProcessPoolExecutor). A Future object represents a pending result. The same API works for both executors, making it easy to switch between thread and process parallelism. ## BEGINNER > The idea Instead of doing slow tasks one after another, `concurrent.futures` runs many at once — across threads or processes — with almost no extra code. Submit work to a pool, and it hands back results as they finish. It is the simplest doorway into parallelism in Python. ## The executor and map An *executor* manages a pool of workers. The easiest entry point is `executor.map`, which works like the built-in `map` but runs the calls in parallel. ```python from concurrent.futures import ThreadPoolExecutor import urllib.request urls = [ "https://example.com", "https://python.org", "https://docs.python.org", ] def fetch(url): with urllib.request.urlopen(url) as response: return url, len(response.read()) # Sequentially this would wait for each download in turn. # With a thread pool, they download concurrently. with ThreadPoolExecutor(max_workers=3) as executor: # map runs fetch on every url in parallel, yields results in order for url, size in executor.map(fetch, urls): print(f"{url}: {size} bytes") # The 'with' block automatically waits for all tasks and shuts down # the pool when done — no manual cleanup needed. ``` > map returns results in input order `executor.map` yields results in the same order as the inputs, even though the tasks finish in a different order. If you want results as soon as each one completes (not in order), use `as_completed` with `submit`, shown next. ## Futures and submit ```python from concurrent.futures import ThreadPoolExecutor, as_completed import time def slow_task(n): time.sleep(n) # simulate work taking n seconds return f"task {n} done" with ThreadPoolExecutor(max_workers=4) as executor: # submit() schedules one call and returns a Future immediately futures = [executor.submit(slow_task, n) for n in [3, 1, 2]] # as_completed yields each future as soon as it finishes # (so the 1-second task comes back first, not the 3-second one) for future in as_completed(futures): print(future.result()) # blocks until THIS future is ready # A Future also lets you check status and handle errors fut = executor.submit(slow_task, 1) print(fut.done()) # False (probably still running) result = fut.result(timeout=5) # wait up to 5s; raises on timeout # If the task raised an exception, .result() re-raises it here: # try: # fut.result() # except SomeError as e: # handle(e) ``` ✅ Beginner tab complete - I can run tasks in parallel with ThreadPoolExecutor and map - I know the with-block waits for all tasks and shuts down the pool - I know map returns results in input order - I can fetch multiple URLs concurrently Continue to [queue →](/coding/languages/python/standard-library/queue/) ## INTERMEDIATE ## Threads vs processes — the crucial choice The two executors look identical but suit opposite workloads. The deciding factor is Python's Global Interpreter Lock (GIL), which lets only one thread run Python bytecode at a time. ```python from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor import math # ThreadPoolExecutor — best for I/O-BOUND work # (network requests, file reads, database queries) # While one thread waits for I/O, others run. The GIL is released # during I/O, so threads genuinely overlap waiting time. # ProcessPoolExecutor — best for CPU-BOUND work # (heavy computation, number crunching, image processing) # Each process has its own interpreter and GIL, so they run on # multiple CPU cores in true parallel. def cpu_heavy(n): return sum(math.isqrt(i) for i in range(n)) numbers = [10_000_000] * 8 # For CPU-bound work, processes use all cores — threads would not help # because the GIL serialises Python bytecode execution. with ProcessPoolExecutor() as executor: # defaults to os.cpu_count() results = list(executor.map(cpu_heavy, numbers)) print(f"Computed {len(results)} results across CPU cores") # Rule of thumb: # Waiting on I/O? -> ThreadPoolExecutor # Burning CPU? -> ProcessPoolExecutor # The API is identical, so switching is a one-line change. ``` > The GIL is why this choice matters Python's Global Interpreter Lock means threads cannot run Python bytecode simultaneously — so for CPU-bound work, threads give no speedup. Processes sidestep the GIL by each having their own interpreter. For I/O-bound work, threads are ideal because the GIL is released while waiting. ## Patterns and pitfalls ```python from concurrent.futures import ThreadPoolExecutor, as_completed # Pattern: map inputs to futures so you know which result is which def process(item): return item * 2 items = ["a", "b", "c"] with ThreadPoolExecutor() as executor: # A dict mapping each future back to its input future_to_item = {executor.submit(process, item): item for item in items} for future in as_completed(future_to_item): item = future_to_item[future] try: result = future.result() print(f"{item} -> {result}") except Exception as e: print(f"{item} failed: {e}") # one task failing won't stop others # PITFALL: ProcessPoolExecutor must run inside if __name__ == "__main__" # on Windows/macOS (spawn start method), or it recursively spawns. # def main(): # with ProcessPoolExecutor() as ex: # ... # if __name__ == "__main__": # main() # PITFALL: arguments and return values for ProcessPoolExecutor must be # picklable (they're sent between processes). Lambdas and local # functions can't be pickled — use module-level functions. ``` **Commonly confused:** - map vs submit + as_completed. Use map when you want results in input order and a clean loop. Use submit + as_completed when you want each result the moment it is ready, or need per-task error handling and status checks. ✅ Intermediate tab complete - I can submit tasks and get Future objects - I use as_completed for results as soon as they finish - I use ThreadPoolExecutor for I/O-bound, ProcessPoolExecutor for CPU-bound - I know the GIL is why CPU-bound work needs processes, not threads Continue to [queue →](/coding/languages/python/standard-library/queue/) ## EXPERT ## Future internals, the GIL’s future, and choosing the right tool A `Future` is a synchronisation primitive representing a computation that may not have completed. Beyond `result()` and `done()`, it supports `add_done_callback` to register functions that fire on completion, `cancel()` to cancel not-yet-started tasks, and `exception()` to retrieve a raised exception without re-raising it. The `wait` function offers fine-grained control over waiting for first-completion, first-exception, or all-completed across a set of futures. ```python from concurrent.futures import (ThreadPoolExecutor, wait, FIRST_COMPLETED, ALL_COMPLETED) with ThreadPoolExecutor() as executor: futures = [executor.submit(pow, 2, n) for n in range(5)] # Callbacks fire when a future completes (in a worker thread) futures[0].add_done_callback(lambda f: print("first done:", f.result())) # wait() with return_when gives precise control done, not_done = wait(futures, timeout=10, return_when=ALL_COMPLETED) print(f"{len(done)} completed, {len(not_done)} pending") # exception() retrieves an error without raising it for f in done: err = f.exception() # None if successful if err: print(f"failed with {err}") # initializer runs once per worker — useful for per-worker setup def init_worker(): # e.g. open a per-process database connection or set up logging pass with ThreadPoolExecutor(max_workers=4, initializer=init_worker) as ex: ... # chunksize matters for ProcessPoolExecutor.map with many small tasks: # batching reduces inter-process overhead # with ProcessPoolExecutor() as ex: # ex.map(func, big_iterable, chunksize=100) # InterpreterPoolExecutor (3.14+) — sub-interpreters, each with its own # GIL, giving process-like parallelism with lower overhead than processes. ``` The concurrency landscape is shifting. PEP 703 introduced an experimental free-threaded build of CPython (3.13+) that can disable the GIL entirely, which would let `ThreadPoolExecutor` achieve true multi-core parallelism for CPU-bound work — historically the exclusive domain of `ProcessPoolExecutor`. Until free-threading is the default and the ecosystem adapts, the established guidance holds: `ThreadPoolExecutor` for I/O-bound concurrency, `ProcessPoolExecutor` for CPU-bound parallelism, and `asyncio` for very high-concurrency I/O with explicit `async`/`await` control. The strength of `concurrent.futures` is that its uniform `Executor` interface makes moving between thread and process pools a one-line change, letting you defer and revise the threads-versus-processes decision as you measure where the real bottleneck lies. ✅ Expert tab complete - I can use add_done_callback, cancel, and exception on a Future - I can use wait with return_when for fine-grained control - I know ProcessPoolExecutor args/returns must be picklable - I know free-threading (PEP 703) may let threads do true CPU parallelism Continue to [queue →](/coding/languages/python/standard-library/queue/) ## SOURCES - Python Standard Library — concurrent.futures. docs.python.org/3/library/concurrent.futures.html. - Python Standard Library — ProcessPoolExecutor. docs.python.org/3/library/concurrent.futures.html. - PEP 3148 — futures: execute computations asynchronously. peps.python.org/pep-3148/. - PEP 703 — Making the Global Interpreter Lock Optional in CPython. peps.python.org/pep-0703/. **Primary source:** docs.python.org concurrent.futures *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # queue — Thread-Safe Queues ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/queue/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The queue module provides thread-safe queues for passing work between threads safely — the standard way to coordinate producers and consumers without manual locking.* ## CANONICAL DEFINITION The queue module implements multi-producer, multi-consumer queues that are thread-safe, meaning multiple threads can add and remove items without corrupting the queue or needing explicit locks. It provides FIFO (Queue), LIFO (LifoQueue), and priority (PriorityQueue) variants. ## BEGINNER > The idea When several threads need to hand work to each other, sharing a plain list is dangerous — two threads modifying it at once can corrupt it. A `queue.Queue` is built to be used by many threads simultaneously and safely. One thread puts items in; another takes them out; the queue handles all the locking. ## Why a thread-safe queue The queue module's queues handle all the locking internally, so threads can `put` and `get` freely without stepping on each other. This is the recommended way to communicate between threads in Python. ```python import queue # A FIFO queue — first in, first out q = queue.Queue() # Add items (put) — thread-safe q.put("first") q.put("second") q.put("third") # Remove items (get) — thread-safe, returns in FIFO order print(q.get()) # "first" print(q.get()) # "second" # Check size and emptiness (approximate in multithreaded use) print(q.qsize()) # 1 print(q.empty()) # False # Optional maximum size — put() blocks when the queue is full limited = queue.Queue(maxsize=10) limited.put("item") # blocks if 10 items already waiting # get() on an empty queue BLOCKS until an item is available — # this is the key behaviour that makes producer/consumer work. ``` > get() blocks by default Calling `q.get()` on an empty queue waits until something is available — it does not return immediately or raise. This blocking is the feature: a consumer thread can sit on `get()` and wake up the moment work arrives. Pass `block=False` or use `get_nowait()` to raise `queue.Empty` instead of waiting. ## The producer-consumer pattern ```python import queue import threading import time work_queue = queue.Queue() def producer(): for i in range(5): item = f"job-{i}" work_queue.put(item) print(f"Produced {item}") time.sleep(0.1) work_queue.put(None) # sentinel: signal "no more work" def consumer(): while True: item = work_queue.get() # blocks until an item is available if item is None: # sentinel received -> stop break print(f" Consumed {item}") time.sleep(0.2) # Run producer and consumer on separate threads t1 = threading.Thread(target=producer) t2 = threading.Thread(target=consumer) t1.start(); t2.start() t1.join(); t2.join() print("All work done") # The queue safely passes items from producer to consumer with no # manual locks — even if you ran several producers and consumers. ``` ✅ Beginner tab complete - I know queue.Queue is thread-safe; a plain list is not - I can put and get items safely from multiple threads - I know get() blocks on an empty queue by default - I can use get_nowait() to raise queue.Empty instead of waiting Continue to [socket →](/coding/languages/python/standard-library/socket/) ## INTERMEDIATE ## Queue variants: LIFO and priority Beyond the standard FIFO queue, the module offers a stack-like LIFO queue and a priority queue that always returns the smallest item first. ```python import queue # LifoQueue — last in, first out (a thread-safe stack) stack = queue.LifoQueue() stack.put(1); stack.put(2); stack.put(3) print(stack.get()) # 3 (most recent first) print(stack.get()) # 2 # PriorityQueue — always returns the LOWEST value first pq = queue.PriorityQueue() pq.put((2, "medium priority")) pq.put((1, "high priority")) # lower number = higher priority pq.put((3, "low priority")) print(pq.get()) # (1, 'high priority') print(pq.get()) # (2, 'medium priority') # Common pattern: (priority, data) tuples. If priorities tie, Python # compares the data, so include a tiebreaker (e.g. a counter) when the # data isn't comparable: import itertools counter = itertools.count() pq = queue.PriorityQueue() pq.put((1, next(counter), {"task": "a"})) # (priority, seq, data) pq.put((1, next(counter), {"task": "b"})) # seq breaks the tie safely print(pq.get()[2]) # {'task': 'a'} — first inserted at this priority ``` ## Coordinating completion with join and task_done ```python import queue import threading q = queue.Queue() def worker(): while True: item = q.get() # get the next task try: print(f"Processing {item}") # ... do the work ... finally: q.task_done() # signal THIS task is complete # Start a pool of daemon worker threads for _ in range(3): threading.Thread(target=worker, daemon=True).start() # Enqueue 10 tasks for item in range(10): q.put(item) # Block until EVERY enqueued task has had task_done() called q.join() # waits for the count to reach zero print("All tasks completed") # How it works: put() increments an internal counter; task_done() # decrements it; join() blocks until the counter hits zero. This lets # the main thread wait for all work to finish without a sentinel. ``` > Every get() needs a matching task_done() When using `q.join()`, each item you `get()` must be followed by a `task_done()` call (use a `try/finally` to guarantee it). Miss one and `join()` blocks forever; call it too many times and you get a ValueError. **Commonly confused:** - queue.Queue vs collections.deque. A deque is fast and thread-safe for appends/pops but has no blocking get, no maxsize blocking, and no task tracking. Use queue.Queue for inter-thread communication (blocking, join); use deque as a general-purpose double-ended queue within one thread. ✅ Intermediate tab complete - I can use LifoQueue (stack) and PriorityQueue (lowest first) - I use (priority, counter, data) tuples to break priority ties - I call task_done() after each get() when using join() - I know join() blocks until every task is marked done Continue to [socket →](/coding/languages/python/standard-library/socket/) ## EXPERT ## Blocking semantics, SimpleQueue, and the shutdown method Every `get` and `put` accepts `block` and `timeout` parameters giving precise control: `get(block=False)` (equivalently `get_nowait()`) raises `queue.Empty` immediately if nothing is available, while `get(timeout=2)` waits up to two seconds before raising. The symmetric behaviour applies to `put` on a bounded queue, raising `queue.Full`. These let you build responsive workers that poll, time out, or fall back rather than blocking indefinitely. ```python import queue q = queue.Queue(maxsize=5) # Non-blocking and timed operations try: q.put("x", timeout=1.0) # wait up to 1s if full, else queue.Full item = q.get(timeout=2.0) # wait up to 2s if empty, else queue.Empty except queue.Full: print("queue was full") except queue.Empty: print("nothing available in time") # SimpleQueue (3.7+) — a simpler, unbounded FIFO with fewer features # but stronger guarantees: its put() never blocks and is reentrant-safe, # making it usable from signal handlers and __del__ methods. sq = queue.SimpleQueue() sq.put("safe even from a signal handler") print(sq.get()) # Queue.shutdown (3.13+) — cleanly stop a queue # After shutdown(), put() raises ShutDownError; get() drains remaining # items then raises ShutDownError, so worker loops exit without sentinels. # q2 = queue.Queue() # q2.shutdown() # signal no more work; replaces sentinels # Subclassing to customise the underlying container — override _put, # _get, _init. PriorityQueue and LifoQueue are themselves just Queue # subclasses overriding these three methods over a list/heap. class MaxPriorityQueue(queue.PriorityQueue): def _put(self, item): # invert priority to pop the LARGEST first super()._put((-item[0], item[1])) # Thread-safety note: qsize(), empty(), and full() are reliable at the # instant called but may be stale immediately after in a multithreaded # program. Never gate logic on them; rely on blocking get/put or the # Empty/Full exceptions instead. ``` A subtle but important design point distinguishes `queue.Queue` from `queue.SimpleQueue`: the former supports the full `task_done`/`join` task-tracking protocol and bounded sizes, while the latter is a minimal, unbounded FIFO whose `put` is guaranteed never to block and is safe to call from contexts like signal handlers and finalizers where the more complex queue could deadlock. The newer `shutdown` method (Python 3.13) modernises the producer-consumer pattern by replacing the traditional sentinel-value approach — instead of putting `None` markers to tell workers to stop, you call `shutdown()` and worker `get()` calls raise `ShutDownError` once the queue drains, giving a cleaner and less error-prone shutdown. For the relationship to other tools: `queue` is for threads, `multiprocessing.Queue` and `multiprocessing.Manager().Queue()` for processes, and `asyncio.Queue` for coroutines — each matching its concurrency model, all sharing the same conceptual put/get interface. ✅ Expert tab complete - I can use block and timeout to control get/put waiting - I know SimpleQueue is signal-safe and never blocks on put - I know shutdown() (3.13) replaces sentinel values for clean stops - I know queue is for threads; multiprocessing.Queue for processes Continue to [socket →](/coding/languages/python/standard-library/socket/) ## SOURCES - Python Standard Library — queue. docs.python.org/3/library/queue.html. - Python Standard Library — Queue.task_done and join. docs.python.org/3/library/queue.html. - Python Standard Library — queue.SimpleQueue. docs.python.org/3/library/queue.html. - Python Standard Library — multiprocessing.Queue (for processes). docs.python.org/3/library/multiprocessing.html. **Primary source:** docs.python.org queue *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # socket — Low-Level Networking ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/socket/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *The socket module is Python’s direct interface to the network — the foundation beneath every HTTP library, web framework, and networked application.* ## CANONICAL DEFINITION The socket module provides access to the BSD socket interface, the low-level networking primitive used to send and receive data across a network. A socket is an endpoint of a communication channel, identified by an address and port, supporting protocols such as TCP (reliable streams) and UDP (datagrams). ## BEGINNER > The idea A socket is a phone line between two programs over a network. One side listens for a call (the server), the other dials in (the client), and once connected they send bytes back and forth. Every web request, chat message, and API call ultimately rides on a socket. ## What a socket is A socket is an endpoint for network communication, identified by an IP address and a port number. The two main types are TCP (a reliable, ordered byte stream — like a phone call) and UDP (fast, connectionless messages — like postcards that may arrive out of order or not at all). ```python import socket # Create a TCP socket # AF_INET = IPv4 addressing, SOCK_STREAM = TCP (reliable stream) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # A UDP socket would use SOCK_DGRAM instead # u = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # Look up the IP address behind a hostname (DNS) print(socket.gethostbyname("example.com")) # e.g. '93.184.216.34' # Your own machine's hostname print(socket.gethostname()) # Always close a socket when done (or use a with-block) s.close() # Ports below 1024 are "well-known" (80=HTTP, 443=HTTPS, 22=SSH). # Use ports above 1024 for your own applications. ``` > Sockets send bytes, not strings Everything sent over a socket is `bytes`, not `str`. Encode before sending (`"hello".encode()`) and decode after receiving (`data.decode()`). Forgetting this is the most common first-time socket error. ## A TCP client ```python import socket # Connect to a server and exchange data — using a with-block for cleanup with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.connect(("example.com", 80)) # (host, port) tuple # Send an HTTP request (must be bytes) request = "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n" s.sendall(request.encode()) # sendall sends ALL the bytes # Receive the response in chunks until the server closes chunks = [] while True: data = s.recv(4096) # read up to 4096 bytes if not data: # empty bytes = connection closed break chunks.append(data) response = b"".join(chunks).decode("utf-8", errors="replace") print(response[:200]) # first 200 chars of the response # In real code you would use the 'requests' library or urllib — this # shows what those libraries do underneath. ``` ✅ Beginner tab complete - I know a socket is a network endpoint (address + port) - I know TCP is a reliable stream; UDP is fast connectionless messages - I can write a TCP client: connect, sendall, recv in a loop - I always encode strings to bytes before sending Continue to [http.server →](/coding/languages/python/standard-library/http-server/) ## INTERMEDIATE ## A TCP server A server binds to an address, listens for connections, accepts them one at a time, and communicates with each client. This four-step pattern — bind, listen, accept, recv/send — is the heart of every network server. ```python import socket HOST = "127.0.0.1" # localhost — only this machine can connect PORT = 65432 # a port above 1024 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server: # Allow reusing the address immediately after the program restarts server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) server.bind((HOST, PORT)) # claim the address and port server.listen() # start listening for connections print(f"Listening on {HOST}:{PORT}") while True: # accept() blocks until a client connects, returning a NEW # socket for that specific client plus their address conn, addr = server.accept() with conn: print(f"Connected by {addr}") while True: data = conn.recv(1024) if not data: break # client disconnected # Echo the data back, uppercased conn.sendall(data.upper()) ``` > This server handles one client at a time The loop above serves connections sequentially — a second client waits until the first disconnects. Real servers handle many clients concurrently using threads, `selectors`, or `asyncio`. For anything beyond learning, a framework built on these primitives is the right choice. ## UDP and connection details ```python import socket # UDP — connectionless. No connect/accept; you send to an address directly. # UDP server server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) server.bind(("127.0.0.1", 9999)) data, addr = server.recvfrom(1024) # returns (data, sender_address) server.sendto(b"reply", addr) # send back to that address # UDP client client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) client.sendto(b"hello", ("127.0.0.1", 9999)) # no connection needed reply, _ = client.recvfrom(1024) # Timeouts prevent a socket from blocking forever s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(5.0) # raise socket.timeout after 5 seconds try: s.connect(("example.com", 80)) except socket.timeout: print("Connection timed out") except socket.gaierror: print("Could not resolve the hostname (DNS failure)") finally: s.close() # TCP guarantees order and delivery; UDP is faster but does neither. # Use TCP for web, email, files. Use UDP for video, games, DNS. ``` **Commonly confused:** - send vs sendall. send() may transmit only part of the data and returns how many bytes were sent; you would have to loop. sendall() keeps going until everything is sent (or it errors). Use sendall unless you have a specific reason not to. ✅ Intermediate tab complete - I can write a TCP server: bind, listen, accept, recv/send - I know this basic server handles one client at a time - I can use UDP with sendto/recvfrom - I use settimeout to prevent indefinite blocking Continue to [http.server →](/coding/languages/python/standard-library/http-server/) ## EXPERT ## Blocking vs non-blocking, selectors, and the path to asyncio By default sockets are *blocking*: `recv`, `accept`, and `connect` halt the thread until they complete. This is simple but does not scale — one thread per connection is expensive at thousands of connections. The solutions are non-blocking sockets combined with an I/O multiplexer (`selectors`), or the higher-level `asyncio` framework built precisely on these primitives. ```python import socket, selectors # A selector lets ONE thread watch many sockets, reacting only to # those that are ready — the foundation of high-concurrency servers. sel = selectors.DefaultSelector() server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.bind(("127.0.0.1", 65432)) server.listen() server.setblocking(False) # non-blocking mode sel.register(server, selectors.EVENT_READ, data=None) def event_loop(): while True: events = sel.select(timeout=None) # wait for ANY socket to be ready for key, mask in events: if key.data is None: # The listening socket is ready -> accept a new client conn, addr = key.fileobj.accept() conn.setblocking(False) sel.register(conn, selectors.EVENT_READ, data=addr) else: # A client socket is ready -> read without blocking data = key.fileobj.recv(1024) if data: key.fileobj.sendall(data.upper()) else: sel.unregister(key.fileobj) key.fileobj.close() # socketserver — the stdlib's higher-level server framework import socketserver class EchoHandler(socketserver.BaseRequestHandler): def handle(self): data = self.request.recv(1024) self.request.sendall(data.upper()) # with socketserver.ThreadingTCPServer(("127.0.0.1", 0), EchoHandler) as srv: # srv.serve_forever() # threaded server, one thread per client # socket.create_connection — convenience for TCP clients (handles DNS, # IPv4/IPv6 fallback, and timeout in one call) with socket.create_connection(("example.com", 80), timeout=5) as s: s.sendall(b"GET / HTTP/1.0\r\n\r\n") ``` The progression in Python networking is a ladder of abstraction: raw `socket` for full control, `selectors` for efficient multiplexing of many connections in one thread, `socketserver` for a threaded or forking server framework, and `asyncio` for high-concurrency asynchronous I/O with `async`/`await` syntax. Above all of these sit application libraries — `http.client`, `urllib`, `requests`, and full web frameworks — that almost everyone uses in practice. Understanding the socket layer matters not because you will write raw sockets often, but because it demystifies everything built on top: when a request hangs, a connection resets, or a timeout fires, the behaviour makes sense once you know the bind/listen/accept/recv lifecycle underneath. ✅ Expert tab complete - I know blocking sockets halt the thread until they complete - I can use selectors to watch many sockets in one thread - I know socketserver provides a threaded server framework - I know asyncio is built on these same socket primitives Continue to [http.server →](/coding/languages/python/standard-library/http-server/) ## SOURCES - Python Standard Library — socket. docs.python.org/3/library/socket.html. - Python HOWTO — Socket Programming HOWTO. docs.python.org/3/howto/sockets.html. - Python Standard Library — selectors. docs.python.org/3/library/selectors.html. - Python Standard Library — socketserver. docs.python.org/3/library/socketserver.html. **Primary source:** docs.python.org socket *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # http.server — Built-in HTTP Server ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/http-server/ **Type:** stdlib-module | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-17 *http.server provides a ready-made HTTP server for quick file sharing, local testing, and learning how the web works — no framework required.* ## CANONICAL DEFINITION The http.server module defines classes for implementing HTTP servers. It includes SimpleHTTPRequestHandler for serving files from a directory, and BaseHTTPRequestHandler for building custom request handlers. It is intended for development, testing, and learning — not production deployment. ## BEGINNER > The idea With a single command, `http.server` turns any folder into a website you can open in a browser. It is the fastest way to share files on a local network, test a static site, or see HTTP working firsthand — all built into Python, nothing to install. ## The instant file server The most common use needs no code at all — just a command. From any directory, run the module and it serves those files over HTTP. ```python # Serve the current directory on port 8000 $ python -m http.server 8000 # Then open http://localhost:8000 in a browser. # You'll see a clickable listing of the folder's files. # Serve on a specific address (e.g. to share on your local network) $ python -m http.server 8000 --bind 0.0.0.0 # Serve a different directory without changing into it $ python -m http.server 8000 --directory /path/to/files # Press Ctrl-C to stop the server. ``` > --bind 0.0.0.0 exposes it to your network By default the server binds to localhost (only your machine). `--bind 0.0.0.0` makes it reachable by other devices on your network — handy for sending a file to your phone, but be aware anyone on the network can access those files while it runs. ## The same thing, in code ```python from http.server import HTTPServer, SimpleHTTPRequestHandler # Serve the current directory programmatically server = HTTPServer(("localhost", 8000), SimpleHTTPRequestHandler) print("Serving on http://localhost:8000") try: server.serve_forever() # run until interrupted except KeyboardInterrupt: server.shutdown() print("\nServer stopped") # Serve a specific directory (Python 3.7+) from functools import partial handler = partial(SimpleHTTPRequestHandler, directory="/path/to/public") server = HTTPServer(("localhost", 8000), handler) # server.serve_forever() ``` ✅ Beginner tab complete - I can serve a directory with python -m http.server - I know --bind 0.0.0.0 exposes it to my local network - I can run the same server in code with HTTPServer - I know it is for development and learning, not production Continue to [concurrent.futures →](/coding/languages/python/standard-library/concurrent-futures/) ## INTERMEDIATE ## Building a custom request handler To do more than serve files — return JSON, handle different URLs, accept form data — subclass `BaseHTTPRequestHandler` and define methods named after HTTP verbs: `do_GET`, `do_POST`, and so on. ```python from http.server import HTTPServer, BaseHTTPRequestHandler import json class MyHandler(BaseHTTPRequestHandler): def do_GET(self): # self.path is the requested URL path, e.g. "/api/status" if self.path == "/": self.send_response(200) # status code self.send_header("Content-Type", "text/html") self.end_headers() # blank line ends headers self.wfile.write(b"Hello from Python") elif self.path == "/api/status": body = json.dumps({"status": "ok", "version": "1.0"}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) else: self.send_error(404, "Not Found") # built-in error response def do_POST(self): # Read the request body using the Content-Length header length = int(self.headers.get("Content-Length", 0)) body = self.rfile.read(length) data = json.loads(body) if body else {} # ... process data ... reply = json.dumps({"received": data}).encode() self.send_response(201) self.send_header("Content-Type", "application/json") self.end_headers() self.wfile.write(reply) server = HTTPServer(("localhost", 8000), MyHandler) # server.serve_forever() ``` > The response order is fixed Always in this sequence: `send_response(code)`, then `send_header(...)` for each header, then `end_headers()`, then write the body to `self.wfile`. Skipping `end_headers()` or writing the body too early produces a broken response. ## Why it is not for production ```python # http.server is single-threaded by default — one request at a time. # A slow request blocks all others. You can add threading: from http.server import HTTPServer, SimpleHTTPRequestHandler from socketserver import ThreadingMixIn class ThreadingHTTPServer(ThreadingMixIn, HTTPServer): pass # Python 3.7+ actually provides http.server.ThreadingHTTPServer directly: from http.server import ThreadingHTTPServer # handles requests in threads # But even threaded, http.server lacks what production needs: # - No HTTPS/TLS out of the box # - No protection against malformed/malicious requests at scale # - No process management, load balancing, or graceful restarts # - Limited performance under real load # For production, use a real WSGI/ASGI server with a framework: # - Flask or Django + Gunicorn/uWSGI (WSGI) # - FastAPI + Uvicorn (ASGI, async) # http.server is for development, testing, and learning ONLY. ``` **Commonly confused:** - http.server vs a web framework. http.server is a minimal, batteries-included server for local use. Flask, Django, and FastAPI are application frameworks with routing, templating, security, and an ecosystem. Use http.server to serve files or learn; use a framework to build an actual web application. ✅ Intermediate tab complete - I can subclass BaseHTTPRequestHandler with do_GET and do_POST - I follow the order: send_response, send_header, end_headers, write body - I can read a request body using the Content-Length header - I know to write the body to self.wfile as bytes Continue to [concurrent.futures →](/coding/languages/python/standard-library/concurrent-futures/) ## EXPERT ## The handler lifecycle, BaseHTTPRequestHandler internals, and WSGI Each incoming connection causes the server to instantiate the handler class and call its `handle_one_request` method, which parses the request line and headers, then dispatches to the `do_VERB` method matching the HTTP method. The handler exposes the parsed request through attributes: `self.command` (the method), `self.path` (the raw URL path including query string), `self.headers` (an email.message.Message of the headers), and the `self.rfile`/`self.wfile` streams for body I/O. ```python from http.server import BaseHTTPRequestHandler, HTTPServer from urllib.parse import urlparse, parse_qs class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" # enable keep-alive (needs Content-Length) def do_GET(self): # Parse path and query string properly parsed = urlparse(self.path) path = parsed.path # "/search" params = parse_qs(parsed.query) # {"q": ["python"]} # Inspect request headers user_agent = self.headers.get("User-Agent", "unknown") body = f"path={path} params={params}".encode() self.send_response(200) self.send_header("Content-Type", "text/plain") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) def log_message(self, format, *args): # Override to customise or silence the default request logging pass # silent server # For real applications, the standard interface is WSGI (PEP 3333), # which decouples the app from the server. The stdlib includes a # reference WSGI server in wsgiref: from wsgiref.simple_server import make_server def app(environ, start_response): # environ holds the request; start_response sets status + headers start_response("200 OK", [("Content-Type", "text/plain")]) return [b"Hello from a WSGI app"] # with make_server("", 8000, app) as httpd: # httpd.serve_forever() # This same 'app' runs unchanged under Gunicorn, uWSGI, etc. in production. ``` The relationship to production servers runs through WSGI (PEP 3333), the standard interface between Python web applications and servers. The stdlib's `wsgiref.simple_server` is a reference implementation useful for testing, and the key insight is that a WSGI `app` callable written against it runs unchanged under production servers like Gunicorn or uWSGI — only the server is swapped. For asynchronous applications, the analogous standard is ASGI, served by Uvicorn or Hypercorn. The progression mirrors the socket module's: `http.server` teaches the request/response mechanics directly, `wsgiref` introduces the application/server boundary, and production deployment swaps in a robust server while keeping the application code stable. ✅ Expert tab complete - I know the server dispatches to do_VERB based on the HTTP method - I can parse path and query string with urllib.parse - I know WSGI (PEP 3333) decouples the app from the server - I know a wsgiref app runs unchanged under Gunicorn/uWSGI in production Continue to [concurrent.futures →](/coding/languages/python/standard-library/concurrent-futures/) ## SOURCES - Python Standard Library — http.server. docs.python.org/3/library/http.server.html. - Python Standard Library — BaseHTTPRequestHandler. docs.python.org/3/library/http.server.html. - Python Standard Library — wsgiref. docs.python.org/3/library/wsgiref.html. - PEP 3333 — Python Web Server Gateway Interface v1.0.1. peps.python.org/pep-3333/. **Primary source:** docs.python.org http.server *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/ **Type:** language-reference | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06-01 *General-purpose, readable, interpreted — with batteries included and the most active data science ecosystem on Earth.* ## CANONICAL DEFINITION Python is a general-purpose, high-level, interpreted, dynamically typed, multi-paradigm programming language designed for readability, with significant whitespace as block delimiters, a comprehensive standard library, and an extensive ecosystem centred on data science, machine learning, and automation. ## BEGINNER > One sentence Python is the language designed to be readable above all else — code that looks almost like plain English, with a huge standard library and the most active ecosystem in data science and machine learning. ## What Python is Python is a general-purpose, high-level, interpreted programming language created by Guido van Rossum and first released in 1991. Its central design principle — *readability counts* — is enshrined in The Zen of Python (PEP 20). Python uses significant whitespace (indentation) to define code blocks instead of braces, which forces readable structure. It is dynamically typed, multi-paradigm (procedural, OOP, functional), and ships with a comprehensive standard library described as "batteries included." ```python # Python requires no boilerplate to write useful code name = input("Your name: ") print(f"Hello, {name}! Welcome to Python.") # The Zen of Python (excerpt) — import this # Beautiful is better than ugly. # Simple is better than complex. # Readability counts. ``` ## Core syntax at a glance ```python # Variables — no declaration keyword x = 42 name = "Priya" pi = 3.14159 is_active = True # Collections fruits = ["mango", "apple", "banana"] # list — mutable, ordered coords = (10.5, 20.3) # tuple — immutable config = {"host": "localhost", "port": 5432} # dict — key-value unique = {1, 2, 3, 4} # set — unordered, unique # Control flow — indentation defines blocks if x > 10: print("large") elif x > 0: print("small positive") else: print("zero or negative") # Loop for fruit in fruits: print(fruit) # Function def greet(name, greeting="Hello"): return f"{greeting}, {name}!" # Class class Counter: def __init__(self, start=0): self.count = start def increment(self): self.count += 1 return self # List comprehension — functional style squares = [x**2 for x in range(10) if x % 2 == 0] # [0, 4, 16, 36, 64] ``` ## Dynamic typing and type hints Python is dynamically typed: variable types are determined at runtime, not declared at compile time. Python 3.5 introduced type hints (PEP 484) — optional annotations that describe intended types. They are not enforced at runtime by Python itself; external tools like mypy, pyright, and Pyrefly check them statically. Type hints are now standard practice in professional Python code. ```python from typing import Optional def process(items: list[str]) -> Optional[str]: """Return the first item, or None if empty.""" return items[0] if items else None # Type hints don't change runtime behaviour — just documentation + static analysis result: str = process(["a", "b"]) # type checker knows this is str | None ``` ## The Python ecosystem Python's package manager is **pip**; its package index is **PyPI** (pypi.org) with over 500,000 packages. The most widely used packages: **NumPy** (arrays and numerical computing), **pandas** (dataframes and data manipulation), **Matplotlib** (plotting), **scikit-learn** (machine learning), **TensorFlow** and **PyTorch** (deep learning), **Django** and **FastAPI** (web frameworks), **requests** (HTTP), **SQLAlchemy** (ORM). ## What Python is used for Data science and machine learning (dominant language), web backends (Django, FastAPI, Flask), automation and scripting, scientific computing, computer vision (OpenCV), NLP, DevOps tooling, API development, teaching and education. Python is the #1 language on TIOBE and GitHub by most metrics in 2026. ## Python versions Python 2 reached end-of-life on 1 January 2020. Always use Python 3. Current stable: Python 3.13 (October 2024). Significant additions by version: 3.5 — type hints (PEP 484); 3.6 — f-strings; 3.7 — guaranteed dict insertion order, dataclasses; 3.8 — walrus operator (:=); 3.10 — structural pattern matching (match/case); 3.11 — 10–60% speed improvement; 3.12 — improved error messages; 3.13 — experimental free-threaded mode (GIL optional). ## INTERMEDIATE ## The data model and dunder methods Everything in Python is an object — including integers, functions, and classes. Python's behaviour for built-in operations is defined by **dunder methods** (double-underscore): `__add__` for +, `__len__` for len(), `__iter__`/`__next__` for iteration, `__enter__`/`__exit__` for context managers (with statement). Implementing these protocols lets user-defined classes integrate seamlessly with built-in syntax. ```python class Vector: def __init__(self, x, y): self.x, self.y = x, y def __add__(self, other): return Vector(self.x+other.x, self.y+other.y) def __repr__(self): return f"Vector({self.x}, {self.y})" def __len__(self): return 2 def __eq__(self, other): return self.x==other.x and self.y==other.y v1 = Vector(1, 2) v2 = Vector(3, 4) print(v1 + v2) # Vector(4, 6) — uses __add__ print(len(v1)) # 2 — uses __len__ print(v1 == v1) # True — uses __eq__ ``` ## Generators and lazy evaluation A **generator** is a function that uses `yield` instead of `return`. Calling a generator function returns a generator object — an iterator that produces values lazily, one at a time. Generators are memory-efficient for large sequences: a generator for all integers uses O(1) memory, where a list would use O(n). ```python def fibonacci(): a, b = 0, 1 while True: # infinite sequence yield a a, b = b, a + b # never stores the sequence gen = fibonacci() for _ in range(10): print(next(gen), end=" ") # 0 1 1 2 3 5 8 13 21 34 # Generator expressions — like list comprehensions but lazy big_squares = (x**2 for x in range(10_000_000)) # O(1) memory first = next(big_squares) # 0 — computed on demand ``` ## Decorators A decorator is a function that takes a function as input and returns a modified function. The `@decorator` syntax is sugar for `func = decorator(func)`. Decorators enable cross-cutting concerns — caching, timing, authentication, validation — without modifying the underlying function. ```python import functools, time def timer(func): @functools.wraps(func) # preserve function metadata def wrapper(*args, **kwargs): start = time.perf_counter() result = func(*args, **kwargs) elapsed = time.perf_counter() - start print(f"{func.__name__}: {elapsed:.4f}s") return result return wrapper @timer def slow_sum(n): return sum(range(n)) slow_sum(10_000_000) # slow_sum: 0.1234s ``` ## Context managers and the with statement A context manager implements `__enter__` and `__exit__`. The `with` statement calls `__enter__` on entry and `__exit__` on exit — even if an exception occurs. This guarantees cleanup. File handles, database connections, locks, and network sockets are all context managers in Python's standard library. ```python # File — automatically closed on exit with open("data.csv", "r") as f: content = f.read() # f is closed here, even if an exception occurred # contextlib.contextmanager: decorator-based context manager from contextlib import contextmanager @contextmanager def timer(label): import time start = time.perf_counter() yield print(f"{label}: {time.perf_counter()-start:.4f}s") with timer("my block"): result = sum(range(10_000_000)) ``` ## Standard library highlights | Module | Purpose | Key functions | | --- | --- | --- | | `os` | OS interface | os.path, os.environ, os.listdir | | `sys` | Interpreter | sys.argv, sys.path, sys.exit | | `json` | JSON encode/decode | json.dumps, json.loads | | `re` | Regular expressions | re.match, re.search, re.findall | | `datetime` | Dates and times | datetime.now(), timedelta | | `pathlib` | File paths (OOP) | Path(), Path.read_text() | | `collections` | Specialised containers | deque, Counter, defaultdict | | `itertools` | Iterator tools | chain, product, combinations | | `functools` | Functional tools | lru_cache, partial, reduce | | `threading` | Threads | Thread, Lock, Event | | `asyncio` | Async I/O | async/await, gather, run | | `unittest` | Testing | TestCase, assertEqual, mock | ## Virtual environments and packaging **venv**: `python -m venv .venv` creates an isolated Python environment with its own packages, preventing version conflicts between projects. `source .venv/bin/activate` (Unix) or `.venv\Scripts\activate` (Windows) activates it. Always use a venv per project. **pip**: `pip install requests` installs from PyPI. `pip freeze > requirements.txt` records dependencies. **uv** (2024): a Rust-based Python package manager 10–100× faster than pip, rapidly becoming the standard. **Poetry** and **PDM**: dependency management + build tools. **Commonly confused:** - Python 2 and Python 3 are not compatible. print is a statement in Python 2 (print "hello") and a function in Python 3 (print("hello")). Division behaves differently: 7/2 = 3 in Python 2 (integer division), 3.5 in Python 3 (float division). Python 2 reached end-of-life January 2020. Use Python 3 only. - Python is not slow — it depends on what you measure. Pure Python loops are ~100× slower than C. But most Python performance-critical code calls into C extensions (NumPy, pandas, scikit-learn). A NumPy matrix multiplication runs at C/FORTRAN speed. Python's "slowness" is mostly a concern in pure compute loops, not I/O-bound or library-calling code. How this connects Requires first [Variables](/coding/concepts/variables/)[Functions](/coding/concepts/functions/) Enables [JavaScript (comparison)](/coding/languages/javascript/)[All data structures](/coding/data-structures/) Primary source [docs.python.org/3/ — official](https://docs.python.org/3/) ## EXPERT ## CPython internals: the reference implementation CPython (github.com/python/cpython) is the reference implementation of Python. Python is formally specified by the CPython implementation and the Python Language Reference (docs.python.org/3/reference/). The compilation pipeline: source code → tokeniser → parser → AST → compile to bytecode (`.pyc` in __pycache__) → CPython virtual machine (a stack-based VM) executes bytecode. The bytecode format is documented in the `dis` module: `import dis; dis.dis(function)` shows the bytecode instructions. ```python import dis def add(x, y): return x + y dis.dis(add) # Bytecode output (Python 3.13): # LOAD_FAST 0 (x) # LOAD_FAST 1 (y) # BINARY_OP 0 (+) # RETURN_VALUE ``` ## Memory management: reference counting + cyclic GC CPython uses **reference counting** as its primary memory management strategy. Every Python object has a reference count. When the count reaches 0, the object is freed immediately. Reference counting is deterministic and low-latency — unlike tracing GC, it doesn't require stop-the-world pauses. However, reference counting cannot handle reference cycles (object A points to B, B points to A). Python's **cyclic garbage collector** (`gc` module) periodically detects and collects cycles — it runs by default in three generations (gen 0, 1, 2) with collection frequencies configurable via `gc.set_threshold()`. ## PEP process and language governance Python evolves through the **PEP (Python Enhancement Proposal)** process. Any Python developer can write a PEP; significant ones are reviewed by the Steering Council (a five-person elected body established in PEP 13 following Guido van Rossum's resignation as BDFL in July 2018). Key PEPs: PEP 8 (style guide), PEP 20 (Zen of Python), PEP 484 (type hints), PEP 572 (walrus operator), PEP 634 (structural pattern matching), PEP 703 (no-GIL / free-threaded CPython, accepted 2023, experimental in 3.13). ## Performance: CPython vs. PyPy vs. Cython **PyPy**: a JIT-compiled Python implementation. Typically 4–10× faster than CPython on pure Python code. Excellent for compute-heavy Python without C extensions. **Cython**: a superset of Python that compiles to C. Adding type annotations to a Cython file can yield 50–100× speedups for numerical code. Used internally by pandas, scikit-learn, and scipy. **Numba**: a JIT compiler for NumPy-heavy code; `@numba.jit` can achieve near-C performance for numerical loops without rewriting in C. > Specification reference Python Language Reference — docs.python.org/3/reference/ (authoritative). Python Language Reference §3 — Data model (object model, dunder methods, reference counting). PEP 20 — The Zen of Python. PEP 703 — Making the Global Interpreter Lock Optional in CPython (accepted 2023). Van Rossum, G. & Drake, F. L. Jr. (eds.) (2009). *The Python Language Reference*. Python Software Foundation. ## SOURCES - Python Software Foundation. The Python Language Reference. docs.python.org/3/reference/. — Authoritative language specification. - Python Software Foundation. The Python Standard Library. docs.python.org/3/library/. — Standard library documentation. - PEP 8 — Style Guide for Python Code. peps.python.org/pep-0008/. - PEP 20 — The Zen of Python. peps.python.org/pep-0020/. - PEP 484 — Type Hints. peps.python.org/pep-0484/. - PEP 703 — Making the Global Interpreter Lock Optional. peps.python.org/pep-0703/. **Primary source:** docs.python.org/3/reference/ *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India* ================================================================================ # Python Standard Library ## Python — The Codex Coding **URL:** https://thecodex.expert/coding/languages/python/standard-library/ **Type:** reference | **Language:** Python | **Confidence:** high | **Last verified:** 2026-06 *The "batteries included" standard library — everything you need to work with files, time, data, text, networks, and more, without installing anything.* *The Codex Coding — thecodex.expert/coding/ — Free forever — Official sources only — Mumbai, India*