Documentation Index
Fetch the complete documentation index at: https://www.aidonow.com/llms.txt
Use this file to discover all available pages before exploring further.
Executive Summary
Standard engineering documentation describes intended system state — how a system is designed to behave. This description decays from the moment it is written, as code changes while documentation does not. A failure ledger, by contrast, records observed events: the symptoms, root causes, and resolutions of actual incidents. Observed events do not go stale in the same manner as state descriptions, because the failure pattern remains valid regardless of subsequent changes to the surrounding system. Analysis of a ten-entry incident ledger maintained over six months demonstrates that append-only failure records provide greater operational utility than architecture documentation for both human operators and autonomous AI agents performing debugging tasks. The discipline required to maintain this format — write once, never edit, never omit — is the primary determinant of long-term value.Key Findings
- Architecture documentation describes intended state and degrades continuously from the moment of authorship, because no event forces synchronization when the underlying system changes.
- Failure records describe observed events, which are permanently true, making them structurally resistant to the staleness that afflicts state-based documentation.
- The append-only constraint is not a limitation but a design property: it eliminates maintenance overhead, preserves original wrong assumptions, and prevents retrospective rationalization of incident timelines.
- Recurring patterns in a failure ledger are diagnostic signals about systemic failure modes that are invisible in architecture documentation.
- Autonomous AI agents require the same failure memory that human operators require, but the delivery mechanism differs: agents need failure records embedded in task context, not available as a reference to search.
- Silent failure modes — incidents with no error, no log, and no obvious signal — represent a distinct failure class warranting separate architectural attention.
1. Introduction
Every engineering organization produces documentation. Architecture diagrams, service overviews, deployment guides, and onboarding wikis represent substantial authorship investment. The operational value of this investment degrades over time at a rate proportional to the pace of system change. The failure modes of documentation-as-intended-state are well understood: a service is refactored, a port is changed, a configuration option is deprecated. The documentation is not updated because no event compels the update. Engineers learn to treat documentation as approximately correct and compensate by reading the code directly. The documentation persists as an artifact that orients new team members and misleads experienced ones. This paper examines a structurally different approach to operational documentation: the failure ledger. Rather than describing how a system is intended to work, the failure ledger records how the system actually broke. This paper presents the format, structural properties, and operational outcomes of a ten-entry failure ledger maintained across a self-hosted Kubernetes infrastructure over six months.2. Structural Analysis: Intended State vs. Observed Events
2.1 The Decay Problem in State-Based Documentation
Documentation that describes system state answers the question: “How does this system work?” The answer is true at the moment of writing. Its accuracy decreases as the system evolves and documentation does not. The mechanism of decay is asymmetric: code changes are forced by functional requirements and are therefore inevitable; documentation updates are not forced by any external event and are therefore optional. Over time, the gap between described state and actual state widens.This decay is not a failure of discipline or process. It is structural. Any documentation format that depends on synchronization with a changing system will degrade unless the synchronization cost is zero. No realistic documentation process achieves zero synchronization cost.
2.2 Why Failure Records Are Structurally Different
A failure ledger records observed events rather than describing current state. The entry: “Harbor credentials not mounted at/root/.docker/config.json caused a six-second TCP timeout that presented as a network error” is permanently true. That specific incident occurred. The pattern it represents — missing registry credentials produce misleading timeout errors rather than clear authentication failures — remains valid regardless of Harbor version upgrades, Kubernetes migrations, or namespace reorganizations.
The key structural distinction is that events are facts about the past. They do not require synchronization with a changing system because they do not describe the current system; they describe what happened. This property makes failure records immune to the primary failure mode of state-based documentation.
2.3 Comparative Analysis
The following table presents the key structural differences between architecture documentation and failure ledger entries:| Property | Architecture Documentation | Failure Ledger Entry |
|---|---|---|
| Describes | Intended system state | Observed incident event |
| Accuracy over time | Degrades as system changes | Permanently accurate |
| Maintenance requirement | Continuous synchronization needed | None — append only |
| Value at 2am during incident | Moderate (describes design intent) | High (describes actual failure pattern) |
| Useful to new team members | High (provides system overview) | Medium (requires context to interpret) |
| Useful to returning investigator | Low (may not reflect current state) | High (pattern recognition) |
| Authorship investment | High (significant initial effort) | Low (5–10 minutes per entry) |
| Risk of misleading reader | High (stale state descriptions) | Low (events do not change) |
3. The Ledger Format
Each entry in a failure ledger follows a four-field structure. The fields are deliberately minimal. Elaboration reduces value; brevity forces the author to identify what matters. Pattern Name — A short, searchable label for the failure class. The naming convention matters: the name should describe the failure pattern, not the date or the affected service. “DinD CI Runner — Harbor Credentials Required” is correct. “Incident on March 3rd” is not. The pattern name is written to be recognizable when the same class of failure recurs in a different context. What happened — One or two sentences describing the observable symptom, not the cause. The symptom is what the next investigator will observe first. The entry meets them at the point of observation. Why — The actual root cause. This field should include specifically what assumption was wrong, not merely what configuration was incorrect. What fixed it — Specific and complete. The resolution must be actionable from the text alone, without reference to external context the reader may not have.4. The Append-Only Constraint
The append-only constraint — no editing of completed entries — is the most important property of the failure ledger and the most counterintuitive.4.1 The Case Against Editing
The instinct to update a completed entry is a rationalization. The edited version is more polished and less accurate. The raw observation written forty-five minutes after an incident contains information the polished version loses: the exact wrong assumptions that led to the incident in the first place. Those wrong assumptions are operationally significant. When the next investigator encounters a similar failure, they will start from the same wrong assumptions the original author had. An unedited entry meets them at that point of error. An edited entry that removes the wrong assumptions — because they are embarrassing or because “the answer is obvious now” — fails the next investigator.4.2 The Maintenance Cost of Editable Documents
Editability implies maintenance obligation. If entries can be updated, they should be updated when the system changes. This creates ongoing maintenance overhead that, in practice, is not performed consistently. The result is a document that is neither reliably current nor reliably historical — it is partially updated, which is the most misleading state. An append-only ledger has zero maintenance overhead. New incidents produce new entries. Old entries remain as written. The file grows; nothing rots.5. Analysis of Ten Documented Patterns
The following ledger covers ten incidents from a self-hosted Kubernetes infrastructure. Each entry is presented with the pattern name, root cause analysis, and operational lesson. Authentik Forward Auth — 3-Component Requirement. The SSO proxy requires exactly three correctly configured components: an auth-proxy ConfigMap, an outpost ingress, and the application ingress annotations. The absence of any single component results in an unprotected endpoint with no authentication error, no redirect, and no log indication of the missing component. This silent failure mode was discovered by deploying a new service and observing that it was accessible without authentication credentials. DinD CI Runner — Harbor Credentials Required. A missing Docker credentials mount produces a six-second TCP connection timeout that presents as a network error. The actual cause — unauthenticated registry access — is not indicated in the error message. Every first-time investigator proceeds to the network layer before the credentials layer. yamllint Config — Non-Obvious Rule Names. The yamllint configuration syntax uses rule names that do not correspond to intuitive English equivalents.document-start: disable is not present: false. The truthy rule does not allow yes and no by default, which causes every Ansible playbook to fail yamllint validation. The error message is clear in retrospect; before knowing the rule name, it presents as a parsing failure.
Gitea API Token Format — PBKDF2, Not SHA256. Gitea stores API tokens as PBKDF2-SHA256 hashes, not plain SHA256. This distinction is relevant only in automation scenarios that compare tokens against stored hashes — specifically, cluster initialization scripts that pre-generate tokens. The Gitea API documentation describes the token value; it does not describe the stored hash format.
ExpressVPN + kubectl + Go TCP. Go’s network stack uses a TCP connection approach that ExpressVPN’s network driver intercepts. This affects all Go binaries making outbound TCP connections, including kubectl and helm. The symptom — kubectl hangs with no output — is indistinguishable from a cluster connectivity failure until the VPN dependency is recognized.
klipper-lb DNAT Intercepts Ports 22 and 443. Every k3s node installs iptables DNAT rules for LoadBalancer services. These rules apply to all traffic arriving at the node, including traffic from pods on the same node. Port 22 belongs to klipper-lb if any LoadBalancer service is configured on port 22. This caused SSH connections to a node to route to an nginx ingress pod.
ArgoCD repoURL Must Use Internal Cluster DNS. Pointing ArgoCD at Gitea’s external ingress URL introduces TLS certificate complexity and unnecessary DNAT overhead. The internal cluster DNS service name (gitea.gitea.svc.cluster.local) provides direct pod-to-pod communication without TLS termination.
Authentik OAuth2 — Scope Mappings Not Default. The openid, email, and profile scopes must be explicitly added to OAuth2 providers in Authentik. They are not included by default. The resulting failure is a cryptic authentication error with no indication that missing scopes are the cause.
Service-to-Service Internal URL Routing. Services in the same cluster should communicate via service.namespace.svc.cluster.local. Using the external ingress hostname for internal traffic routes through TLS termination and DNAT unnecessarily. This pattern was documented after being relearned twice.
ArgoCD Repo Sync Failure — DNS + Credentials. dnsmasq wildcard entries catch all subdomains of the configured domain, including .svc.cluster.local subdomains. If the wildcard points to the wrong IP, internal service lookups resolve incorrectly. Explicit address= entries for cluster-internal hostnames are required when wildcard DNS entries are present.
6. Pattern Analysis and Systemic Signals
6.1 Silent Failure Modes
Three of the ten entries document failures that produced no error message, no log entry, and no obvious signal — only wrong behavior requiring investigation to trace to a root cause. These cases are the Authentik 3-component requirement, the klipper-lb port interception, and the ExpressVPN kubectl hang. Silent failure modes constitute a distinct failure class that warrants dedicated design attention. The question “what is the failure signal when this component is misconfigured?” should be asked explicitly when adding new services to the infrastructure.6.2 Credential-Related Failures
Two of the ten entries document credential-related failures: the Harbor credentials mount and the Gitea token hash format. In both cases, the error message directed investigation to the wrong layer — network errors in the Harbor case, token value in the Gitea case — while the actual cause was credential configuration. Credential verification is now the first debugging step for any new service integration.6.3 Recurring Patterns as Systemic Signals
When a pattern appears in the ledger twice, it indicates a gap in the system’s error feedback. When it appears three times, the pattern should be encoded as a structural check — a lint rule, a validation script, or a checklist item — rather than remaining as documentation.7. Application to Autonomous AI Agent Workflows
7.1 The Shared Memory Problem
The failure ledger solves a memory problem for human operators: relevant failure history must be accessible at the moment of investigation without requiring the investigator to remember it independently. The ledger externalizes that memory into a searchable, structured format. Autonomous AI agents exhibit the same memory constraint in a different form. An agent’s context window is bounded. Failure history that is not present in the session context is not available to the agent during a debugging task. When an agent encountered the DinD credentials timeout pattern during an infrastructure task, it spent thirty minutes attempting network-layer remediation before the session timed out. The relevant ledger entry existed; it was not in the agent’s context.7.2 Structural Difference in Delivery Mechanism
For human operators, the failure ledger functions as a reference: the operator searches it when something looks familiar. For autonomous agents, the ledger must be part of the session context for debugging tasks — specifically, the pattern names and symptom descriptions, so the agent can recognize a known pattern before investing time rediscovering it. This distinction has implementation implications. Human access to the ledger requires findability: good naming, searchability, and a stable file path. Agent access requires inclusion: the relevant entries must be injected into the agent’s task context at session initialization, not merely available for search. The broader principle — that failure memory is operational memory, and that both human and autonomous operators require it structured, findable, and honest — applies equally at both levels of the operational stack.The relationship between the failure ledger and automated constraint enforcement is direct. Every manual correction to agent behavior that recurs more than twice should be encoded as an automated constraint — a hook, a lint rule, or a validation step. The ledger is where corrections are recorded before they become constraints. See the companion analysis on automated standards enforcement for the downstream implementation of this principle.
8. Implementation Protocol
The barrier to beginning a failure ledger is lower than the operational value it provides would suggest. No specialized tooling, templates, or process changes are required.Create the file
Create
docs/runbook.md in the primary code repository. The file path should be stable and the location should be communicated to all team members and, where applicable, included in agent session initialization scripts.Write the first entry
After the next incident, write one entry before closing the ticket. Follow the four-field format: pattern name, what happened, why, what fixed it. Set a five-minute timer. Stop when the timer goes off.
Enforce the append-only constraint
Establish the append-only rule explicitly with all contributors. New learning produces new entries. Completed entries are never edited. This rule should be documented in the file itself.
9. Recommendations
- Maintain a failure ledger as a first-class operational artifact, stored in the primary code repository alongside architecture documentation. The failure ledger should be treated as authoritative for observed system behavior; architecture documentation should be treated as authoritative for intended system design.
- Enforce the append-only constraint without exception. Completed entries are never edited. New learning produces new entries. Organizations that allow entry editing will find that entries are selectively updated to remove wrong assumptions, destroying the ledger’s primary diagnostic value.
- Use pattern names, not dates or services, as entry identifiers. Names should describe the failure class in terms that enable recognition when the same class of failure recurs in a different context.
- Treat recurring patterns as systemic signals. A pattern that appears twice indicates a gap in error feedback. A pattern that appears three times should be encoded as an automated structural check.
- Include the failure ledger in autonomous agent session context for debugging tasks. For agents, failure memory must be present in the context window to be useful; it is not sufficient for it to be available as a searchable reference.
- Analyze ledger patterns at regular intervals. The distribution of failure modes across entries reveals systemic issues that are invisible in architecture documentation — specifically, the prevalence of silent failure modes, credential-related failures, and configuration complexity hotspots.
10. Conclusion
The runbook-as-failure-ledger is not a novel concept. The operational value of documented incident records is well established. What is less commonly articulated is the structural reason for that value: failure records are immune to the primary failure mode of state-based documentation because events, unlike states, do not require synchronization with a changing system. As infrastructure complexity increases — and as autonomous AI agents become participants in operational workflows alongside human engineers — the demand for accurate, accessible failure memory will grow. The append-only failure ledger, maintained with discipline and consulted before debugging, provides a foundation that serves both human operators and autonomous agents. The ten patterns documented here represent a modest beginning; the organizational value scales with every entry added. As the proportion of operational tasks delegated to autonomous agents increases, the failure ledger will transition from a human operational reference to a shared knowledge substrate used by both human and machine operators. Organizations that establish this discipline early will be better positioned to leverage autonomous agents effectively, because the agents will have access to the accumulated failure memory that experienced human operators currently carry in their heads.All content represents personal learning from personal and side projects. Infrastructure details are generalized. No proprietary information is shared. Opinions are my own and do not reflect my employer’s views.