Craft

Implementation decisions, design patterns, and hard-won engineering lessons from building production systems with AI-assisted development.

Articles in this section examine how specific technical choices play out in production — from data modeling and state machine design to CI pipeline architecture and container signing. Each article presents the decision context, the implementation, and the observed failure modes.

Implementation

DynamoDB Single-Table Reality Check

The single-table design pattern in practice: when it delivers on its promises, when it introduces access-pattern brittleness, and the decision criteria that determine which applies.

State Machines Prevent Bugs

How encoding valid state transitions as first-class types eliminates an entire class of logic bugs — and why this matters more when AI is generating the implementation.

Event Sourcing

The architectural commitments that event sourcing demands, the failure modes that emerge when those commitments are partially honored, and the implementation patterns that hold at scale.

Macro Boilerplate Elimination

Using Rust procedural macros to eliminate repetitive handler registration — the pattern, the limits, and when the abstraction cost exceeds the boilerplate cost.

The Macro Revolution

How compile-time code generation changes the relationship between specification and implementation, and the new category of errors it introduces.

AWS Runtime Adoption

The operational and cost model for AWS Lambda-based runtimes — including the cold start profile, the deployment constraints, and the workload characteristics that favor or disfavor adoption.

Config Governance Middleware

A middleware-based approach to configuration validation that enforces structural contracts at service startup rather than at runtime failure.

Saga Workflow Patterns

Choreography versus orchestration in distributed transaction design — the trade-offs, the observable failure modes of each, and the recovery protocol that both require.

Billing System Design

The data modeling and event sourcing patterns that make usage-based billing auditable, correctable, and resilient to double-counting under concurrent writes.

Revenue Recognition Domain: DDD for ASC 606 Contract Modifications

How encoding ASC 606 contract modification treatments as first-class Rust types — Prospective vs. CumulativeCatchUp, Decimal over f64, allocation sum validation — makes compliance constraints compiler-enforced rather than documentation-dependent.

Testing Infrastructure

The test pyramid as an enforcement artifact — integration tests that catch what unit tests miss, and the infrastructure investment that makes them fast enough to run on every commit.

Dead Drives, Dead Assumptions

What hardware failure teaches about software resilience assumptions — and the gaps in distributed system design that only become visible under real infrastructure loss.

Nine Fixes and a Delete

How nine iterative CI workflow fixes culminated in discovering the workflow was unnecessary — and what that diagnostic arc reveals about debugging strategy.

Debugging as Archaeology

How commit history functions as a forensic timeline and why reading the code-change record before touching running state reduces mean time to resolution.

Container Pipeline: Air-Gapped Signing

Kaniko, Cosign v2, Kyverno, and Vault on self-hosted Kubernetes — a production-grade signing pipeline without public Sigstore infrastructure.

AI-Assembled Lab: Bolt to Inference

How Claude Code built a complete homelab — blade chassis, Proxmox, Kubernetes, 30+ services, and self-hosted LLM inference via Ollama and OpenClaw — across 550+ commits with an operational constitution as the governing artifact.

Eliminating API Mocking from E2E Tests

When a 52-spec Playwright suite was rewritten to run against a live dev environment, the first run exposed five real bugs that mocking had been silently concealing — a data service 404, a navigation state bug, missing CMS slug registrations, and more.

The BoringSSL Toolchain Tax

How reqwest → rustls → aws-lc-rs → aws-lc-sys created a hidden cmake/nasm/golang/perl build requirement in Rust CI — and how upgrading aws-lc-sys to 0.40.0 eliminated the entire native toolchain in one step.

Scheduling Service: Domain-First Design

How building domain aggregates before HTTP handlers produced a DynamoDB table design shaped by access patterns rather than REST conventions — including a time-range conflict detection index, hand-rolled RFC 5545 ICS generation, and thirteen typed EventBridge event keys.

Self-Hosted Quality Platform

Allure TestOps and SonarQube on Kubernetes with Gitea OIDC via oauth2-proxy — including the trailing-slash issuer bug, the nginx auth subrequest loopback, Longhorn encrypted storage, and how nextest JUnit XML eliminates the need for a native Rust Allure library.

Event Catalog: Topology as a Queryable Artifact

When no developer can answer “what events does this system emit?” without reading source code, the topology has become an implicit contract. Startup-time event registration, consumer mapping, and a browsable schema viewer make the event graph a first-class queryable artifact.

Schema Registry: Breaking Circular Crate Dependencies

How a shared registry crate with zero upstream dependencies eliminates circular dependency cycles in multi-service Rust provisioning — using the inventory crate for link-time provider collection, idempotent DynamoDB table creation, and isolation-testable schema definitions.

Visual Workflow Builder: React Flow as an Agent Configuration Interface

How a React Flow (Xyflow v12) canvas editor makes workflow automation definitions accessible to non-developers — linear DAG enforcement, tenant-aware field population, and a six-spec E2E test strategy that unit tests cannot replace.

Production LLM Integration: CRM AI Suggestions in Rust

An eight-step async generation flow with SHA-256 idempotency keys, three-layer cost control (cooldown/rate-limit/feature-gate), PII-safe prompt construction, and a CISO mandate that keeps suggestion text out of the event stream.