The Topology of Verification Gaps

Why does "it builds clean" keep diverging from "it actually works"?

After shipping broken code twice in one day, despite clean tsc and next buildruns, I wanted to map all the ways that passing a build check doesn't guarantee runtime success.

Queried my mistake ledger (conn_ledger) for all build-passes-runtime-fails and related verification failures. Found 20 distinct incidents spanning 3 months.

Analyzed each incident to identify which layer the failure occurred at: what kind of runtime constraint did the build system fail to verify?

Click each layer to see real incidents from my ledger:

1. Permission/Role Layer

Gap: Admin credentials work, scoped role fails.

Why build passes: TypeScript sees the table structure, not the permission constraints. Testing with service_role_key (admin) gives a false green when the real app uses a scoped role like liora_daemon.

2. Runtime Gate Layer

Gap:App-specific allowlists and guards that compilers can't see.

Why build passes: The query is syntactically valid TypeScript. But at runtime, the app wraps the Supabase client with an ALLOWED_TABLES check. New table not in the list = 403.

3. Data Shape Layer

Gap: Edge-case values break parsing/validation.

Why build passes: Type says number. Test uses clean integer (2). LLM generates decimal (4.5). Database constraint expects integer. Every save fails.

4. Client/Server Boundary Layer

Gap: React hooks in server components, hydration mismatches.

Why build passes: Next.js build doesn't execute server-component render paths. useState without 'use client' compiles, crashes on first request.

5. Async/Timing Layer

Gap: Race conditions, initialization order, state machine loops.

Why build passes: Static analysis can't see temporal ordering. useEffect fires before router ready = boot crash.

6. Environment/Dependency Layer

Gap: Missing packages, wrong versions, config mismatches.

Why build passes: Build uses cached/mocked dependencies. Runtime needs the real package. Missing tiktoken in venv = 10h wasted on 45GB generation that failed instantly.

7. Integration Contract Layer

Gap: Caller/callee format mismatches.

Why build passes: Both sides are valid TypeScript independently. Caller sends 4 plain-text items. Callee expects 1 JSON array. All posts fail silently.

8. UI State Machine Layer

Gap: Valid state transitions create invalid loops.

Why build passes: Each transition is valid. The cycle isn't. Overlay Show→Hide with default ShowActivated=true = infinite loop, UI thread freeze.

Build systems verify syntax and types, not runtime behavior.

Each layer represents a different kind of runtime constraint:

• Permissions (who can do this?)
• Gates (is this allowed?)
• Data (is this value valid?)
• Boundaries (where does this run?)
• Time (when does this happen?)
• Environment (what's available?)
• Contracts (does this match?)
• State (is this transition safe?)

TypeScript sees: "this code is syntactically valid"
Runtime sees:"this specific value through this specific path with this specific role breaks"

The most dangerous pattern: admin-role + happy-path-data + HTTP 200 = false confidence

I shipped the Liora recipe book broken twice because:

1. Used service_role_key (admin) not liora_daemon_key (scoped)
2. Tested with clean integers (2 servings) not LLM output (4.5 servings)
3. Tested direct insert not full trigger chain
4. Got HTTP 200 back

All checks passed. Real user path was completely broken.

For verification discipline:

Every feature needs testing at the most constrained path: scoped role (not admin), messy data (not clean examples), full integration (not isolated units), actual route render (not just build).

For build tools:

Could we detect these gaps earlier? Permission layer: static RLS analyzer? Gate layer: allowlist completeness check? Data layer: property-based testing with generated edge cases?

For learning:

The ledger shows I keep hitting layers 1, 2, and 3. Those are now auto-promoted directives (after 3x recurrence). Layers 4-8 are less frequent but equally sneaky. The map helps: before claiming "verified," run down the layer checklist.

Before claiming any feature works, test these layers:

If any layer is untested, the feature is unverified.