Skip to main content

Supabase Read-After-Write Consistency

This guide explains the critical read-after-write consistency issue with Supabase read replicas and how CONA solves it for financial operations.
Current architecture (CONA-1013): prismaDirect means primary-routed, not session-affine. A Supabase DIRECT_URL on the shared Supavisor host is normalized to transaction mode (6543) with Prisma’s pgbouncer=true compatibility flag. True direct database URLs and the local Supabase URL remain unchanged. Port 5432 versus 6543 selects a pooling mode; primary versus replica routing is determined by the configured database endpoint and credentials.

🚨 The Problem

What Was Happening

Shopify invoices were created successfully, but 20-30% had no accounting impacts. The workflow reported success, but financial data was incomplete.

Root Cause Discovery

The issue wasn’t a workflow failure - it was silent data corruption:
  1. ✅ Document created in PRIMARY database
  2. ✅ Line items created in PRIMARY database
  3. ❌ Child workflow queries REPLICA (via Supavisor pooler)
  4. ✅ Document found in replica
  5. Line items missing (not yet synced from primary)
  6. ✅ Accounting impact creation “succeeds” with empty line items
  7. No GL entries created (or incomplete)
This is worse than a failure because:
  • No retries triggered (workflow thinks it succeeded)
  • No error logs (silent corruption)
  • Discovered days later by users

Technical Details

Why it happens:
  • documents table row syncs quickly (50-500ms)
  • line_items table (foreign key) lags behind
  • JOIN queries return incomplete data

✅ The Solution

1. Direct Primary Connection

We created a separate Prisma client that connects directly to Supabase PRIMARY database, bypassing read replicas entirely. Files Created/Modified:

How It Works

2. READ_AFTER_WRITE_RETRY Policy

Added specialized retry policy for activities that query recently created data:
Applied to accounting impact activities in workflow configuration.

3. Fixed Sleep Placement

Moved workflow.sleep("500ms") to BEFORE re-queries in parent workflows:

🔧 Setup Required

Environment Variables

Add DIRECT_URL to your environment files:
Get both URLs from Supabase:
  1. Go to Project Settings → Database
  2. Copy the appropriate replica connection string → DATABASE_URL
  3. Copy the primary Transaction pooler string → DIRECT_URL

Build & Deploy

📊 When to Use Each Connection

🎯 Expected Results

Before Fix

  • ❌ 20-30% invoices missing accounting impacts
  • ❌ Silent failures (no error logs)
  • ❌ Discovered days later by users
  • ❌ Manual intervention required

After Fix

  • >99.9% success rate for accounting impacts
  • Complete line items always fetched
  • Zero replica lag issues
  • Immediate consistency

🎛️ Tuning Connection Pools (Advanced)

When to Adjust Connection Limits

Increase direct connection limit if:
  • ✅ You see frequent P2024 pool timeout errors
  • ✅ Financial entry creation is consistently slow (>5s)
  • ✅ Your PRIMARY database can handle more connections
  • ✅ You’re running on dedicated infrastructure (not serverless)
Keep it conservative if:
  • ⚠️ Running on serverless (multiple instances × connections = exhaustion)
  • ⚠️ Sharing database with other applications
  • ⚠️ Limited database connection capacity

Environment-Based Configuration

For different environments, you can override defaults:

Connection Pool Monitoring

Note: prisma.$metrics.json() was removed in Prisma 6.14. The project is on Prisma 6.19+, so the metrics preview feature is no longer available. Use OpenTelemetry for per-instance client pool gauges once the OTel integration is in place. For database-wide connection monitoring, see the periodic pg_stat_activity snapshots in @cona/database/pool-metrics.
The startPoolMetrics() function (called in instrumentation.ts) periodically logs database-wide connection counts from pg_stat_activity. These are useful for overall capacity monitoring but represent all sessions to the database, not a single app instance’s Prisma pool. Each log entry includes host and pid fields so dashboards can correlate entries with specific instances.

Watch for Pool Exhaustion

Error Code P2024 indicates pool timeout:
Solutions:
  1. Increase connection_limit (if database can handle it)
  2. Increase pool_timeout (if operations legitimately take longer)
  3. Optimize slow queries
  4. Add external pooler (PgBouncer) for serverless

Optimal Settings by Deployment

🔍 Monitoring

Connection Pool Health

Find Documents Missing GL Entries

Check Replication Lag

Monitor Connection Types

🧪 Testing

Verify Direct Connection Works

⚠️ Important Notes

Connection Pooling Optimizations

Question: Should we optimize Prisma’s connection pooling?
Answer: Yes! While pooling wasn’t the root cause, we can optimize it for better performance.

Current Configuration

According to Prisma’s connection pool documentation, the default pool size is:
For a 4-core machine: 4 * 2 + 1 = 9 connections

Optimizations Implemented

1. Direct Connection Pool (Conservative)
Why conservative for direct connections:
  • Bypasses Supavisor pooler
  • Connects directly to PRIMARY database
  • Too many direct connections can overwhelm primary
  • Used only for critical operations (accounting impacts, reconciliation)
2. Pooled Connection (via Supavisor)
Why more relaxed:
  • Routes through Supavisor connection pooler
  • Load-balanced across read replicas
  • Supavisor handles connection management
  • Used for high-volume queries

Connection Pool Strategy

Performance Impact

Direct connections:
  • Slightly higher latency (~5-20ms more)
  • Limited connections (use sparingly)
  • Trade: Acceptable for critical operations
Pooled connections:
  • Lower latency (nearby replica)
  • Unlimited scaling
  • Trade: May have stale data

When Replica Lag Exceeds 2 Seconds

  1. Check Supabase Status: https://status.supabase.com
  2. Contact Supabase Support: “FRA region replica lag > 2s”
  3. Request synchronous replication for your organization

🚀 Additional Prisma Performance Optimizations

1. Select Field Optimization

Problem: Including all fields wastes bandwidth and memory.
Impact: 50-70% reduction in data transfer for large result sets.

2. Pagination Instead of Fetching All

Why cursor-based is better:
  • Offset pagination gets slower with larger offsets
  • Cursor pagination maintains constant performance
  • Essential for workflows processing thousands of documents

3. Batch Operations with createMany and updateMany

Impact: 10-100x faster for bulk operations (depending on batch size).

4. Transaction Batching

Benefits:
  • Atomic operations (all or nothing)
  • Single round trip for simple transactions
  • Consistent data state

5. Avoid N+1 Queries with Proper Includes

But be careful with useDirect:

6. Use Indexes Effectively

Check if your queries are using indexes:
In Prisma Schema:

7. Query Logging and Analysis

Enable query logging to identify slow queries:

8. Prisma Accelerate (Optional - Paid Service)

Prisma Accelerate provides:
  • Global connection pooling (solves serverless connection issues)
  • Query caching at edge (Redis-backed)
  • Automatic query optimization
When to consider:
  • Running on serverless (Fly.io) with connection exhaustion
  • Global user base (edge caching beneficial)
  • Repetitive expensive queries
Cost: Starts at $29/mo (may not be worth it yet for CONA)

9. Selective Field Loading for Large JSON/Text

10. Parallel Queries for Independent Operations

But be cautious with direct connections:

🎯 Performance Optimization Checklist

Apply these in order of impact:
  • Connection pooling (✅ Already done)
  • Use select to fetch only needed fields (5-10x faster for large records)
  • Batch operations with createMany/updateMany (10-100x faster)
  • Add database indexes for common query patterns
  • Use transactions for multi-step operations
  • Cursor pagination instead of offset for large datasets
  • Avoid N+1 queries with proper include
  • Parallel independent queries with Promise.all
  • Query logging to identify bottlenecks
  • Consider Prisma Accelerate for serverless at scale

🎓 Key Learnings

  1. Replica lag affects JOINs more than simple queries
    • Parent table syncs fast
    • Child tables (foreign keys) lag behind
    • Queries with include return incomplete data
  2. Silent failures are worse than loud failures
    • Workflow “succeeds” with incomplete data
    • No retries triggered
    • Discovered too late
  3. Direct connections solve read-after-write
    • Bypass pooler → No replica routing
    • 100% data consistency
    • Acceptable trade-off for critical operations
  4. Connection pooling is not the root cause
    • Both Prisma and Supabase pooling work fine
    • Issue was replica routing, not pooling
  5. Performance is multi-faceted
    • Connection management (pooling)
    • Query optimization (select, batch, indexes)
    • Application architecture (parallel queries, caching)
    • Right tool for the job (direct vs pooled connections)

🚀 Deployment Checklist

  • Add DIRECT_URL to staging environment
  • Build and deploy @cona/database package
  • Build and deploy @cona/core package
  • Build and deploy @cona/temporal-workflows package
  • Deploy temporal workers
  • Monitor for 24-48 hours using SQL queries
  • Verify no documents with missing GL entries
  • Add DIRECT_URL to production environment
  • Deploy to production
  • Monitor production for 48 hours

Status: ✅ IMPLEMENTED
Impact: Critical - Fixes 100% of missing accounting impacts
Last Updated: 2025-12-02