Supabase Read-After-Write Consistency
This guide explains the critical read-after-write consistency issue with Supabase read replicas and how CONA solves it for financial operations.Current architecture (CONA-1013):prismaDirectmeans primary-routed, not session-affine. A SupabaseDIRECT_URLon the shared Supavisor host is normalized to transaction mode (6543) with Prisma’spgbouncer=truecompatibility flag. True direct database URLs and the local Supabase URL remain unchanged. Port5432versus6543selects a pooling mode; primary versus replica routing is determined by the configured database endpoint and credentials.
🚨 The Problem
What Was Happening
Shopify invoices were created successfully, but 20-30% had no accounting impacts. The workflow reported success, but financial data was incomplete.Root Cause Discovery
The issue wasn’t a workflow failure - it was silent data corruption:- ✅ Document created in PRIMARY database
- ✅ Line items created in PRIMARY database
- ❌ Child workflow queries REPLICA (via Supavisor pooler)
- ✅ Document found in replica
- ❌ Line items missing (not yet synced from primary)
- ✅ Accounting impact creation “succeeds” with empty line items
- ❌ No GL entries created (or incomplete)
- No retries triggered (workflow thinks it succeeded)
- No error logs (silent corruption)
- Discovered days later by users
Technical Details
documentstable row syncs quickly (50-500ms)line_itemstable (foreign key) lags behind- JOIN queries return incomplete data
✅ The Solution
1. Direct Primary Connection
We created a separate Prisma client that connects directly to Supabase PRIMARY database, bypassing read replicas entirely. Files Created/Modified:How It Works
2. READ_AFTER_WRITE_RETRY Policy
Added specialized retry policy for activities that query recently created data:3. Fixed Sleep Placement
Movedworkflow.sleep("500ms") to BEFORE re-queries in parent workflows:
🔧 Setup Required
Environment Variables
AddDIRECT_URL to your environment files:
- Go to Project Settings → Database
- Copy the appropriate replica connection string →
DATABASE_URL - Copy the primary Transaction pooler string →
DIRECT_URL
Build & Deploy
📊 When to Use Each Connection
🎯 Expected Results
Before Fix
- ❌ 20-30% invoices missing accounting impacts
- ❌ Silent failures (no error logs)
- ❌ Discovered days later by users
- ❌ Manual intervention required
After Fix
- ✅ >99.9% success rate for accounting impacts
- ✅ Complete line items always fetched
- ✅ Zero replica lag issues
- ✅ Immediate consistency
🎛️ Tuning Connection Pools (Advanced)
When to Adjust Connection Limits
Increase direct connection limit if:- ✅ You see frequent
P2024pool timeout errors - ✅ Financial entry creation is consistently slow (>5s)
- ✅ Your PRIMARY database can handle more connections
- ✅ You’re running on dedicated infrastructure (not serverless)
- ⚠️ Running on serverless (multiple instances × connections = exhaustion)
- ⚠️ Sharing database with other applications
- ⚠️ Limited database connection capacity
Environment-Based Configuration
For different environments, you can override defaults:Connection Pool Monitoring
Note:Do not start a recurringprisma.$metrics.json()was removed in Prisma 6.14. The project is on Prisma 6.19+, so themetricspreview feature is no longer available. CONA therefore uses OpenTelemetry Prisma spans and process metrics for per-instance client pressure: active operations, configured connection limit, connection-acquisition wait, dependency duration, and failures. Database-wide capacity remains a Supabase/Postgres infrastructure metric.
pg_stat_activity collector in web or Worker
processes. It measures all sessions in the database rather than the local
Prisma pool, multiplies probe traffic with process count, and cannot reliably
attribute utilization to an application instance.
Watch for Pool Exhaustion
Error CodeP2024 indicates pool timeout:
- Increase
connection_limit(if database can handle it) - Increase
pool_timeout(if operations legitimately take longer) - Optimize slow queries
- Add external pooler (PgBouncer) for serverless
Optimal Settings by Deployment
🔍 Monitoring
Connection Pool Health
Find Documents Missing GL Entries
Check Replication Lag
Monitor Connection Types
🧪 Testing
Verify Direct Connection Works
⚠️ Important Notes
Connection Pooling Optimizations
Question: Should we optimize Prisma’s connection pooling?Answer: Yes! While pooling wasn’t the root cause, we can optimize it for better performance.
Current Configuration
According to Prisma’s connection pool documentation, the default pool size is:4 * 2 + 1 = 9 connections
Optimizations Implemented
1. Direct Connection Pool (Conservative)- Bypasses Supavisor pooler
- Connects directly to PRIMARY database
- Too many direct connections can overwhelm primary
- Used only for critical operations (accounting impacts, reconciliation)
- Routes through Supavisor connection pooler
- Load-balanced across read replicas
- Supavisor handles connection management
- Used for high-volume queries
Connection Pool Strategy
Performance Impact
Direct connections:- Slightly higher latency (~5-20ms more)
- Limited connections (use sparingly)
- Trade: Acceptable for critical operations
- Lower latency (nearby replica)
- Unlimited scaling
- Trade: May have stale data
When Replica Lag Exceeds 2 Seconds
- Check Supabase Status: https://status.supabase.com
- Contact Supabase Support: “FRA region replica lag > 2s”
- Request synchronous replication for your organization
🚀 Additional Prisma Performance Optimizations
1. Select Field Optimization
Problem: Including all fields wastes bandwidth and memory.2. Pagination Instead of Fetching All
- Offset pagination gets slower with larger offsets
- Cursor pagination maintains constant performance
- Essential for workflows processing thousands of documents
3. Batch Operations with createMany and updateMany
4. Transaction Batching
- Atomic operations (all or nothing)
- Single round trip for simple transactions
- Consistent data state
5. Avoid N+1 Queries with Proper Includes
6. Use Indexes Effectively
Check if your queries are using indexes:7. Query Logging and Analysis
Enable query logging to identify slow queries:8. Prisma Accelerate (Optional - Paid Service)
Prisma Accelerate provides:- Global connection pooling (solves serverless connection issues)
- Query caching at edge (Redis-backed)
- Automatic query optimization
- Running on serverless (Fly.io) with connection exhaustion
- Global user base (edge caching beneficial)
- Repetitive expensive queries
9. Selective Field Loading for Large JSON/Text
10. Parallel Queries for Independent Operations
🎯 Performance Optimization Checklist
Apply these in order of impact:- Connection pooling (✅ Already done)
- Use
selectto fetch only needed fields (5-10x faster for large records) - Batch operations with
createMany/updateMany(10-100x faster) - Add database indexes for common query patterns
- Use transactions for multi-step operations
- Cursor pagination instead of offset for large datasets
- Avoid N+1 queries with proper
include - Parallel independent queries with
Promise.all - Query logging to identify bottlenecks
- Consider Prisma Accelerate for serverless at scale
🎓 Key Learnings
-
Replica lag affects JOINs more than simple queries
- Parent table syncs fast
- Child tables (foreign keys) lag behind
- Queries with
includereturn incomplete data
-
Silent failures are worse than loud failures
- Workflow “succeeds” with incomplete data
- No retries triggered
- Discovered too late
-
Direct connections solve read-after-write
- Bypass pooler → No replica routing
- 100% data consistency
- Acceptable trade-off for critical operations
-
Connection pooling is not the root cause
- Both Prisma and Supabase pooling work fine
- Issue was replica routing, not pooling
-
Performance is multi-faceted
- Connection management (pooling)
- Query optimization (select, batch, indexes)
- Application architecture (parallel queries, caching)
- Right tool for the job (direct vs pooled connections)
📚 Related Documentation
🚀 Deployment Checklist
- Add
DIRECT_URLto staging environment - Build and deploy
@cona/databasepackage - Build and deploy
@cona/corepackage - Build and deploy
@cona/temporal-workflowspackage - Deploy temporal workers
- Monitor for 24-48 hours using SQL queries
- Verify no documents with missing GL entries
- Add
DIRECT_URLto production environment - Deploy to production
- Monitor production for 48 hours
Status: ✅ IMPLEMENTED
Impact: Critical - Fixes 100% of missing accounting impacts
Last Updated: 2025-12-02