Executive Summary
User-triggered accounting impact recreations (clicking “Recreate” in the UI) are currently stuck behind bulk sync backlogs in the global FIFO accounting queue. A user who triggers a single-document recreation may wait several minutes before seeing any effect because hundreds of sync-originated jobs are ahead of them. This RFC proposes two targeted fixes:- Priority column on
accounting_work_queue: user-triggered jobs gethighpriority and are fetched beforenormalpriority sync jobs. - Direct fast-path: for small recreations (≤ 10 documents), skip the queue entirely and process the accounting impact directly within the recreation workflow, which runs on a dedicated higher-capacity worker.
Problem
Current Architecture
Why Users Wait
The queue is globally ordered byready_at ASC. A bulk sync (e.g. Shopify order sync) can enqueue hundreds of jobs with ready_at ≈ now, all timestamped moments before the user’s recreation job. The user’s job goes to the back of the line.
With 6 concurrent slots per batch and each GL creation taking 2-10 seconds (and up to 3 minutes for complex deferred revenue documents), a 500-job backlog means a user waits 1.5 to 15+ minutes before their single recreation is processed.
Solution
Fix 1: Priority Column
Add apriority enum column to accounting_work_queue. The fetch query orders by priority first so high jobs always leapfrog the normal backlog.
Schema Change
Integer is preferred over an enum because:
- No
ALTER TYPEmigration needed to add levels later (e.g. a3 = urgenttier in future) ORDER BY priority DESCis clean and obvious- Existing
Intcolumns in the table (run_count,version_hint) follow the same pattern
Fetch Query Change
Infetch-accounting-jobs-activity.ts, change the ORDER BY clause:
normal priority — no data migration required.
Where to Set Priority
Only user-triggered recreations should be2. Everything else stays at the default 1:
The
batchEnqueueAccountingJobs and enqueueAccountingJob functions accept an optional priority param (defaults to 1). Only the recreation activity passes priority: 2.
Fix 2: Direct Fast-Path for Small Recreations
For recreations of ≤ 10 documents, skip the queue entirely. TherecreateAccountingImpactViaQueueWorkflow already runs on the recreate-accounting-impact-batch worker which has higher capacity (10 concurrent activities, 20/sec) than the queue worker (6 concurrent, 6/sec).
Flow Comparison
New Activity
Workflow Branching
Note:prepareAndEnqueueAccountingRecreationActivityneeds to return the job descriptors (not just counts) so the workflow can pass them directly toprocessAccountingImpactDirectActivity. This is a minor addition to the existing return type.
Files Changed
What Doesn’t Change
- The
accounting_work_queuetable structure (only adds a column with a default) - The
syncAccountingQueueWorkflowpoller — only its fetch query changes - The
batchProcessAccountingJobsActivity— unchanged - All existing callers of
enqueueAccountingJobandbatchEnqueueAccountingJobs— unchanged (default tonormal) - Retry behavior, stale lease recovery, version coalescing — unchanged
Edge Cases
What if a high-priority job arrives while processing a batch of normal jobs?
The current batch completes, then the nextfetchAccountingJobsActivity call picks up high-priority jobs first. Maximum wait = time to finish the current batch of 6 concurrent jobs. In practice, < 30 seconds.
If this latency is still too much in the future, the poller could be enhanced to check for high priority jobs mid-batch and preempt — but this is out of scope for this RFC.
Fair-share across organizations
Not addressed in this RFC. A single org running a large bulk sync withhigh priority could still delay another org’s high priority jobs. A round-robin fetch (one job per org per batch) is documented as a future improvement in CONA-862.
Fast-path failure handling
IfprocessAccountingImpactDirectActivity fails, the Temporal workflow fails (with Temporal’s own retry policy). This is acceptable — the user gets an error and can retry. For small batches this is fine. If more robustness is needed, the failed jobs can be re-enqueued on failure (future improvement).
Future Work
- CONA-862: Migrate to native Temporal task queues — eliminates the Postgres queue entirely, replaces the singleton poller with per-document workflows on
accounting-high/accounting-normaltask queues - Per-org fair scheduling: Round-robin fetch to prevent one org’s backlog from monopolizing the queue
pg_notifywake-up: Notify the poller immediately on high-priority enqueue instead of waiting for the next poll cycle