Skip to main content

@siglo/postgres Adapter Guide

@siglo/postgres provides an enterprise ACID-compliant persistence, event auditing, distributed locking, and idempotency coordination layer for Siglo workflows using PostgreSQL. It is engineered for mission-critical transactional workflows requiring zero state loss across crashes, network partitions, and multi-node worker clusters.

1. Overview & Architecture

When workflows coordinate financial, compliance, or enterprise operations, persistence must survive node failures and process crashes. @siglo/postgres implements two primary classes:
  1. PostgresStorageAdapter: Implements the Siglo StorageAdapter interface for durable execution checkpoints with Optimistic Concurrency Control (OCC), immutable context mutation audit logging, durable asynchronous wait/continuation records, and distributed session-level advisory workflow locking.
  2. PostgresIdempotencyStore: Implements the Siglo IdempotencyStore interface for atomic, distributed single-flight execution claims, cached result retrieval, and automated lock re-claiming upon TTL expiration.

2. Installation & Requirements

Install @siglo/postgres alongside @siglo/core and the pg driver:
[!WARNING] Connection Pooling & Session Advisory Locks (PgBouncer): PostgresStorageAdapter uses PostgreSQL session-level advisory locks (pg_try_advisory_lock) to coordinate single-flight workflow execution across distributed worker nodes. These locks are tied to the physical PostgreSQL backend connection. If deploying behind connection proxies such as PgBouncer, configure pool_mode = session. Transaction-level pooling (pool_mode = transaction) causes advisory locks to be associated with random connections, breaking mutual exclusion and triggering premature lock releases.

Peer Dependency Matrix


3. Database Schema & Automated DDL

Both PostgresStorageAdapter and PostgresIdempotencyStore provide automated schema synchronization via ensureTables() and ensureTable(). These methods execute idempotent DDL statements (CREATE TABLE IF NOT EXISTS, ALTER TABLE ... ADD COLUMN IF NOT EXISTS, CREATE INDEX IF NOT EXISTS).

DDL Definitions

SQL Injection Protection

Custom table names passed to PostgresStorageAdapter or PostgresIdempotencyStore are validated against strict SQL identifier constraints:
  • Must match /^[a-zA-Z_][a-zA-Z0-9_]*$/.
  • Must not exceed 63 characters (PostgreSQL standard identifier length limit).
  • Throws an descriptive Error if invalid characters or injection attempts are detected.

4. PostgresStorageAdapter API Reference

Implements the Siglo StorageAdapter interface (name = 'postgres').

Constructor Options (PostgresStorageAdapterOptions)

Checkpointing & Optimistic Concurrency Control (OCC)

ensureTables(): Promise<void>

Initializes checkpointsTable, eventsTable, and waitRecordsTable along with indexes and columns if they do not exist.

saveCheckpoint(checkpoint: SerializedExecutionStack, expectedVersion?: number): Promise<void>

Persists or updates an execution stack.
  • OCC Enforcement: If expectedVersion is specified (or exists on checkpoint.occVersion), executes an atomic version-guarded update:
  • Conflict Handling: If rowCount === 0, it fetches the current database version and throws an OptimisticLockException(stackId, currentVersion, expectedVersion).
  • Cancellation Preservation: If a workflow was previously cancelled, updating the checkpoint preserves the 'cancelled' status atomically.
  • Version Tracking: On success, assigns the new incremented version to checkpoint.occVersion and checkpoint.revision.
  • Write Tracking: Automatically serialized under withWorkflowLock scopes to drain all pending writes before lock release.

getCheckpoint(stackId: string): Promise<SerializedExecutionStack | undefined>

Retrieves the serialized execution stack by stackId.
  • Automatically assigns occVersion and revision from the database version column.
  • Returns undefined if stackId is empty or not found.

cleanExpiredCheckpoints(): Promise<number>

Deletes checkpoints where (state->>'expiresAt')::bigint <= (EXTRACT(EPOCH FROM NOW()) * 1000)::bigint. Returns the count of deleted rows.

Context Mutation Audit Logging

onContextChange(entry): Promise<void>

Records step mutation audit events into eventsTable.
  • Parameter entry:
  • Converts timestamp (in milliseconds) using to_timestamp(timestamp / 1000.0).
  • Operates as a non-blocking stream: internal errors are caught to avoid disrupting primary workflow execution.

Distributed Workflow Locking

withWorkflowLock<T>(stackId: string, action: () => Promise<T>): Promise<T>

Executes action inside a PostgreSQL session-level advisory lock using pg_try_advisory_lock:
  • Distributed Mutex: Serializes step continuations, event resumptions, and compensations across multi-instance worker fleets.
  • Auto-Release on Process Failure: Because PostgreSQL advisory locks are bound to the database session/connection, if a worker node crashes or disconnects, PostgreSQL automatically releases the lock immediately.
  • Re-entrancy: Safely allows nested invocations for the same stackId within the same execution context.
  • Pool Awareness: Detects pooled clients ('totalCount' in pool) and calls pool.connect(), ensuring a dedicated connection is held for the lock duration and released back to the pool in a finally block.
  • Non-Pooled / Direct Client Guard: Guards direct Client instances against concurrent re-use via directClientBusy.
  • Write Queue Draining: Uses AsyncLocalStorage and internal write tracking (tail promise) to ensure all pending checkpoint and wait record writes are completely flushed to PostgreSQL before the advisory lock is unlocked.
  • Throws: DuplicateEventException(stackId) if the lock cannot be acquired or if the direct client is busy.

Durable Wait Records & Resumption

saveWaitRecord(record: any): Promise<void>

Persists or updates a durable wait record in waitRecordsTable and embeds the record into the checkpoint metadata (checkpoint.metadata.waitRecord).
  • Key properties on record: correlationKey, stackId, namespace, eventName, status, continuationStepIndex, version, data, expiresAt.

getWaitRecord(correlationKeyOrStackId: string): Promise<any | undefined>

Retrieves a wait record from waitRecordsTable matching either correlation_key or stack_id. Falls back to querying siglo_checkpoints.state->'metadata'->'waitRecord' if not found in the wait records table.

claimWaitRecord(correlationKey: string, claimantId?: string, options?: { allowExpired?: boolean; eventPayload?: any }): Promise<any>

Atomically claims a wait record for resumption via a single Compare-And-Swap (CAS) update:
  • Expiration Safety: If the record has expired and allowExpired is not set, throws ExpiredWaitException(correlationKey, expiresAt, eventName).
  • Duplicate Safety: If the record is already claimed or resumed, throws DuplicateEventException(correlationKey).
  • Returns: The claimed wait record with status: 'RESUMING', lockedBy, and parsed JSON resumePayload.

completeWaitRecord(correlationKey: string, claimantId: string): Promise<void>

Marks the claimed wait record as 'RESUMED', setting resumed_by and resumed_at = NOW(). Drains pending async writes before completing.

cancelWaitRecord(correlationKey: string): Promise<void>

Transitions a wait record status to 'CANCELLED'.

cancelWorkflow(stackId: string): Promise<void>

Atomically cancels both active wait records and the execution checkpoint for stackId:
  • Cancels all wait records: UPDATE siglo_wait_records SET status = 'CANCELLED' WHERE stack_id = $1.
  • Updates the checkpoint state: UPDATE siglo_checkpoints SET state = jsonb_set(state, '{status}', '"cancelled"'), version = version + 1 WHERE stack_id = $1.

Retention & Cleanup


5. PostgresIdempotencyStore API Reference

Implements the Siglo IdempotencyStore interface.

Constructor Options (PostgresIdempotencyOptions)

Methods

ensureTable(): Promise<void>

Creates tableName and the index on expires_at if they do not exist.

claim(key: string, ttlMs?: number): Promise<'claimed' | 'completed' | 'in-progress'>

Atomically attempts to claim the key:
  1. Initial Insert: Executes INSERT ... VALUES ($1, 'in-progress', $2, NOW()) ON CONFLICT (key) DO NOTHING. Returns 'claimed' if row inserted.
  2. Expired Lock Reclamation: If the key exists but expires_at <= NOW(), updates status back to 'in-progress', updates expires_at, clears result, and returns 'claimed'.
  3. Completed Result: If status === 'completed', returns 'completed'.
  4. Active Claim: If actively running and unexpired, returns 'in-progress'.

complete(key: string, result: unknown, ttlMs?: number): Promise<void>

Transitions key status to 'completed', serializes result into result JSONB, and sets expires_at = NOW() + ttlMs.

fail(key: string, _error?: unknown): Promise<void>

Deletes in-progress claims (WHERE key = $1 AND status = 'in-progress') so subsequent retries can proceed immediately.

getResult(key: string): Promise<unknown | undefined>

Retrieves and deserializes the stored result JSONB for completed keys. Returns undefined if key is not completed or missing.

get(key: string): Promise<unknown | undefined>

Convenience alias for getResult(key).

getRecord(key: string): Promise<any | undefined>

Inspects the full raw idempotency record:

cleanupExpired(): Promise<number>

Deletes expired idempotency records (DELETE FROM siglo_idempotency WHERE expires_at < NOW()). Returns the count of deleted rows.

6. Concurrency & Locking Architecture

Siglo workflows employ a defense-in-depth concurrency model combining PostgreSQL advisory session locks with table-level Optimistic Concurrency Control (OCC):

Advisory Locks vs. Optimistic Concurrency Control (OCC)

  • withWorkflowLock (PostgreSQL Advisory Locks): Used around step executions, timer/event resumptions, and compensations. Guarantees that only one worker executes steps for a given stackId at any time. If the worker crashes, the PostgreSQL session lock automatically drops.
  • OCC (version column): Used by saveCheckpoint. Guards against stale writes or late-arriving updates across worker partitions. If a write conflicts with an unexpected version, an OptimisticLockException is thrown.

7. Core Exception Types

All exceptions extend SigloError and can be imported from @siglo/core:

1. OptimisticLockException

Thrown by saveCheckpoint when an update fails because the version in the database differs from expectedVersion:
  • err.stackId: The workflow execution stack ID.
  • err.currentVersion: The version currently in PostgreSQL.
  • err.expectedVersion: The version expected by the writer.

2. DuplicateEventException

Thrown by withWorkflowLock when another worker currently holds the workflow lock, or by claimWaitRecord when a wait record is already claimed or resumed:
  • err.correlationKey / err.stackId: The resource key that could not be locked.

3. ExpiredWaitException

Thrown by claimWaitRecord when attempting to claim a wait record whose expires_at timestamp has passed and allowExpired is not set:
  • err.correlationKey: The wait record correlation key.
  • err.expiresAt: Expiration timestamp in milliseconds.
  • err.eventName: Name of the expected event.

8. Complete Production Example

Below is a complete, production-ready example demonstrating connection pool configuration, automatic schema initialization, workflow definition, idempotency guards, OCC error handling, and scheduled retention cleanup.