Skip to content

Test stand history API and evidence storage

Protected Linux build and dependency installation

Deploy without replacing the server .env. Use npm run install:safe followed by npm run build on the remote Linux stand. Both run in the fixed transient st-test-stand-build.service with systemd/cgroup-v2 limits: 2 GiB memory, zero swap, 64 tasks, 200% CPU quota, ten-minute runtime. Concurrent protected jobs are rejected; unavailable or ineffective isolation fails closed. Non-root invocation requires a user systemd manager with delegated resource controllers.

install:safe installs from the lock file into a fresh temporary cache with dependency scripts disabled, checks esbuild, then executes dependency scripts inside the same bounded service and checks again. Linux esbuild must be ELF and, when available, match its package-manifest binary checksum. Temporary diagnostic caches are retained. Build logs identify each TypeScript/Vite stage and emit ten-second liveness heartbeats, which are not progress guarantees.

Plain npm ci, direct compiler commands and direct vite are not intercepted or protected. Stop the stand before changing dependencies; do not deploy during an active test. Protection does not replace API authentication or change wire schemas.

Trade mutation coverage (2026-10-03)

Basic runs thirteen independent trade mutation sections after trade queries and before trade traces. They cover BUY/SELL delete/cancel/restore, invalid states, bulk delete/cancel, all six specialized trade Sync commands, group close/cancel, historical imports and downloaded CSV/XLSX exports. No trading API wire format or server/Core behavior is changed by this test-stand addition.

Financial expectations come from a separate planned ledger, never monetary response fields: delete reverses historical net, cancel removes exposure without realizing floating net, and restore reverses the old realized net before valuing the newly open position at a different quote. Checks include signed commission and storage, exact open/history tickets, partial volume conservation, concurrent duplicates, settled replay and three delayed observation cycles. Queued accepted responses are not treated as proof of execution. Concurrent calls exercise competition but do not guarantee a specific engine interleaving.

Sync checks validate completed response ticket/login/state and requested fields. Generic update Sync rejects pending orders; MngUpdatePendingTradeSync edits all four pending types. Identical opens allocate distinct tickets; valid repeated partial closes are new operations, not idempotent metadata edits.

Import checks reserve four tickets through normal opens/closes on an owned, successfully reconciled disposable donor account. That donor alone is deleted to free its history before importing those tickets into the target test account. They never guess future ticket numbers, overwrite existing trades, or artificially advance global counters. Duplicate ticket import must not add its net twice; distinct tickets with identical contents remain separate historical entries. Mixed batches validate exact successes/failure indexes and active exposure.

Exports are downloaded and parsed, not merely acknowledged. Checks compare ordered ticket sets, selected columns, lot-denominated volumes/totals, signed gross/net/costs, quoted commas/newlines/XML text, repeated-read neutrality, invalid fields/formats, and client denial of global export while market/pending positions remain active.

The new report direction is Trade mutations, accessible through the existing assertion filter:

GET /api/runs/:id/assertions?limit=50&after=0&failedOnly=true&section=Trade%20mutations

trade-mutations-summary is a checkpoint array of {name,status} rows. Successful rows also contain login; failed rows contain error and incidentId. Failure incidents use the existing envelope and execution category, with context.section and context.logins; evidence contains section, error, and fixtures. Each fixture contains login, observed events, and available account/trades reads. Capture failures are listed in missing; a preceding financial incident is linked as parentIncidentId. Failed fixtures are retained; remaining independent sections continue before the block reports aggregate failure.

Remaining scope includes active missed-swap restoration (current restore tests use charge_missed_swaps=false), deterministic Sync timeout/rejection correlation, restricted-staff permission matrices, same-account multi-symbol group isolation, group partial failures, import restart persistence and interrupted execution. Local verification is non-compiling source inspection only. Build remotely and run node --import tsx --test tests/trade-mutation-oracle.test.ts tests/trade-export-reader.test.ts before Basic regression. Runtime outcomes remain unverified until that remote run.

Multicurrency portfolio coverage (2026-10-02)

Basic now ends with four isolated portfolios of 30 original positions each, after the existing recovery block. Each account uses a distinct synthetic two-decimal currency and three instruments with mixed BUY/SELL and varied volumes: direct conversion, inverse conversion, and identity conversion. Existing pair prefixes in either orientation cause setup to fail rather than overwrite live currency pairs. Two groups check net uncovered-volume margin; two check large-leg margin. Symbol hedge books are reconciled separately.

Independent expectations use current conversion prices for profit (direct ASK for BUY / BID for SELL; inverse 1/BID for BUY / 1/ASK for SELL) and frozen opening conversion rates for margin. Monetary rounding follows conversion. For these two-decimal fixtures, the scoped oracle rounds the scaled double using the server's half-away-from-zero rule (std::round(value * 100) / 100), including negative P/L. This applies to per-trade profit/margin, portfolio totals and realized balance, without adding epsilon or relaxing comparisons. For example, inverse BUY 60 at entry 1.3002, contract 100000, leverage 100, divider 2 and opening rate 1.25 has margin 487.575 rounded to 487.58. The incident's complete 30-position portfolio has equity 99558.39 and margin 3556.91. The shared rounding helper used by other stand scenarios is unchanged. Dependent instrument ticks are explicitly published after conversion quote changes, so conversion-only scheduling is not claimed.

Checks cover partial/full closure, independent original-volume conservation, open portfolio restart, persisted audit barriers, local backup, dry restore, applied accounts/trades restore with immediate financial validation before fresh ticks, phantom post-backup order/trace removal, restored restart and final zero exposure/balance/history/audit stability. Failure is fail-fast for global restore; scoped diagnostic evidence and fixtures are retained.

The new report section uses the existing filter:

GET /api/runs/:id/assertions?limit=50&after=0&failedOnly=true&section=Multicurrency

multicurrency-summary records completion. Failure incident context.phase identifies the failing phase. Its polymorphic evidence contains available beforeRestart, baseline, backupRunId, newOrders, mutated, dryRestore, appliedRestore, immediate financial snapshots and snapshots. Per-account snapshots include login/currency, independently specified instruments and positions, original and closed ledgers, request/response/event operations, TCP events and available account/trade reads. Capture failures use missing. No envelope, endpoint, or report-summary schema version changes are required.

This does not cover separate instrument profit/margin currencies, triangulation, missing rates, JPY/VND rounding, nonzero costs, or interrupted execution. Each instrument uses matching quote/profit and margin currencies. Synthetic currencies do not validate an ISO-currency whitelist. Deploy the stand and remotely run node --import tsx --test tests/multicurrency-oracle.test.ts before full regression.

Stop Out portfolio coverage (2026-10-02)

Basic adds Stop Out portfolios checks under the existing Stop Out report category, after trading delivery and before backup/recovery. No public API or response-schema changes are introduced.

Six mixed BUY/SELL sections use varied volumes and explicit large-leg margin for worst-loss/FIFO/LIFO policies. Four single-side sections check one point above the SO boundary, exact 50%, and one point below. Full/partial manual-close races use a fixed adverse quote and concurrent SO-threshold activation, for both directions, repeated according to executionRaceRepeats. They do not guarantee a particular thread interleaving: unique parent/child volumes and independently calculated P&L determine the legal final exposure and finances.

Race checks assert settled open-plus-closed volume conservation and each open parent's volume against the independent closed ledger, then verify account finances before waiting for matching WS balances. A corrupted portfolio therefore fails with an accounting/volume assertion instead of only a WS equality timeout.

The corresponding server/Core fix uses synchronous Stop Out admission under the same cached-order lock as local client/manager closes. Pending partial requests are skipped until settlement, and stale partial results cannot reopen finalized parents. Server and stcore must be rebuilt and deployed together: Core API and stcore ABI versions are now 10, rejecting mixed old/new deployments. In ICoreApi::enqueueCalculatedTrades(vector<TradeCalcRecord>&&, bool notify=false), a singleton initial SO request (state=TS_CLOSE_REQUEST, requested_state=TS_CLOSE_REQUEST, activation=ACTIVATION_STOPOUT, update_state=1) is a synchronous reservation handshake, not a queued result. Only RET_OK authorizes Core execution. A missing order returns RET_NOT_FOUND, a competing request/state or ownership mismatch returns RET_TRADE_INCORRECT_STATE, and stale volume returns RET_TRADE_BAD_VOLUME. Ordinary calculated result batches remain asynchronous; admission does not echo a trade delta into Core. Public trading and incident wire fields are unchanged.

SL/TP protection closing now uses the same synchronous singleton handshake: state=TS_CLOSE_REQUEST, activation=ACTIVATION_SL or ACTIVATION_TP, requested_state equal to the source TS_OPEN_NORMAL, TS_OPEN_RESTORED, or TS_OPEN_UPDATE_REQUEST state, and update_state=1. The cached source state, login, command, volume, entry price, SL and TP must match before RET_OK permits execution. A mismatch returns RET_TRADE_INCORRECT_STATE; a missing order returns RET_NOT_FOUND. Successful admission stores CLOSE_REQUEST atomically and marks the result as a close rather than modify completion. A rejected old calculation keeps floating exposure and schedules another pass without a new quote.

Open-position modify admission is synchronous. Quote-only valuation cannot release its pending request; actual modify completion is explicitly tagged. The ABI 10 SDK adds bool ICoreApi::getTradeSnapshot(int order, TradeRecord& out). order is the ticket to look up; out receives a complete copy of the existing TradeRecord when the method returns true. It returns false for an absent ticket and leaves out unchanged. The adapter uses the TradeManager cache under a shared lock, not SQL or an account-wide history scan. This read is not itself an admission reservation; the atomic protection handshake remains authoritative.

Core checks modify deltas against the current in-memory manager snapshot before insertion, so closed positions cannot be resurrected by delayed deltas. The execution-protection section covers BUY/SELL and SL/TP with staggered race schedules and now asserts no OPEN event after CLOSED, alongside independent realized balance, flat equity and zero-margin checks. Incident order 1982 had correct balance 9920 but phantom floating profit 32 and margin 440.08; the flat account requires equity 10920 and margin zero. Non-compiling source/model checks are not substitutes for the remote C++ tests and Basic runtime regression.

Each fixture contains massTradePositions simultaneous positions (default 50). At ten race repetitions this adds 50 sections and 2,500 market positions, extending run duration. Checks include TCP/WS terminal and financial delivery, exact SO plans, system traces, delayed history/financial stability beyond recovery hold, and one server restart with surviving exposure and immutable persisted traces. The new block retains complete scoped evidence in linked incidents and records stopout-portfolios-summary. Failed fixtures are preserved, not forcibly closed.

Use the existing filter to retrieve these checks:

GET /api/runs/:id/assertions?limit=50&after=0&failedOnly=true&section=Stop%20Out

This is deterministic USD Forex/large-leg margin coverage, not all hedge policies, non-Forex instruments, or a throughput benchmark. Source verification is local; compilation and full scenarios must run remotely.

Basic trading delivery coverage (2026-10-02)

Restrictions observe rejected requests with symbol trade=0 using three cancellation-aware observations separated by 250 ms, checking unchanged trade history and independently expected account finances. No fresh tick is required while disabled: the server's market-state gate rejects incoming ticks in this mode. Switching back to close-only or full trading requires a fresh deterministic quote before further mutations. The shared quote timestamp check is unchanged; these bounded observations do not prove absence of arbitrarily delayed effects.

Basic now includes Pending bursts and WS isolation report sections after its existing HTTP/WS lifecycle and before backup/recovery. Section names are also accepted by the existing assertion-page section filter. The response envelopes, incident schema and report-summary schema version remain unchanged.

Pending bursts cover all four types at a precise boundary and through a gap: eight isolated portfolios, each using massTradePositions (default 50, range 30–100). Tests retain pre-trigger pending orders, activate original tickets from one crossed quote, reconcile execution/finances, close and audit history/replay. WS isolation uses two independent trading users with one account each and 50 positions per account by default. It checks own trade and balance delivery, foreign-switch denial, same-connection user reauthorization and explicit reconnect. Positive cross-account switching for SESSION_CUSTOMER is CRM coverage, not basic. Password/token payloads are never included in these incidents. Bounded observation windows do not prove absence of arbitrary long-delayed execution or leakage.

Examples of existing section-filter requests:

GET /api/runs/:id/assertions?limit=50&after=0&failedOnly=true&section=Pending%20bursts
GET /api/runs/:id/assertions?limit=50&after=0&failedOnly=true&section=WS%20isolation

Both blocks run independently and aggregate their explicitly linked incidents in trading-delivery-summary.json. A failed delivery batch prevents subsequent global backup/restore stages. No trading API or server/stcore behavior is changed.

Starting, cancelling, and deleting runs now require a standalone username/password cookie session and an exact same-origin request. See Authenticated controls and interactive reports for the complete authentication, deletion, section-filter, and report-summary-v2 contracts. Read-only monitoring and evidence remain public within the stand's trusted-network perimeter. Deleted runs retain original artifact evidence but are excluded from history and skipped during recovery.

Incident bundles (new runs)

Incidents are additional immutable journal entries (kind: "incident"), not replacements for assertions or checkpoints. No incident payload is pushed through WS or embedded in run summaries. There is no automatic backfill for old runs. The optional incidentId string on a failed assertion links it to an explicitly captured incident.

GET /api/runs/:id/incidents?after=0&limit=25 returns an indexed page, using the existing entries pagination envelope (items, hasMore, nextAfter, storage). id is the run ID; after is an exclusive incident sequence cursor (default 0, nonnegative safe integer); limit defaults to 100, range 1..200. Sequences are journal revisions and may have gaps. Each item contains sequence, revision, incidentId, nullable parentIncidentId, title (compact preview), category, capturedAt (ISO timestamp), and detailUrl (raw single-entry endpoint). The inherited failedOnly and tail options are intended for assertion/log pages, not incident navigation.

GET /api/runs/:id/entries/incident/:sequence returns the original single incident packet after evidence checksum verification. GET /api/runs/:id/incidents/:sequence returns a complete bundle with explicitly linked ancestors and related incidents for the same run. sequence is a positive safe integer, not an incident UUID. Response header: Cache-Control: no-store. The UI copies/downloads this complete JSON; it does not truncate evidence to the visible scroll area.

Bundle fields: schemaVersion (1), incident (selected packet), parents (packets reached through parent links), related (packets reached through related links), missing (bundle assembly problems). Links are traversed breadth-first, parents before related links on each packet; shared packets and cycles are visited once. Up to 128 linked IDs are resolved; unresolved queued IDs are explicitly reported when this limit is reached. Missing, unindexed or corrupt linked packets are reported individually without hiding other available evidence. Packet fields: schemaVersion (1), incidentId (UUID), parentIncidentId (UUID or null), relatedIncidentIds (UUID array, defaults to empty; absent on older packets), category (financial, execution, recovery, assertion, scenario), title (full string), capturedAt (ISO timestamp), run (id, scenarioId, scenarioName, nullable testedBuild, nullable startedAt), error (stack/message or null), context (operation-specific JSON or null), evidence (complete scoped JSON or null), missing (capture failures), scope (correlation description), and fullJournalUrl (existing full-evidence export). testedBuild uses the existing run-summary build object. Context/evidence are intentionally polymorphic diagnostic JSON.

Complete bundle example for an unscoped assertion:

{
  "schemaVersion": 1,
  "incident": {
    "schemaVersion": 1,
    "incidentId": "eea2f29b-5e08-44e7-a940-ff9d1c78d96e",
    "parentIncidentId": null,
    "relatedIncidentIds": [],
    "category": "assertion",
    "title": "partial profit",
    "capturedAt": "2026-10-01T12:00:00.000Z",
    "run": {"id": "run-1", "scenarioId": "basic-regression", "scenarioName": "Basic regression", "testedBuild": null, "startedAt": "2026-10-01T11:00:00.000Z"},
    "error": null,
    "context": null,
    "evidence": {"name": "partial profit", "passed": false, "expected": 16, "actual": -4},
    "missing": ["This assertion has no explicit operation correlation; related failures are not automatically merged."],
    "scope": "Explicit scenario evidence; not an inferred time-window correlation",
    "fullJournalUrl": "/api/runs/run-1/export"
  },
  "parents": [],
  "related": [],
  "missing": []
}

Financial evidence includes reconciliation (login, phase, startedAt, status, expected, inputs, firstMismatch, latest, lastConsistent, error) and portfolio (events, open, closed, finance when available). Execution incidents contain the existing scoped command/request/accepted/observed or phase/inputs/events diagnostics, plus account/orders and, where captured, groups/symbol. Recovery evidence includes available beforeRestart, baseline, mutated, newOrders, dryRestore, appliedRestore, observations, summary. Observations contain login, section, optional runId, and independently captured account/trades/events. Unavailable fields are identified in missing. These are observations, not a claim of the root cause. Full HTTP/WS traffic is not automatically recorded.

Execution-integrity market/protection races additionally retain their full scoped checkpoint in evidence: status, error (failed cases), repeat, schedule (first, delayMs), requests, trace, initialBalance, original, order, cursor, account, history, quote, events. Each trace item has index, optional startedAt, completedAt, result (HTTP status/data or quote result), and error. Diagnostic read failures are reported in packet missing and may appear as { "captureError": "..." } in the affected evidence field. Failed race checkpoints additionally include serverTraces, an array whose entries contain order (integer ticket) and history (an array of complete GetTracesByOrder items) for the parent and child tickets observed in scoped history. Each history is paginated without truncation; failed reads contain captureError and are listed in missing. Lifecycle correlations use the trace operation IDs, not timestamps. Section incidents reference the race via parentIncidentId. The final batch incident references every failed section via relatedIncidentIds; its evidence.summary contains section, status, optional login/error/incidentId, and elapsedMs. Opening the batch packet returns these sections and their underlying races together. Error links are assigned only after incident persistence succeeds. Existing historical incidents are not backfilled.

List failures: 404 RUN_NOT_FOUND, 400 invalid pagination/filter, 503 RUN_INDEX_UNAVAILABLE (with storage status). Bundle failures: 404 RUN_NOT_FOUND or INCIDENT_NOT_FOUND, 400 INVALID_PAGINATION, 503 INCIDENT_EVIDENCE_UNAVAILABLE. Raw entry failures follow the existing entry contract. PG indexing may lag; refresh incident pages after indexing. Missing parents remain explicit and can appear on a later fetch. Structured password/token/authorization/cookie/secret/API-key fields are redacted in diagnostic copies. The exact account flag enable_change_password is preserved when numeric or boolean; a string value remains redacted. Arbitrary free-text strings are not guaranteed secret-free: treat exports as sensitive trusted-network diagnostics. Original evidence and trading checks are unchanged. Independent failures are never merged by timestamp.

The Node.js test stand is a separate control plane, not the trading server. Deploy the updated backend and frontend together. This contract replaces full-run list/WS responses. Trading assertions and tolerances are unchanged. Keep this API on a trusted network: read-only endpoints provide access to diagnostic evidence, while modifying run-control endpoints require stand user sign-in. The stand PostgreSQL database and artifact directory must be outside trading-server restore scope.

Storage and configuration

Node >=22.12 loads <test-stand>/.env; existing process environment takes precedence. Set DATABASE_URL or the standard PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD. Keep .env mode 0600 and out of Git/artifacts. PostgreSQL stand_runs stores summaries; stand_entries indexes entry previews and file digests. Complete payloads live in <artifacts>/<run>/segments/*.ndjson for new v3 runs (storage-v3.json), or records/ for existing v2 runs. PostgreSQL is a rebuildable index, not a replacement for artifact backups.

V3 journals batch up to 1000 records or 4 MiB per immutable segment (a single larger record is allowed). Writes, file fsync, rename and directory fsync use asynchronous filesystem I/O; durable summaries become visible only after the batch is committed. Payload serialization remains synchronous to snapshot mutable inputs. Synchronous assertion/log callbacks enqueue records rather than acknowledge durability. TCP requests, HTTP requests, quote publication, completed checkpoints, lifecycle persistence and final reports await a durability barrier. A 10 ms timer drains idle queues. An abrupt crash can lose an uncommitted tail, not an acknowledged durable batch. Queue admission stops explicitly after 32 MiB plus one record; previously queued evidence is retained and drained. Oversized bursts fail the run rather than dropping records silently. Temporary segment files are retained but ignored as uncommitted; corrupt committed segments stop recovery. V2 journals are not rewritten. New checkpoint aliases are hard links to immutable, UUID-qualified versioned files, atomically replaced without copying the full payload. Do not modify these files/aliases in place. PG indexing commits up to 1000 records / approximately 8 MiB per transaction with their summary revision. While behind, another asynchronous pass follows immediately; idle/error retries use the periodic timer. Failed transactions are replayed. Two nullable columns, evidence_file and evidence_offset, are added to stand_entries; existing rows remain valid. New locators reference byte ranges in segments. Complete detail payloads remain verified against the indexed SHA-256. PostgreSQL remains a rebuildable index. If PG is unavailable, evidence continues to be stored locally and index-dependent requests return 503. Local evidence write failure blocks subsequent scenario operations/new runs until storage is repaired and the process restarted. Run a single runner process per artifact directory. No automatic deletion or retention policy is enabled.

Legacy import retains run.json, summary.json, and checkpoints. It imports every assertion and log, compares counts, and records the original run.json SHA-256 in storage-v2.json. Incomplete imports resume from a durable import-progress.json watermark before the completion marker is committed. Batches contain up to 100 records or approximately 4 MiB (a single larger record is allowed). Every file is fsynced, then the record directory is fsynced before the watermark advances. Existing uncommitted files are verified before adoption; conflicting evidence stops recovery without overwriting it. Original legacy files are never removed. Legacy payloads, including their original log sequence fields, are retained; new pagination sequences are assigned independently. Original cross-stream ordering cannot be inferred when legacy assertions have no timestamps. HTTP/WS starts before recovery. A dedicated worker parses/imports one legacy run at a time and generates historical reports outside the HTTP event loop. Recovery failure leaves HTTP/WS and new scenarios available. Old-history recovery does not gate execution. Directories created by the current runner are excluded from recovery, including runs submitted during the initial scan. Data never persisted by the previous process cannot be recovered. New run.json and run-v2.json are bootstrap metadata, not full live exports; the record journal is authoritative.

RunSummary

All fields in this example are the wire fields. Optional fields are startedAt, finishedAt, snapshotId, snapshotName, testedBuild, progress, and error; absent optional fields are omitted, not null.

progress contains completed (non-negative integer completed stage count), total (positive integer planned stage count), and current (string current stage label). completed is never greater than total. New basic regression runs report 18 stages; other scenarios and older runs may omit this field. Progress is persisted in the journal summary and included in existing run HTTP responses and WebSocket run snapshots. The UI displays completed/total stages below the run header. Stages are equally counted, not equally timed: the percentage is not a time estimate, assertion target, or engine coverage. After all stages finish, report finalization may still be running; only the terminal run status determines success. Failed or cancelled runs retain their last stage count. Active legacy runs without stage data have an indeterminate indicator; passed legacy runs show 100% completion.

{
  "id": "2026-10-01T10-00-00-000Z-basic-regression-example",
  "scenarioId": "basic-regression",
  "scenarioName": "Basic regression",
  "status": "failed",
  "requestedAt": "2026-10-01T10:00:00.000Z",
  "startedAt": "2026-10-01T10:00:01.000Z",
  "finishedAt": "2026-10-01T10:05:00.000Z",
  "snapshotId": "clean",
  "snapshotName": "Clean baseline",
  "testedBuild": { "version": "1.0.0", "buildDate": "2026-10-01" },
  "progress": { "completed": 4, "total": 18, "current": "Trading rules" },
  "error": "One or more assertions failed",
  "revision": 12,
  "logCount": 6,
  "assertionCount": 2,
  "passed": 1,
  "failed": 1,
  "levels": { "info": 4, "success": 1, "warning": 0, "error": 1 },
  "sections": { "Trading": { "passed": 1, "failed": 1 } },
  "reportReady": true
}

status: queued/preparing/running/passed/failed/cancelled. Times are ISO strings. revision increases on each durable record; it is not a log sequence. levels counts all logs; sections counts assertions by report direction. reportReady means the compact HTML was generated; it does not imply the test passed or PG caught up. No logs, assertions, or filesystem directory is included in RunSummary. Names/error summaries are bounded (scenario/snapshot names 200 characters; error 2000), with an explicit truncation suffix. Full evidence is retained separately.

Storage status, attached where specified:

{"mode":"postgres-with-local-journal","pendingRuns":0,"pendingRecords":0,"pendingWrites":0,"indexReady":true,"error":null,"recovery":{"state":"ready","totalRuns":22,"loadedRuns":22,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}

pendingRuns counts local runs awaiting index catch-up. pendingRecords counts durable revisions not yet known to be indexed; startup may temporarily overestimate it until stored PG revisions are read. pendingWrites counts enqueued revisions not yet acknowledged as durable. Neither value counts failures or additional test cases. indexReady indicates schema initialization, and error is null or an infrastructure diagnostic string. indexReady=true does not guarantee PG is currently reachable.

recovery is always present in storage status, including HTTP responses and WS heartbeats:

  • state: loading, ready, or failed; describes historical recovery only and does not gate new scenarios.
  • totalRuns: number of artifact directories discovered; loadedRuns: number processed successfully (including directories without a run manifest).
  • currentDirectory: current artifact directory name, or null before scanning/after completion.
  • phase: scan, import, report, or done.
  • processedEntries, totalEntries: durable imported entries and total entries for the current/last directory; zero when no import progress is available.
  • error: recovery diagnostic string or null; independent of the index error above.

During recovery, previously indexed runs can already appear in lists while their local detail endpoints are not loaded yet. Do not interpret a temporary 404 during loading as evidence deletion. ready means local recovery completed, not that PostgreSQL has caught up; inspect pendingRuns separately.

HTTP

GET /api/runs

Query: limit integer 1..100 (default 25), optional opaque cursor from nextCursor. Order: requestedAt descending, then id descending. Exclusive keyset cursor. Never return full payloads. Response fields: items (RunSummary array), hasMore boolean, nextCursor string or null, storage as above. A complete empty response is:

{"items":[],"hasMore":false,"nextCursor":null,"storage":{"mode":"postgres-with-local-journal","pendingRuns":0,"pendingRecords":0,"pendingWrites":0,"indexReady":true,"error":null,"recovery":{"state":"ready","totalRuns":22,"loadedRuns":22,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}}

For nonempty responses every item has the RunSummary format shown above. Summaries here reflect committed PG revisions; during catch-up a new run can be absent. Invalid limit/cursor returns 400. PG unavailability returns:

{"error":"RUN_INDEX_UNAVAILABLE","storage":{"mode":"postgres-with-local-journal","pendingRuns":1,"pendingRecords":100,"pendingWrites":0,"indexReady":true,"error":"Run index unavailable; full local journals retained for replay","recovery":{"state":"ready","totalRuns":22,"loadedRuns":22,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}}

GET /api/runs/:id

Returns the latest local RunSummary, potentially ahead of PG. Unknown ID: 404 {"error":"RUN_NOT_FOUND"}. Clients must tolerate index lag and should not infer missing evidence from an empty indexed page while pendingRuns is nonzero.

POST /api/scenarios/:id/run

Optional JSON body {"snapshotId":"clean"}. Returns 202 and a RunSummary, not a full run. Requires an authenticated stand user session cookie and matching Origin; anonymous requests return 401. History recovery and PostgreSQL index lag do not block this endpoint. Local evidence write failures still block new scenarios to prevent unrecorded execution. Trading checks, queue sequencing and scenario execution are unchanged.

Unknown scenario: 404 {"error":"SCENARIO_NOT_FOUND"}. This endpoint still starts the scenario and its configured reset workflow.

GET /api/runs/:id/logs and /api/runs/:id/assertions

Query fields: after exclusive integer sequence >=0 (default 0), limit 1..200 (default 100), failedOnly boolean text true/false (default false; useful only for assertions), tail=true starts at the last indexed count minus limit. Tail is selected before the failure filter and is not a filtered-history tail. Assertion pages also accept optional section (nonempty string, maximum 200 characters), matching the exact preview section name. Supplying this filter on log or incident pages returns 400 INVALID_FILTER. Use after to continue, not timestamps or WS arrival order. Invalid pagination/filter returns 400. Unknown run returns RUN_NOT_FOUND/404; unavailable index returns RUN_INDEX_UNAVAILABLE/503 with storage status.

Log response (all preview fields; assertionUrl appears only on linked assertion logs):

{
  "items": [{"sequence":6,"timestamp":"2026-10-01T10:05:00.000Z","level":"error","phase":"assertion","message":"Trade profit mismatch","hasDetails":true,"assertionUrl":"/api/runs/example/entries/assertion/2","revision":10,"detailUrl":"/api/runs/example/entries/log/6"}],
  "hasMore":false,"nextAfter":6,
  "storage":{"mode":"postgres-with-local-journal","pendingRuns":0,"pendingRecords":0,"pendingWrites":0,"indexReady":true,"error":null,"recovery":{"state":"ready","totalRuns":22,"loadedRuns":22,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}
}

Log level: info/success/warning/error. phase is bounded to 100 characters and message to 1000 plus truncation suffix. hasDetails indicates the original log has a details value. The full log remains accessible even when hasDetails is false.

Assertion response:

{
  "items": [{"sequence":2,"name":"Trade profit mismatch","passed":false,"section":"Trading","hasDetails":true,"revision":9,"detailUrl":"/api/runs/example/entries/assertion/2"}],
  "hasMore":false,"nextAfter":2,
  "storage":{"mode":"postgres-with-local-journal","pendingRuns":0,"pendingRecords":0,"pendingWrites":0,"indexReady":true,"error":null,"recovery":{"state":"ready","totalRuns":22,"loadedRuns":22,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}
}

Assertion name is bounded to 1000 characters plus suffix. sequence is per run and kind; revision is global within a run. For empty pages nextAfter equals the effective input cursor. hasMore describes the indexed matching data at query time, not whether an active run can produce more records later. Page queries do not include full expected/actual or details.

GET /api/runs/:id/entries/:kind/:sequence

kind is log/assertion/state/artifact. sequence is a positive safe integer. Returns the complete original payload; its file digest is verified against the index before returning it. No preview truncation applies. Missing run/kind/entry: 404 {"error":"ENTRY_NOT_FOUND"}. Index/file/checksum failure: 503 {"error":"EVIDENCE_UNAVAILABLE"}.

Assertion fields are name, passed, optional expected, actual, detail. expected/actual accept arbitrary JSON. Complete example: {"name":"Trade profit mismatch","passed":false,"expected":16,"actual":-4,"detail":"partial close"}. Log fields are sequence, timestamp, level, phase, message, optional arbitrary JSON details. Complete example: {"sequence":6,"timestamp":"2026-10-01T10:05:00.000Z","level":"error","phase":"assertion","message":"Trade profit mismatch","details":{"assertionSequence":2}}. Linked assertion logs deliberately reference their complete assertion rather than duplicate its expected/actual. State payloads are one of: {"created":true}, {"imported":true}, {"reportReady":true}, or {"status":"failed","error":"Full error including stack"} (error optional). Artifact payload: {"name":"orders","file":"orders-v11.json","bytes":2,"sha256":"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"}. It describes the complete immutable versioned checkpoint, not the latest-version alias.

GET /api/runs/:id/export

Streams application/x-ndjson, attachment filename evidence.ndjson, directly from the local journal, including every successful/failed assertion, log, state and artifact-reference record. It does not require PG. Each line has revision, kind, sequence, summary (RunSummary), preview (same indexed preview or an empty object for state/artifact), and payload (full kind-specific payload above). Unknown run: RUN_NOT_FOUND/404. Checkpoint file bytes are downloaded separately; the NDJSON is not an archive of those files. An export of a running run captures the latest acknowledged durable revision at export start; export again after completion for a complete final journal.

GET /api/runs/:id/artifacts

Query: after integer offset >=0 (default 0), limit 1..200 (default 100). Files sorted by name; directories and temporary files excluded. Response fields: items with name/bytes/url, nextAfter offset, hasMore. Complete example:

{"items":[{"name":"orders-v11.json","bytes":2,"url":"/api/runs/example/artifacts/orders-v11.json"}],"nextAfter":1,"hasMore":false}

This is a directory listing, not a stable snapshot while new artifacts are being created. Use the final listing after completion. Unknown run returns RUN_NOT_FOUND/404. GET /api/runs/:id/artifacts/:filename streams a file as an attachment, content type application/octet-stream. Nested paths, temporary files and paths escaping the run directory are rejected. Missing/invalid file: 404 {"error":"ARTIFACT_NOT_FOUND"}. Full legacy run/summary files remain downloadable.

GET /api/runs/:id/report

Returns compact text/html. Includes aggregate results, direction counts, links to the first 20 failures, latest financial summaries for up to 200 accounts and full evidence links. Limits affect presentation only. Missing run: RUN_NOT_FOUND/404; report not generated: REPORT_NOT_READY/404. Links require access to the stand.

GET /api/status

Response retains stand, environment, defaultSnapshotId, ISO now, and quoteSource, and adds storage and activeRunIds (array of strings). activeRunIds lists current-process queued/preparing/running runs from the in-memory runner, independent of PostgreSQL index lag and old-history import. Empty means no current active runs. Use this field, not historical indexed statuses, for run/stop controls. quoteSource fields: listening boolean; connections, authenticatedSessions, subscribedSessions, authRequests, subscribeRequests, timeRequests, ticksSent numbers; optional lastTick object with symbol, bid, ask, timestamp. Complete example:

{"stand":"test","environment":"test","activeRunIds":[],"defaultSnapshotId":"clean","now":"2026-10-01T10:00:00.000Z","quoteSource":{"listening":true,"connections":1,"authenticatedSessions":1,"subscribedSessions":1,"authRequests":1,"subscribeRequests":1,"timeRequests":0,"ticksSent":1,"lastTick":{"symbol":"EURUSD","bid":1.1,"ask":1.1002,"timestamp":1790848800000}},"storage":{"mode":"postgres-with-local-journal","pendingRuns":0,"pendingRecords":0,"pendingWrites":0,"indexReady":true,"error":null,"recovery":{"state":"ready","totalRuns":22,"loadedRuns":22,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}}

WebSocket /ws

Deploy backend and frontend together. The changed invalidation protocol is replaced by live data frames. There is no periodic browser REST polling, including during WS disconnection. The browser reconnects instead. Initial metadata, user-requested history/check pages, full details and downloads remain HTTP operations.

Client subscription (all fields required):

{"type":"subscribe","requestId":1,"runId":"example","after":661,"tail":false,"follow":true}
  • requestId: nonnegative safe integer chosen by the client; increment on selection/follow changes. Echoed in frames.
  • runId: string up to 250 characters, or null to subscribe only to global status/history summaries.
  • after: nonnegative safe integer, exclusive per-run log sequence last applied by the client.
  • tail: boolean; true resets to the latest indexed window (up to 200 logs), false resumes from after.
  • follow: boolean; false suppresses live log batches while still delivering status/summaries (for historical browsing).

The server sends normal updates/heartbeats approximately once per second. While a log backlog exists, acknowledged batches can follow at intervals of 100 ms (up to ten frames per second). It waits for each application-level ACK before sending another:

{"type":"ack","sequence":1}

sequence is a safe integer matching the received frame, not a log sequence. Unknown/duplicate ACKs do not advance the cursor. A late ACK for a previous subscription never changes the new subscription's cursor. Maximum incoming frame: 4096 bytes. Invalid message/JSON closes with code 1008.

Complete initial empty frame example:

{
  "type":"live",
  "sequence":1,
  "requestId":0,
  "runId":null,
  "status":{
    "stand":"test","environment":"dedicated-test-only","defaultSnapshotId":"clean",
    "activeRunIds":[],"now":"2026-10-01T10:00:00.000Z",
    "quoteSource":{"listening":true,"connections":0,"authenticatedSessions":0,"subscribedSessions":0,"authRequests":0,"subscribeRequests":0,"timeRequests":0,"ticksSent":0},
    "storage":{"mode":"postgres-with-local-journal","pendingRuns":0,"pendingRecords":0,"pendingWrites":0,"indexReady":true,"error":null,
      "recovery":{"state":"ready","totalRuns":0,"loadedRuns":0,"currentDirectory":null,"phase":"done","processedEntries":0,"totalEntries":0,"error":null}}
  },
  "runs":[],
  "nextCursor":null,
  "run":null
}

Frame fields:

  • type: always live.
  • sequence: contiguous frame counter starting at 1 for each connection. ACK only after applying the frame.
  • requestId, runId: subscription identity. Discard selected-run data from an outdated subscription, but ACK the frame.
  • status: complete /api/status response, including active run IDs, quote-source statistics and storage/recovery state.
  • Optional runs: latest 25 locally loaded RunSummary objects, ordered by requestedAt/id descending; sent initially, after subscription changes and when this snapshot changes. No logs/assertions/full payloads are embedded.
  • Optional nextCursor: opaque HTTP history cursor or null; always accompanies runs. Local summaries can be ahead of the PostgreSQL history index. Older pages remain user-requested HTTP reads; index lag may temporarily omit history.
  • Optional run: selected RunSummary or null if none is selected/loaded; sent initially and when it changes.
  • Optional logs: object with items (log previews in the HTTP log-page format), nextAfter (last delivered log sequence), hasMore (more indexed entries exist), and reset (replace the displayed window instead of merging).
  • Optional error: RUN_INDEX_UNAVAILABLE. Status/summaries remain available, and log reads retry without advancing the cursor. The field is omitted when no index-read error occurs in that frame.

Example logs field (all fields shown):

{"items":[{"sequence":662,"timestamp":"2026-10-01T10:00:00.000Z","level":"error","phase":"assertion","message":"Trade profit mismatch","hasDetails":true,"assertionUrl":"/api/runs/example/entries/assertion/2821","revision":900,"detailUrl":"/api/runs/example/entries/log/662"}],"nextAfter":662,"hasMore":false,"reset":false}

Log batches contain at most 200 previews and target at most 200 KiB of serialized previews (one preview is indivisible). Frames have a hard 512 KiB limit; excess closes with 1009. No complete evidence is discarded/truncated by transport limits. Unchanged summaries/logs are omitted; the remaining status frame is a heartbeat and must not trigger HTTP requests. One unacknowledged frame is retained per connection. Missing ACK for 30 seconds terminates the connection. Internal streaming errors close with 1011. The browser detects a missing heartbeat after 20 seconds and reconnects.

On reconnect, sequence restarts at 1. Subscribe with the last applied after and tail=false to replay missed log previews from PostgreSQL. Merge/deduplicate by per-run log sequence, not by connection frame sequence. Status and summaries are refreshed as snapshots; intermediate summary states are not replayed. Index catch-up continues even after execution finishes so the final logs are not lost. hasMore=false does not mean indexing has caught up. If after exceeds the locally known run log count, the server sends a tail reset via logs.reset=true. Initial selection and explicit Follow latest use tail=true; historical pages and the complete export remain available.

The UI retains at most 500 live log previews. Assertion/log details expand inline and load their existing detailUrl only when opened. Requests are cancelled on collapse/unmount, time out after 15 seconds, and offer Retry on error. Successful details use an LRU cache bounded by 32 entries and 4 Mi characters; larger details display completely but are not cached. The checks list loads on opening, filter/page changes or explicit refresh, not on every live revision. Live check counts still update through WS. None of these UI limits changes test execution, assertions or stored evidence.