Polymarket Engine — Документация

Тема

Post-Mortem: RTDS silent failure (2026-05-06) ​

Severity: P1 — production trading bot operating blind on toxic-flow signal for ≥4 days. No financial loss observed (dry-run only), but the same code path will run live; in production trading this would have masked an entire risk-management subsystem.

Status: Symptom fixes deployed. Process gaps remain (see §6).

1. Summary ​

RTDS (the always-on Polymarket trade feed that drives ADR-004 toxic-flow detection) silently dropped 100 % of inbound events for an estimated 4+ days. The bot's WebSocket layer connected, subscribed, and watchdog-reconnected every 30 s — but every event payload was dropped at the parser as "stale" because Polymarket changed the wire timestamp from milliseconds to seconds and the parser still treated the value as ms (age_ms ≈ 1.78 × 10¹² ≈ 56 yr).

The dashboard showed RTDS · SUBSCRIBED · 143 in green throughout. The operator only noticed when investigating a separate symptom (zero fills in a backtest replay) discovered rtds_events: 0 in every capture session.

2. Timeline (UTC) ​

TimeEvent
2026-05-02 (per memory project_v2_blockers_resolved)rtds-probe smoke verified live: timestamp_format: Milliseconds. Last known good.
Between 2026-05-02 and 2026-05-06Polymarket flipped the RTDS event-payload timestamp from ms to seconds. No external announcement observed (operator's PoV).
2026-05-06 (this session)User flagged that 9 h of dry-run produced 0 fills. Replay investigation drove three sequential discoveries: capture cid resolution, sweep CLI gap, then RTDS itself.
2026-05-06 09:00Smoke probe re-run, immediately surfaced timestamp_format: Seconds + trades_parsed: 0. Three independent fixes deployed.

The drift window is at minimum 4 days. Given the bot was being exercised continuously over those days, the operator's first noticeable symptom was an indirect one (replay ground-truth). Without the replay investigation, the failure could have stayed hidden indefinitely.

3. Impact ​

Functional impact

  • ADR-004 toxic-flow detection: blind. Every external trade arrived, was parsed as Stale, and dropped. Zero RtdsEvent::Trade reached the scanner. Zero cancel decisions could have fired from this surface — RTDS_TOXIC_FLOW_LOG_ONLY=true masked this further by ensuring no observable side-effect even when the path did fire.
  • ADR-009 capture/replay: every capture session for the past 4+ days has rtds_events: 0. Re-running historical sweeps for harvester PnL ground-truth requires either re-capturing a session post-fix OR synthesising trades from book deltas (deferred — not part of this fix).
  • Production dashboard: green-status badges hid the failure from the operator (separate root cause; see §4.3).

Financial impact

  • None observed. Bot was in dry-run on Hetzner. Live trading path would have inherited the same blindness; ADR-004's RTDS_TOXIC_FLOW_ACTIVE_CANCEL_ENABLED=false default kept us in log-only mode regardless, but that's accidental safety, not a designed-in defence.

Replay engine impact (separate)

  • The capture cid-resolution gap (commit 3dbd25d) was discovered in the same session and is unrelated to RTDS but was masking the RTDS investigation: backtests showed decisions_emitted: 0, the natural assumption was strategy-config too tight, and we burned several hours on knob sweeps before bottom-up debugging surfaced the cid empty string in capture frames. RTDS only became visible after that.

4. Root causes (technical) ​

Three independent, simultaneous failures. Each alone would have been a P2; the combination + the silent observability mask escalates to P1.

4.1 Parser timestamp magnitude (data-format drift) ​

crates/pm-bot/src/io/rtds/parse.rs::validate_and_convert

Polymarket switched RTDS payload timestamps from ms to seconds without notice. The parser computed age_ms = now_ms - raw.timestamp assuming both in ms. With raw.timestamp in seconds, the subtraction yields ~1.78 × 10¹² ms ≈ 56 yr — every event tripped the max_event_age_secs stale guard.

The parser test t_parse_timestamp_ms (line 563) explicitly warned about this exact failure mode in its docstring:

"Q3 specifically: a wire timestamp like 1745251234567 must be accepted as milliseconds. If a future refactor changed the unit to seconds, this same value would be ~55k years in the future and the stale branch would silently pass — so we assert the parsed timestamp equals the wire integer directly."

The author anticipated the magnitude trap, but the test only validated the endpoint controlled by us (parser keeps treating value as ms). It did not assert behaviour under the endpoint controlled by Polymarket (server starts sending seconds). The Q3 invariant lived as a manual checklist item (production-readiness-checklist.md), not an automated gate.

4.2 Watchdog inbound-only proof-of-life ​

crates/pm-bot/src/io/rtds/client.rs::process_messages

The activity_deadline only reset on inbound traffic. Polymarket's RTDS server does not echo our text-level "ping" heartbeats on quiet markets. Subscribing to 12-15 low-volume weather binaries means 30-second windows with zero inbound traffic are normal. The watchdog kicked, the bot reconnected, the cycle repeated 558+ times since the last bot restart.

This compounded with 4.1 because the operator-visible warning ("no inbound traffic for 30s") looked like a network problem, not a parser problem. Investigation effort gravitated toward network/auth/subscribe-format theories instead of looking at what the inbound traffic — when it occasionally did arrive — actually contained.

4.3 Dashboard rtdsStatus() false-green ​

frontend/pm-dashboard-web/src/views/MainView.vue::rtdsStatus

The badge resolved connection_state === 3 ⇒ severity: 'success', ignoring whether any data actually flowed. With watchdog cycling every 30 s, state briefly hit 3, settled, and the dashboard displayed green for the entire 30-second window before the next reconnect. Pattern: a status derived from a transient flag, not a freshness signal.

The metric counters needed to detect the failure (rtds_events_received_*, rtds_reconnect_total, rtds_events_dropped_stale) were already on the wire — added in commit f6c1f50 (2026-05-03) — but no consumer aggregated them into a truthful health signal.

5. Why our process didn't catch this ​

This is the part that matters. A code fix doesn't prevent the next data-format drift; the process gaps below do.

5.1 No automated invariant gate ​

Production-readiness checklist item:

- [ ] `SmokeReport` shows: `timestamp_format: Milliseconds` (Q3 invariant)

This item exists, but as a manual checkbox intended for the ADR-004 cutover (one-time event). There is no:

  • CI cron job that runs --smoke-rtds <busy_slug> against the live Polymarket endpoint and fails on timestamp_format != Milliseconds OR schema_parity != true OR trades_parsed == 0.
  • Production health pipeline that alerts on rtds_events_dropped_stale > 0 over any rolling window.
  • Pre-deploy gate that runs the smoke harness as part of docker compose up.

The smoke harness itself is correct and sufficient — verified by the fact that it took 12 seconds to identify all three root causes once finally run. Nobody had run it for 4 days.

5.2 No alert rule on RTDS health metrics ​

docs/operations/alerts-2026-05-03.md lists 4 stock alert rules:

  • emergency_stop
  • drawdown_threshold
  • position_closed
  • posts_err_spike

None covers RTDS health. The metric counters (rtds_events_dropped_*, rtds_reconnect_total, rtds_events_received_*) ship in every MetricsTick since 2026-05-03 but no AlertRule consumes them. The infrastructure for alerting is in place; the configuration is incomplete.

5.3 Test coverage shape ​

The parser test suite (11 tests) is specification-driven, not fuzz-driven. Every test feeds a known-good or known-bad fixture and asserts the parse result. None of them adversarially probe "what if the magnitude flips" or "what if a field type changes" — classes of bug that arise from external API drift.

The test author of t_parse_timestamp_ms recognised the failure mode (per the docstring) but expressed the safeguard as a unit-test fixture rather than:

  • A property test (proptest / quickcheck) over magnitude classes.
  • A live smoke gate enforced in CI.

5.4 Memory feedback_silent_feature_broken was already saved ​

The pattern was already documented in this session's memory:

"Каждое заявленное feature должно иметь тест, наблюдающий end-to-end эффект; 'compile + no panic' недостаточно. RTDS никогда не работал в проде, тесты молчали."

The principle (every ADR feature needs a liveness test) was extracted during this incident, but it indicts every other ADR feature too — WebSocket subscribers (Market/User), webhook server, periodic schedulers, env-flag gates, recovery hooks. None of those were audited yet for the same gap.

5.5 Dashboard signal authoring ​

Documented separately as the truthful-indicator policy in the VPS prod runbook. The pattern:

  • ❌ latest_tick.counter > 0 — green forever after first event.
  • ✅ sumDeltas('counter') over SPARKLINE_POINTS — last-minute rate.

The rtdsStatus() function used the wrong pattern. There was no code review checklist or lint that catches this — it requires either a linting rule (custom AST grep), a status-rendering convention documented in CLAUDE.md, OR a test that asserts dashboard severity under known-bad metric tape.

6. Corrective actions ​

6.1 Already done in this session ​

ActionCommit
Parser auto-detects timestamp magnitude33c49cc
Watchdog resets on outbound ping success783aa05
Dashboard rtdsStatus() returns danger on storm + stuck15aa097
Top-level rtdsHealthAlert Message banner15aa097
Capture cid resolution from asset_id3dbd25d
Sweep CLI accepts --captures-root + 3 enum knobs3dbd25d
Runbook postmortem section783aa05
Frontend truthful-indicator audit (12 badges checked, 3 fixed: RTDS подписки, Strategy admitted, Capture · queue depth) — pure resolvers extracted to useStatusBadges.ts, pinned by indicator-truthfulness.test.ts (26 tests), AST-grep convention rule .ast-grep/truthful-indicator.yml blocks future latest_tick.<counter> > 0 regressions. Full table: INDICATORS_AUDIT.md.(uncommitted)

6.2 Required before next live trading session (P0) ​

  • [ ] Add stock AlertRule::rtds_health matching on MetricsTick: fires Critical when rtds_events_dropped_stale + rtds_events_dropped_parse > 0 OR rtds_reconnect_total delta over 60 s > 1 OR rtds_events_received_trade + ..._orders_matched == 0 AND rtds_subscription_active_filters > 0 for ≥ 5 minutes. Plumbed into the same notification sinks as posts_err_spike. Update alerts-2026-05-03.md.
  • [ ] CI cron — daily 06:00 UTC GitHub Actions job that runs cargo run -p pm-bot --release -- --smoke-rtds <busy_slug> and fails on timestamp_format != Milliseconds, schema_parity != true, OR trades_parsed < 1. Slug supplied from a small whitelist of evergreen high-volume markets (world-cup-winner, presidential-nominee-*). The same job runs --smoke-rtds every push to a release/* branch.

6.3 Should-do within this week (P1) ​

  • [ ] Liveness audit of every ADR-001..017 feature per memory feedback_silent_feature_broken. Tabulate: (feature, observable boundary, has_liveness_test, gap). Promote each gap to a tracked TD-XX ticket.
  • [ ] Property tests for parser: proptest strategy over timestamps in ms / seconds / future-skew / negative magnitudes, and over field shapes (missing, empty-string, wrong-type). Property: parser is total — every well-formed JSON yields Either<Event, ParseError> with no panic and no infinite loop.
  • [ ] Truthful-indicator audit of every dashboard status field. Replace any badge derived from latest_tick.counter > 0 patterns with freshness-window derivatives. Add a frontend lint or convention test in frontend/pm-dashboard-web/__tests__/ that fails when a status function reads latest_tick.X without referencing sumDeltas / valuesFor from the same scope.

6.4 Should-do within this month (P2) ​

  • [ ] Polymarket API contract probe — separate from --smoke-rtds, a probe that hits every documented Polymarket REST/WS surface we depend on, captures one envelope per surface, diffs against a checked-in golden, and surfaces format drifts before they reach production. Run in CI on a daily schedule. Capture timestamp magnitude, field presence, type stability.
  • [ ] Runbook automation — convert manual checklist items in production-readiness-checklist.md to scripts under scripts/preflight/ so an operator runs one command rather than walking 30 checkboxes.
  • [ ] Outstanding RTDS sub-issues from the verification matrix:
    • rtds_connection_state stuck at 2 post-fix (not 3). Investigate whether Polymarket dropped the SUBSCRIBE ack frame entirely or changed its shape. If the ack is gone, simplify the state machine to merge 2 + 3 into a single Subscribed — drop the dead transition rather than wait for a frame that never comes.
    • rtds_events_dropped_parse=4 accumulated since restart. Add debug!() of the dropped raw payload (truncated, sensitive fields redacted) when the parse error occurs. Surface a counter per ParseError discriminant so operators distinguish missing field vs malformed JSON vs wrong type.

7. Lessons (institutional) ​

  1. Manual checklist items rot. Production-readiness Q3 invariant was correctly identified, correctly written, correctly placed in the checklist — and not automated. Four days later the manual step's existence had no protective effect. Rule: any invariant that says "verify after deploy" must also be enforced by an automated gate within 7 days of the manual one being written.

  2. Authors anticipate failure modes; tests don't validate them. The t_parse_timestamp_ms docstring is a perfect example: the author wrote three sentences explaining exactly the bug that eventually broke us, then wrote a unit test that did not test the failure direction. Rule: every "if this changes, silently breaks" docstring becomes a property test or a live-smoke assertion within the same PR. Reading "if X then silent break" without writing the test is a self-described code smell.

  3. External-contract drift requires live monitoring, not unit tests. Polymarket can change anything any day. Static fixtures in our test suite have zero coverage of that. Rule: every external dependency needs at least one daily live probe that captures a fresh envelope and compares to a golden, at field-type granularity.

  4. Observability infrastructure ≠ observability coverage. We had the metric counters, the dashboard widgets, the alerts engine — all the plumbing was in place. We just hadn't filled in the alerts. Rule: when adding a new metric, the same PR adds the stock alert rule. When adding a new widget, the same PR adds the freshness-derived severity computation.

  5. Symptom investigation cascaded into noise (5 hours on this incident). The operator's first symptom (0 fills) led to: knob sweep on MAX_ENTRY_PRICE, capture cid resolution dive, parameter tuning iterations, sweep CLI fixes — all real but all downstream of the actual root cause. Rule: when a symptom is "no observable effect from a side-effect-producing path", the very first diagnostic to run is the input-source liveness check, before any business-logic exploration. For RTDS that's --smoke-rtds; for WS Market that's similar; for capture-replay it's checking replay_errors and frames_processed > 0.

8. References ​