Skip to content

Operator Runbook

The engine runs itself — it boots, trades, checks its own state every hour, fixes problems, and restarts services on its own. When an hourly check fails it reconciles the state automatically — then pauses if the fix holds, or stops for you if it doesn't. You step in for three cases: the app auto-reconciled and paused (the common one — find out why, then resume), the app needs attention (NOT_READY or ERROR — a check failed and reconcile didn't clear it), or a strategy is in error.

The rule for all of them: the engine makes the state correct, but you decide the cause is safe. Find out why it drifted before you resume — resuming something that stopped itself for a real reason is how a small problem becomes a big loss. Stopping is cheap and reversible; when unsure, leave it stopped and escalate.

The dashboard does not refresh on its own. Reload the page before you look and again after every action, so you're always acting on the live state.

Post in #alpha-team whenever you take an action — stop, disable, close, reconcile, resume — so the team knows the system was touched and why.

What the statuses mean

App

  • TRADING — trading normally
  • VERIFYING — hourly self-check (~30s)
  • NOT_READY — a state check failed and the automatic reconcile didn't clear it on the first try (at startup or on the hourly cycle). Trading is stopped, but nothing is broken. The engine re-checks next cycle: it returns to READY on its own if the drift clears, or escalates to ERROR if it fails again. Investigate the cause now — don't wait for the escalation.
  • READY — on but not trading. Normal after boot or the hourly cycle, or someone stopped it — check with the team before starting it. Also where the engine pauses itself after auto-reconciling a failed check: if you see a "state auto-reconciled — PAUSED" alert, the state is already fixed and it's waiting for you to find the cause before resuming
  • ERROR — the automatic path has given up: a check failed twice in a row, the reconcile itself errored, the same drift kept recurring, or a service crashed / an exception was thrown. Fully on you; act
  • SAFE_MODE — emergency stop pulled; act

Strategy

  • ACTIVE — trading
  • INACTIVE — stopped, may auto-resume
  • CLOSE_ONLY — winding down
  • DISABLED — retired, but may still hold open positions
  • ERROR — stopped after a problem; act
  • KILLED — risk kill switch fired; act, and never just turn it back on

Pages and buttons

Page URL What's there
Trading /trading_view Start, Stop, Safe Mode — engine-wide, affects every strategy at once
Strategy · Current /strategy-current Pick an active strategy → Enable, Disable, Console
Strategy · Console /strategy-console Any strategy, current or past → Enable, Disable, Close
System · State Sync /System?tab=state-sync Verify, Reconcile State, Reconcile Positions

What the actions do:

  • Stop — halts all trading. Your first move when something's wrong and it's still trading.
  • Safe Mode — emergency stop; stays engaged until someone clears it manually.
  • Disable (a strategy) — closes its open positions, then retires it. The main way to get exposure off one strategy.
  • Close (a strategy) — closes its positions but doesn't retire it; use when you might bring it back.
  • Enable (a strategy) — start a stopped one.
  • Verify — runs a state check. Leave Quick unchecked for a Full check when you need to know what's actually wrong (Quick is the light hourly check).
  • Reconcile State — rebuilds balances and state from the exchange. The engine runs this for you automatically when a check fails — you only run it by hand if the app is in ERROR (auto-reconcile already tried and couldn't fix it). Leave the Quick and Use-remote toggles alone — they can be destructive; only touch them with full understanding, otherwise escalate.
  • Reconcile Positions — replays fills to rebuild position records. Use when positions specifically are off.

What to do

Refresh the page first. Post what you do in #alpha-team.

The app auto-reconciled and paused (READY + a "state auto-reconciled" alert)

This is the normal recovery now: an hourly check found state drift, the engine reconciled it, re-checked, and paused itself in READY — trading stopped, positions left as they are, nothing closed. It does not resume on its own. The state is already fixed; your job is the cause.

  1. Don't re-reconcile — the engine already did and the re-check passed. Running it again just repeats work.
  2. Find out why it drifted. Read the alert and the verify history. If it points to a bug or needs a deploy → escalate; don't resume.
  3. If the cause is understood and benign → Trading → Start to resume.
  4. If you'd rather flatten before resuming, the state is now correct, so Close or Disable the affected strategy is safe to do here.

The app is in NOT_READY or ERROR

NOT_READY — a state check failed and auto-reconcile didn't clear it on the first try (at startup or on the hourly cycle). Nothing is broken; the engine will re-check next cycle and can recover to READY on its own. Don't wait for that — find the cause now. If it fails again it escalates to ERROR.

ERROR — the automatic path has given up: the check failed twice in a row, the reconcile errored, the same drift kept coming back, or a service crashed / an exception was thrown. It's on you.

Same steps for both:

  1. If it's still trading and clearly wrong → Trading → Stop.
  2. Open State Sync. Click Verify with Quick unchecked (Full) to see exactly what's failing.
  3. If it's state drift → Reconcile State (the manual fallback). If positions specifically are off, also Reconcile Positions. If reconcile still doesn't clear it → escalate; it likely needs a code fix or a manual heal.
  4. Resume only after a fresh Full Verify passes → Trading → Start. If it won't pass, escalate.

A strategy is in ERROR

  1. Does it hold open positions? If so and it's risky → Disable it (closes positions + retires), or Close it if you may bring it back. A DISABLED strategy can still hold positions if an earlier close couldn't flatten everything — check and close those too.
  2. Check the error reason on the strategy page. A risk or rejection quarantine won't auto-resume and will re-trip if you just re-enable it — fix the cause first; never raise a risk limit only to clear the block. Other reasons usually clear on their own.
  3. Enable it again only once the cause is fixed.

The app sits in READY (not trading) for a while

If it stays READY for ~15 minutes without switching to TRADING, trading was probably stopped manually and never resumed.

  1. First check for a "state auto-reconciled — PAUSED" alert. If there is one, this isn't idle — use The app auto-reconciled and paused above instead (find the cause before resuming).
  2. Otherwise check with the team in #alpha-team whether someone stopped it on purpose (admin work, a deploy). If so, leave it until they're done.
  3. If nobody meant to leave it stopped → Trading → Start to resume.

Anything else, or you're unsure

Stop trading, get exposure flat, and escalate. A stopped engine only misses opportunity; a wrongly-resumed one loses money.

Escalate

Escalate in #alpha-team immediately if:

  • Drawdown over $200 in 24h
  • App in ERROR for more than 15 minutes
  • App still in NOT_READY after the next hourly check (it should have cleared to READY or escalated to ERROR by then)
  • Any KILLED strategy
  • The app auto-reconciled and paused more than once for the same drift (the cause isn't being fixed — the engine hard-stops to ERROR on repeat)
  • Anything that needs a code change or deploy
  • Any time you're unsure whether resuming is safe

Post to #alpha-team with the environment, the affected strategy and its status/error reason, the last verification result, and what you've already tried.

Keeping this current

Update this doc in the same PR as the change:

  • New button or page → update Pages and buttons.
  • New status or error reason → update What the statuses mean.
  • New common failure → update What to do.