2026 - 07¶
Goal: Get one strategy to positive profit margin on a significant sample. | Phase: 1 (Validate)
Month headline — the platform month, by design. July made a deliberate trade: freeze the strategy surface, spend the whole engineering capacity on infrastructure. Both halves paid. The freeze let MM run undisturbed to 59,394 fills — the decisive sample the goal asked for — and the freed capacity landed the Copilot → CDK migration on 07-31, retiring an end-of-life deploy tool and taking release time from 21.2 → 15.8 min, non-engine trading downtime to zero, and the Tokyo move from a rebuild to a one-line config change. The strategy answer is the honest kind: MM was margin-positive for ~27k fills, then broke negative at ~07-18 and stayed there. That break is the most interesting object of the month — and it is not yet cleanly attributable, because the size A/B ran on the same book as its own control. Meanwhile the surviving positive edge arrived unplanned: cross-venue arb, now the best margin in the book by 3×.
The other headline is on the research side, and it runs the opposite way: the direction family came back from the dead. June killed it class-wide on an "execution / adverse-selection" verdict; July's fill-probability work showed that verdict was measuring the backtest kernel's design, not the market. On a corrected design, maker beats taker in every window tested. Two of the month's biggest results — MM's break and direction's revival — are the same lesson pointed in opposite directions: the apparatus was the finding.
1. Strategy: MM — answered, with an asterisk — 🔶¶
The July success bar was one MM arm margin-positive on a significant fill sample. It was met mid-month and then lost. Cumulative margin made this look like slow decay; differencing it shows a clean regime break.
Marginal P&L per fill, eth_okx_sigcancel2_r2:
| Window | Δ P&L | Δ fills | Marginal $/fill |
|---|---|---|---|
| → 07-10 | +$4.22 | 8,495 | +4.97e-4 |
| 07-10 → 12 | +$0.27 | 3,923 | +6.9e-5 |
| 07-12 → 18 | +$1.26 | 15,448 | +8.2e-5 |
| 07-18 → 26 | −$3.71 | 18,067 | −2.05e-4 |
| 07-26 → 08-01 | −$4.73 | 13,461 | −3.51e-4 |
- Ran to a decisive sample. 59,394 fills — the "significant sample" the goal required. No sample-size excuse remains for MM on OKX ETH.
- Arms resolved and closed with evidence: basis A/B,
svdefensive-cancel, wider-spread alt — all three canary children closed. Cross-quote follower built, promoted, and retired. Stablecoin peg promoted to LIVE, then superseded by the netted baseline. - The one MM structure still positive:
usdcusdt_net— the netted stablecoin arm, +3.4e-5 over 222 fills. Positive but economically negligible at+$0.08. - The break at ~07-18 is not attributed. Two live hypotheses, and they are not mutually exclusive:
- (a) Regime. The same strategy earned for weeks and then didn't. What changed in the market — volatility, spread, book depth, who was on the other side — and what happened in the world that the data alone won't say. This is the genuinely valuable question and it is unanswered.
- (b) Self-competition (experiment contamination). The size-up arm
q50went live 07-17, quoting the same symbol and venue as the control. Netted MM became the baseline and cross-symbol exit books shipped the same week — so concurrent arms share position and exit logic and compete for one maker queue.r2's marginal P&L flips negative in the very next window.
- A free natural experiment is already running.
q50andvolgate_r3were disabled ~07-31;r2has been alone on the book since. If its marginal $/fill recovers, the July read was contaminated and MM is unmeasured, not dead. If it stays negative alone, MM on OKX is answered. This gates how August starts.
Cost of the MM experiment: the ETH-OKX complex ran -$26.23 (r2 -$2.69 · q50 -$19.48 ·
q100b -$4.06), plus btc_volgate_r3 -$3.45. That is ~46% of the month's realized loss, and it bought the
59k-fill sample plus the size answer.
2. Strategy: Cross-venue arb — the unplanned survivor — ✅¶
Not in the July plan. It is now the best-performing thing we run.
- Promoted to LIVE and hardened to v3: sequential execution mode with reliable orphan close, book-freshness gating keyed to the lead venue, velocity gating retained in LIVE, ETH promoted as a small sequential arm.
-
btcusdt_okxbus_arb_r: +$3.96 over 907 fills at 9.6e-5 margin — roughly 3× the best margin MM ever reached, and it strengthened through the freeze rather than decaying. -
ethusdt_okxbus_seq_rflipped and held positive: +$0.21 / 127 fills / 7.3e-5. - A 4 Hz tick was tried and reverted — it exposed a double-hedge race; the arms run at 1s.
-
usdcusdt_okxbus_arbhas sat at 0 fills; in-flight-hedge guard still open. Sample is young — 907 fills is promising, not decisive.
3. Platform: Copilot → CDK migration — ✅ (the month's largest deliverable)¶
AWS Copilot — which owned every environment, service manifest and deploy pipeline — hit end-of-life 2026-06-12. Staying meant running live trading on a tool that could never change again while everything beneath it did. The cutover executed 07-31, ~01:00–19:00 UTC: both environments live and trading on the new platform, zero data loss, zero unplanned trading loss, first real venue order accepted at 18:26 UTC.
- Measured gains: single-env release 21.2 → 15.8 min; two-env releases now parallel instead of serialized; non-engine releases cost zero trading downtime (was ~5 min every release); a no-op release skips in minutes; full disaster rebuild ~45 min, drilled repeatedly (never rehearsed before).
- Encryption at rest everywhere — every Copilot Aurora cluster was unencrypted and could not be encrypted in place. The restore path was the one moment it could be introduced; all four clusters now run encrypted with deletion protection and 7-day point-in-time recovery.
- The book is protected by construction, not convention — engine deploys are recreate-only, and
READ_ONLYmode lets a live-configured environment boot fully (real keys, real IP, restored book) while structurally unable to write to a venue. That is what made rehearsing the live environment possible. - Egress identity survives rebuilds — NAT EIP pinned by allocation id, so venue IP allowlists survive a full environment rebuild.
- Restore and arm were separate merges. The restored live book was inspected while the engine could not act on it; the flags meaning "this environment is real" moved together in one reviewed diff, after the evidence existed. Rollback stayed free until the final step.
- What the migration surfaced (found because of it, fixed before it mattered): a monitoring API key never provisioned — every monitoring path would have gone blind post-cutover; ML's health probe ported at half its timeout, killing healthy tasks; ML declaring a telemetry token it never received, so ML had shipped no traces, ever.
- Decommission of the retired Copilot stacks remains.
A trading consequence hiding in an infra deliverable: the Tokyo region move — exchange colocation — is now a one-line
region=change rather than a full rebuild. Whether to take it is a trading decision, no longer an infra project.Full detail:
inf-trading/docs/architecture/2026-07-copilot-to-cdk-migration.md
4. Engine: Capital integrity & the risk framework — ✅¶
- Found and fixed real ledger corruption. The rebalance planner, the verification rules and the balance recompute all keyed on account, while balance rows key on the venue slot. This generated phantom internal transfers that corrupted the capital base the entire P&L reads from, and caused the hourly reconcile to overwrite correct balances. Two further races were found where a recompute erased fills that landed mid-pass. Required a one-off data repair. The structural root cause — making the engine account-aware end to end — is filed and still open.
- First real capital-allocation control surface: per-venue exposure caps with reduce-only enforcement, per-strategy capital budgets, and a read-only rebalance plan. This is the precondition for running many strategies against shared capital without one starving another.
- Python 3.9 → 3.12 shipped, which required privately forking the unmaintained
modedependency — closing a runtime-debt item that had been open since the year began. - Simulation fidelity: paper and backtest were measured under-filling ~5–7× versus live; dynamic maker queue decay, latency realism and taker liquidity depletion closed most of the gap, with an orders-level comparator so it stays measurable.
- Crash-resilience: a recurring pattern where one bad row or one transient venue error crash-looped the whole application was converted to degrade-not-crash across a dozen call sites.
- Data retention: strategy-execution records had reached ~1.1 GB/day unbounded; 30-day pruning now caps both that and the order-book table.
Engine shipped v1.158.0 → v1.196.1 — 38 minor versions, 86 releases in 31 days, the highest-velocity month on record.
5. ML — ✅ (the month's other headline: execution stopped being the bottleneck)¶
ML's July output is larger than the engine-side view of it suggests, and it changes a June verdict. The anchor was fill-probability (#299) — ~30 sections and 30+ commits — which delivered a shipping surface, then used it to re-score shelved workstreams, and in doing so reopened the direction family.
- Fill probability is predictable — and separable from fill quality. Within a fixed (offset, horizon) cell, trailing state ranks fill/no-fill at holdout AUC 0.753 (shuffle control 0.50). This is the opposite of toxicity, which is adversarial and unpredictable at ~0.5. Almost all the signal is one axis: realized vol — time-of-day, velocity and flow add nothing or overfit.
- Ship the table, not the model. A doubly-monotone (offset, horizon, realized-vol) surface reaches Brier 0.0745 / ECE 0.010; a GBM reaches 0.0726 / 0.0059 — but the table already closes 77% of the physics-null → GBM gap, and both pick the same argmax-EV offset. The remaining 1.7% is economically trivial.
- A clean factorization, validated on 220k reconstructed orders from real OKX v16 MM runs:
P(fill) = P(trade-through) × q(H)— the first term free from public tape on any coin/venue, the second a live, execution-setup-specific queue-completion number (four values per coin). Trade-through is a near-perfect necessary condition (P(fill | no trade-through) ≈ 0.3%) but not sufficient at short horizons — that residual is pure queue: 44% @1s → 71% @2s → 90% @5s → 99% @15s on BTC. Raw tape alone overstates short-horizon fills ~2×. - "Execution is the bottleneck" was wrong — the wheel kernel was. The June verdict that the direction family was killed class-wide by an adverse-selection fill gate turns out to be a design artifact: long rest × anchored exit. Under the corrected design — short rest ≤300s, skip-if-unfilled, fixed-horizon exit — maker ≥ taker in every one of 8 stratified windows (n=489 signals, ΔEV +2.9…+6.9 bp, t 1.97/2.35/3.54). Unfilled bars are runaway bars, so skipping them is free selection.
- ETH direction reopened on the corrected design — +9.88 bp/signal (t 2.08) across 9 windows back to 2023, with no win-rate collapse. The old ETH kill was a wheel-kernel artifact. A maker-first exit was validated separately (+3.48 bp, t 8.84) and DirectionUp v3 implemented on it.
- A 3-book portfolio (BTC + ETH + SOL) at Sharpe 1.58, with a pre-registered OOS protocol locked for 2026-10-01 — no re-tuning until then.
- Direction optimization closed on every other axis — feature selection adopted (FS-15), 5-minute multiscale rejected, aggTrades tape features rejected as redundant, triple-barrier and up-only labels closed on a pre-registered FINAL veto, hyper-parameter tuning zero-gain over 80 trials. The remaining space was execution — which is exactly where #299 then delivered.
- Two kills upgraded from "shelved" to "measured": cascade-fade maker re-pricing (652 events, 72 cells, all net-negative) and funding-settlement fade (the June positive was a multi-coin pooled artifact; the effect is absent on BTC/OKX-spot). One reopen, two kills — the re-score process is unbiased in both directions.
- A 24-defect architecture audit, each with a regression test. Several were silently invalidating results: a focal-loss gradient that sign-flips into anti-learning below p=0.607 (every custom-loss experiment in the history is invalid), LSTM evaluation fitting calibration and thresholds on test labels, cumulative indicators computed from frame start in training but a few-hundred-bar window live, and live funding features freezing for weeks at each month start.
- ETH predictions served (#298) — and an adversarial audit caught 3 HIGH defects that would have broken the existing BTC endpoint on deploy: resolving endpoints by name alone meant deploying ETH under a shared name routed to UPDATE and would have overwritten BTC's model version, while silently killing BTC's metrics.
- Deep-learning pipelines made dev-only to cut production deploy time; migrated to Python 3.12 alongside the engine.
- ML is wired into a strategy and running — DirectionUp on BTC, maker and taker variants, trading in paper off the live prediction endpoint. Deliberately not promoted: v1's read was thin, and v3 (the validated design) is the arm that deserves the live test.
- A paper-environment defect blocks validating this class. POST_ONLY orders in paper have an effective lifetime of ~one run-interval (39–71s), so a strategy's 480s cancel never fires — the engine cancels first. Paper cannot currently validate any rest-horizon maker strategy, which is precisely what DirectionUp v3 and the MR maker hook are.
- The prediction outage recovered at cutover. Predictions resumed 2026-07-31T20:00Z and have run 13/13
consecutive 2h bars with no gaps through 08-01T20:00Z, identically on both environments —
error_tasks: 0, six active endpoints each. The failure signature is gone: context lag is a flat 2h (the normal one-bar lag), where the defect's signature was a widening gap as the context froze at 07-28 22:00Z while bars advanced. - But the migration likely cleared it, not fixed it — so it will recur. The evidence in the open issue was a byte-identical NaN count (415/1728) across ~450 cycles, meaning the futures-metrics file was never being re-downloaded at all. A stale file on task-local ephemeral storage is exactly what a full task replacement wipes, and the cutover replaced every task ~1.5h before predictions resumed. Nothing yet refreshes that file reliably. The next trigger won't be a migration that happens to fix it.
- The detection gap is the more serious finding. This ran ~3 days at 100% failure and the only signal was a repeated "missed predictions: 2 consecutive bars" ping that never escalated past 2 — a total outage that looked like a minor blip. The counter appears to reset rather than accumulate, so severity never grew. (Inferred from the alert history; not yet confirmed in code.)
- The VSR→MR regime gate was closed as execution-gated rather than shipped.
6. Frontend — ✅¶
- Strategy area redesign: segmented Console (Overview / Executions / Orders-Fills / Trades P&L) and a server-side History hub with filtering, pagination and multi-strategy compare.
- Latency view — reaction, gateway-operation, order-submit and fill-delivery latency, the surface that makes latency claims measurable rather than rhetorical.
- Accounting rebuilt around a balance matrix (account × asset) with graded inventory health, and transfers streamlined to auto-sync.
- Agent-callable operator actions — the de-risked tier of the runbook exposed as API, so routine operator steps can be executed by an agent rather than by hand.
- Benchmark and excess-return display scaffolded on the KPI surfaces; snapshot bundle export; embedded application-services dashboard.
7. Equities viability scan — ✅ (answered: no — and it found something better)¶
The plan's fourth item was delivered on day two and reviewed the same day. It did its job: it came back negative on US equities, which is why equities research was deliberately not pursued further. A scan that kills its own subject early is the cheapest possible outcome.
- Viability memo + Phase-0 results — maps the three edge roots onto US equities. Verdict: don't commit — same three roots, a higher cost floor, the oldest and most-competed substrate, zero transferable infrastructure. Grounded, not hand-waved: PFOF internalization ~80%+, S&P index-effect decay 7.6% → 0.8%, auction flow ~9–10%/day.
- Honest triage of the three directions: value selection → fast-kill; portfolio construction → not an edge (it is a CVaR risk container); long-tail convexity → the only real bandwidth sink. They compose not into a stock-picking book but a convexity barbell — short rich vol funding long cheap tail, CVaR overlay, the tail leg hedging the short-vol ruin.
- Deprioritized on the strength of its own negative result. With the initial read negative, further US-equity research was not worth the founder or research hours against the crypto radar — exactly the opportunity-cost red line the memo itself proposed.
- Open pending author fixes, not a decision: one dangling sibling reference, a title convention, and an outstanding question — what does the model show on free kline data across assets? The point being that if we are not going to run HFT on equities, there is no case for buying tick/order-book data.
The memo's most valuable output isn't about equities at all. Buried under the stock-picking framing is this: US-listed options give us an accessible home for the VRP / vol-selling edge that crypto's derivatives access-gate denied us. June parked VRP as "real but gated on derivatives access" — this un-gates it on a different substrate. That is a vol/options-access question, a sibling to the crypto derivatives-access decision, and it deserves its own thread rather than dying inside an equities memo that was correctly rejected.
The caveat travels with it: the barbell's carry engine is VRP, and VRP is the same edge our own prior work judged marginal and decaying (BTC net Sharpe ~0.6–1.1, ETH uninvestable), while SPX VRP is heavily crowded — the memo's own $127bn vol-ETF and 59% 0DTE figures. The elegant structure inherits the weakness of its most-competed component.
Separately, research bandwidth also produced: miner-hashprice scan proposal, stat-arb taxonomy closure, funding-carry resolved do not build, portfolio-construction and T-bill baseline sleeve.
Financials¶
| Jul 1 | Aug 1 | Δ | |
|---|---|---|---|
| NAV | $11,011.03 | $18,123.00 | +$7,112 — capital added, not earnings |
| Realized P&L | −$70.08 | −$135.15 | −$65.07 |
| ROI | — | −0.78% | flat |
| Win ratio | 0.671 | 0.659 | −0.012 |
| Max drawdown | 0 | 0 | — |
July cost -$65.07 realized. Roughly -$29.7 of that is the MM programme (the ETH-OKX complex plus the BTC
vol-gate arm) — the price of the 59k-fill sample and the size answer. MR contributed -$2.23. The remainder is arm
churn: strategies created and retired within the month. Against it, the live book ends the month with three
margin-positive arms (arb btc +$3.96, arb eth-seq +$0.21, usdcusdt_net +$0.08).
Live book at month end — 12 strategies, 9 active:
| Strategy | Margin | Fills | P&L |
|---|---|---|---|
arb btcusdt_okxbus_r |
+9.6e-5 | 907 | +$3.96 |
arb ethusdt_okxbus_seq_r |
+7.3e-5 | 127 | +$0.21 |
MM usdcusdt_net |
+3.4e-5 | 222 | +$0.08 |
| MR v21 (4 arms) | −5e-4 to −1.1e-3 | 24–53 | −$2.23 |
MM eth_sigcancel2_r2 |
−4e-6 | 59,394 | −$2.69 |
MM btc_volgate_r3 (disabled) |
−9.3e-5 | 2,422 | −$3.45 |
MM eth_q100b (disabled) |
−1.05e-4 | 931 | −$4.06 |
MM eth_q50 (disabled) |
−2.9e-5 | 27,610 | −$19.48 |
Incidents¶
Live trading halted for ~26 minutes (Jul 29). A safety check that compares our books against the exchanges started failing on stale leftover account records — not a money discrepancy, a bookkeeping duplicate. The system did what it is designed to do: it refused to keep trading while it could not verify its own state. An operator cleared the stale records and trading resumed on its own, without a restart. No money was at risk and no positions were affected — the checks that guard against actual drift passed throughout. The fix-forward — don't halt live trading on a bookkeeping-class check — is still open.
A balance mismatch on OKX (Jul 25). Our record of USDT held on OKX disagreed with the exchange by ~2,500 for about an hour, then cleared on the next restart and re-verified clean. Reporting-only; no fund movement.
The capital ledger was quietly corrupted, then repaired (late July). The single most consequential defect of the month. A mismatch in how internal transfers were identified caused the system to invent transfers that never happened, corrupting the capital base every profit figure is measured against — and the hourly self-correction was overwriting correct numbers with wrong ones. Found, fixed, and the historical data repaired. Profit figures from before the repair should be treated with suspicion; figures after it are trustworthy.
Paper environment down ~14 hours. A single malformed record crash-looped the application. Paper only — no live trading impact.
ML predictions offline for ~3 days (Jul 28-31), recovered at cutover. A market-data file the forecasts depend on stopped being refreshed, and the prediction service — correctly — refuses to publish a forecast built on stale inputs, so it published nothing at all. No live strategy depends on ML output, so trading was unaffected. Predictions resumed the moment the migration replaced every running task, and have been clean since.
Two things about it are worth more than the outage. It probably recovered by accident: the stale file lived on storage that a task replacement wipes, so the cutover cleared the symptom without anyone fixing the cause — nothing yet guarantees that file refreshes, so it can return. And it hid in plain sight for three days: the only alert was a repeated "missed 2 consecutive bars" notice that never escalated, because the counter resets instead of accumulating. A total, multi-day outage looked exactly like a brief hiccup.
Team¶
July's commit counts overstate strategy work and understate research: the strategy surface was deliberately frozen after 07-20, and most of the month's volume is platform.
Self-reports from Vicky and Gaddafi to be folded in.
MJ (~138 engine · ~103 infra · ~68 strategies · ~40 frontend · ~22 ml)¶
- Ran the Copilot → CDK migration end to end — platform build, the rehearsal drills, the freeze/restore/verify discipline, and the cutover itself. Including the judgement calls that made it safe: separating restore from arm, and rehearsing the live environment in read-only mode.
- Found and fixed the capital-ledger corruption, plus the two fill-erasing races, and repaired the historical data.
- Built the risk framework — per-venue exposure caps, per-strategy capital budgets, rebalance plan.
- Drove the MM programme to its decisive sample — the arms, the size A/B, and the freeze decision that produced a clean read.
- Promoted and hardened cross-venue arb — the month's one new positive edge.
- Python 3.12 migration including the private
modefork; simulation-fidelity work; retention.
Vicky (~9 ml · ~6 strategies — commit count badly understates the month)¶
- Fill-probability (#299) end to end — ~30 sections, 30+ commits: the spec, the shippable (offset, horizon, vol) surface, live validation against 220k reconstructed orders, and the trade-through × queue factorization. The month's largest single research deliverable.
- Reopened the direction family by finding that June's class-wide execution kill was a wheel-kernel design artifact, then re-scoring 8 stratified windows to show maker ≥ taker everywhere, extending it to ETH (+9.88 bp/signal), implementing DirectionUp v3 on the validated design, and locking a pre-registered OOS protocol for the 3-book portfolio.
- Closed the direction-optimization campaign on every remaining axis — features, targets, hyper-parameters, tape features — with a pre-registered FINAL veto that correctly killed a dev result worth +22 bps that was 55-trade luck.
- The 24-defect architecture audit, including the focal-loss sign-flip that invalidated every custom-loss experiment on record, and the LSTM evaluation that was fitting on test labels.
- Caught 3 HIGH defects before deploy (#298) that would have overwritten the live BTC endpoint's model version.
- Killed the band-resting MR maker hook on live episodes after her own offline proxy said it was worth +2.7 bp — and identified the surviving variant.
- Delivered the equities viability scan on 07-02, two days into the month.
Gaddafi (~6 frontend)¶
- Application Services Dashboard embedded into the platform — service health without leaving the app.
- Benchmark and excess-return metrics in Trading and Account KPIs — the foundation for portfolio benchmarking.
- Compare v2 — Manual and By-Class comparison unified into one workflow with cascading scope selection and instance roll-up.
- Snapshot Bundle Export on System Status — one-step diagnostic capture for troubleshooting.
- Verification requests fixed to send status enum values correctly (was failing server-side validation); UI padding refinements.
Phase Assessment¶
Still Phase 1 (Validate). No strategy is locked. But the shape of the problem changed twice this month.
First: the platform stopped being a constraint. June's lesson was build the measurement before the search. July applied it one level down — to the operational substrate. Iteration speed is no longer gated by a dead deploy tool, a non-engine release no longer costs trading downtime, and a full environment rebuild is a rehearsed 45 minutes. That is what makes August's plan — rapid strategy iteration — actually executable.
Second: the leading candidate changed. July began expecting MM to be the answer. It ends with MM ambiguous and cross-venue arb holding the best margin in the book by 3×, discovered off-plan. The survivor table moves:
| Survivor | State at month end | Gate |
|---|---|---|
| Cross-venue arb | +$3.96 @ 9.6e-5, 907 fills — best margin we run | decisive sample; in-flight-hedge guard; latency→capture elasticity |
| OKX MM (v16) | positive to ~27k fills, then negative to 59k | attribute the 07-18 break — regime vs. self-competition; the r2-alone read decides it |
| Stablecoin netted MM | positive but negligible (+$0.08) | economic size, not sign |
| MR (liquidity provision) | all four arms negative, but fill-starved (24–53 fills, at/below the decisive gate) | sample, then execution design. Offline-vs-live gap explained (closed-bar entry blindness); band-resting maker hook killed on live episodes (−4.63 bp, t −3.9); survivor is trigger-time maker-ization (~+2.3 bp), unpriceable offline → live-tiny A/B |
| Direction family — REOPENED | corrected design: maker ≥ taker in 8/8 windows (ΔEV +2.9…+6.9 bp); ETH +9.88 bp/signal; 3-book Sharpe 1.58 | pre-registered OOS eval 2026-10-01, no re-tuning until then; v3 needs a live test, and paper can't give it (POST_ONLY lifetime defect) |
| Momentum premium | ran inert 21 days undetected, now fixed | needs a clean run before it means anything |
| US equities as a substrate | answered no (memo 07-02, reviewed same day) — most-competed substrate, higher cost floor; deprioritized on its own negative read | closed |
| Vol / options access (the memo's real find) | US-listed options un-gate the VRP edge crypto's derivatives gate denied us | is VRP worth having on a substrate where it's more crowded? — its own decision, not an equities one |
The honest read on the July goal: it asked for one strategy margin-positive on a significant sample. Two arms
are margin-positive at month end, but neither on a significant sample — arb is 907 fills, usdcusdt_net is 222.
The one strategy that reached a significant sample went negative on it. The goal was not met. What was bought
instead is a real answer about MM's ceiling, a platform that no longer taxes iteration, and an unexpected candidate
with better economics than the one we were pursuing.
Carried into August, which is the rapid-iteration month: attribute the MM break before deciding MM's fate; drive arb to a decisive sample; record the MR verdict; and note that hidden/iceberg order support still gates the MM capacity question, and warm-standby handover still costs ~5 minutes of trading per engine release — a tax that compounds at August's intended iteration rate.
Learning¶
Freezing the experiment surface is a measurement decision, not idle time. The instinct is to read eleven days without a strategy merge as lost momentum. It was the opposite: MM needed an undisturbed book to reach a decisive sample, and the migration needed undivided capacity. Both required the same thing — a still surface — so the freeze bought two results at once. The failure mode worth naming is the reverse: constantly reshuffling arms means never accumulating enough uninterrupted fills to conclude anything, which is how the first half of the year went.
Twice this month, a "result" turned out to be a property of the apparatus rather than the market — once costing us a verdict we wanted, once hiding one we'd already thrown away. In MM, the size-up arm quoted the same book as the arm it was being compared against, at exactly the moment netted positions and shared exit books made concurrent arms contend through the queue, the inventory and the exit path; the control's marginal P&L flipped negative in the very next window, leaving a 59,394-fill sample whose interpretation is ambiguous — the one thing a decisive sample was supposed to prevent. In direction, June's class-wide "adverse-selection fill gate" kill turned out to be the backtest kernel's design — long rest × anchored exit — and not a market fact at all; under short rest, skip-if-unfilled and a fixed-horizon exit, maker beats taker in every one of eight windows, and an entire family we had written off came back.
The two errors are mirror images, and the shared lesson is that the measuring apparatus is part of the hypothesis. A negative result earns the same scrutiny as a positive one — we habitually re-check wins for overfitting and accept losses at face value, which is precisely backwards when the apparatus is young. Concretely: isolate A/B arms by symbol, venue or time-slice and never concurrently on one book; and before accepting any class-wide kill, ask which properties of the harness the verdict is actually measuring.
Re-price on the strategy's own episode log before building. The MR maker hook was priced offline at +2.7 to +3.1 bps from a proxy mechanism. Reconstructed against 55 real MR entries with real fees, it came back at −4.63 bps (t −3.9) — and both proxy terms had been wrong in the unfavourable direction, not just one. A proxy is a hypothesis about a mechanism; the strategy's own logged episodes are the only place that hypothesis gets tested. The saved build was worth more than the analysis cost.
Cumulative metrics hide regime breaks; difference them. The MM arc read as gentle decay for two weeks because 27,000 profitable fills were averaging against the losing ones. Differenced into marginal P&L per fill, it is not decay at all — it is a clean break on ~07-18 followed by monotone worsening. A cumulative average is a lagging, self-smoothing statistic, and reading strategy health off one delays every verdict it informs.
Rehearse until the real event is a merge. Every migration defect that mattered — a probe timeout at half its value, an unprovisioned secret, a declared-but-never-delivered token, an API-only ECS constraint — was invisible to synthesis, tests, and code review. All of them were caught by running the real thing and reading what it actually did. The drills earned their cost not by proving the happy path but by making the failures cheap and early, so that the irreversible step was a reviewed diff taken with evidence already on the table.
A component can report healthy and be doing nothing — and our alerting cannot currently tell the difference
between a blip and a multi-day outage. Twice this month. The momentum arm sat ACTIVE for 21 days without
trading: daily bars never reached it, it skipped silently, and no dashboard, alert or status field showed anything
wrong. Then ML predictions failed for three straight days while the only signal was a repeated "missed 2
consecutive bars" ping that never escalated — because the counter resets rather than accumulates, a 100% outage and
a brief hiccup emit the identical alert. Two distinct lessons sit here: anything that can decline to act must log
the decline, and an alert whose severity cannot grow with duration is not a monitor — it will describe a
catastrophe in the vocabulary of a nuisance. Both failures were found by a human eventually looking, which is the
detection method we keep saying we've replaced.
The ledger is not self-validating. The capital-base corruption came from the same class as June's dust incident: a component checked its own arithmetic against itself and agreed. The correction, again, was to recompute ground truth from raw fills. Twice in two months this class has produced numbers that looked plausible and were wrong — and because every profitability judgement is measured against that ledger, it is the one place where "looks fine" is worth nothing.