perf(base): speed up WS order book/trade/ticker parsing for Python - #2
Open
almostintuitive wants to merge 1 commit into
Open
almostintuitive wants to merge 1 commit into
almostintuitive wants to merge 1 commit into
Conversation
Profiling (cProfile) of binance/okx/hyperliquid WS handleOrderBook,
handleTrade(s) and top-of-book (bookTicker/bidAsk) handlers showed
Python-specific bottlenecks that are invisible in JS:
- safeString/safeFloat/safeInteger/safeValue (and _2 variants) used
dictionary[key] indexing wrapped in try/except. A missing key is the
common case (optional fields, safe*_2 fallback keys, order-book
count/id keys that most exchanges never send) and raising+catching
an exception per miss is far more expensive than a dict.get() miss.
Dicts now take a get()-based fast path; every other input type keeps
the original indexing, so behaviour is unchanged for them (verified
with 500k+ randomized input/type combinations against the original
implementation).
- OrderBookSide (python/ccxt/async_support/base/ws/order_book_side.py,
hand-written) rebuilt full snapshots one price level at a time via
bisect+insert even though the input (a REST snapshot, or a
full-book-per-message feed like hyperliquid's l2Book) is already
sorted. merge_snapshot() now bulk-loads in one linear pass when the
input is provably a clean, sorted run, falling back to the original
per-item path otherwise (verified with 2000 randomized update
sequences, including adversarial unsorted/duplicate-cancel input).
- parseOrderBookBidAsk (ts/src/base/Exchange.ts) attempted a
countOrIdKey lookup on every single price level even though almost
no exchange sends one. parseOrderBookBidsAsks now checks this once
per batch (sampling the first level) instead of once per level -
behaviourally a no-op change (a confirmed-absent key still resolves
to "absent"), verified against the original across 7 level shapes
(arrays, objects, missing keys, mixed shapes within a batch).
- hyperliquid resends its *entire* order book on every message.
handleOrderBook now parses raw {"px","sz"} levels directly instead
of routing every level through the generic dict/array-agnostic
safeFloat pipeline, falling back to the generic path (per batch) on
any NaN/malformed result. Verified against the original
implementation end-to-end (300 randomized messages, including
malformed levels) with zero mismatches.
Measured with cProfile + wall-clock benchmarks against synthetic but
realistic WS messages (offline, no network) on this branch vs
unmodified master:
binance handleOrderBook 14.74us -> 11.12us (1.33x)
binance handleTrade 9.19us -> 8.06us (1.14x)
binance handleBidsAsks 9.22us -> 7.80us (1.18x)
okx handleOrderBook 13.68us -> 12.49us (1.10x)
okx handleTrades 8.89us -> 8.33us (1.07x)
okx handleBidAsk 8.78us -> 7.62us (1.15x)
hyperliquid handleOrderBook (l2Book, 400 levels)
320.94us -> 175.37us (1.83x)
hyperliquid handleTrades 16.40us -> 16.14us (~unchanged)
hyperliquid's order book got the largest win since it resends the
full book on every message; binance/okx only send small deltas per
message, so there are fewer safe*() calls to save per message.
Also removes an OrderBookSide._n/`__len__`/`__getitem__` override that
was provably dead code (never set anywhere in the codebase).
Not touched in this PR (deliberately, after prototyping and rejecting
both): a per-symbol level memoization cache for hyperliquid (the
bookkeeping overhead exceeded what it saved, even at ~97% cache-hit
rates) and a new "trusted bulk load" method on OrderBookSide shared
across all 5 language implementations (would close most of the
remaining gap to a 3x order-book speedup, but touches hand-written WS
base classes in JS/PHP/C#/Go/Java that this PR's testing couldn't
fully cover - flagging as a well-scoped follow-up).
Verify-after-change checklist:
- [x] npm run lint
- [x] npm run tsBuild
- [x] npm run transpile (Python + PHP), php -l on all 3 touched files
- [x] npm run transpileCS + npm run buildCS (dotnet build succeeded)
- [x] test-base-ws-py, test-base-rest-py (both pass)
- [x] 5 custom fuzz/property tests vs. the original implementation
(safe_* primitives: 505k checks; OrderBookSide: 2000 rounds;
parseOrderBookBidsAsks: 333 cases; hyperliquid level parsing:
500 trials; full handleOrderBook: 300 end-to-end trials) -
zero mismatches across all of them
- [ ] Go / Java transpile+build - not run in this environment per
operator instruction; CI's own pre-transpile/build steps
regenerate these from ts/src/ regardless of what's committed
- [ ] live smoke test - no network access in this environment
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XcB8Z2PGyuzfqpkdTrWZ2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Profiled (cProfile) the Python WS hot paths for binance/okx/hyperliquid's
handleOrderBook,handleTrade(s), and top-of-book (bookTicker/bidAsk) handlers using realistic synthetic messages (offline, no network — real market data fixtures fromts/src/test/static/markets/). Found and fixed Python-specific inefficiencies that are invisible in JS but real in Python:safeString/safeFloat/safeInteger/safeValue(+_2variants) —dictionary[key]wrapped in try/except raises+catches an actual exception on every missing key. A missing key is the common case (optional fields,safe*_2fallback keys, order-book count/id keys almost no exchange sends) — dicts now take adict.get()-based path that never raises for a miss; every other input type (lists/tuples) keeps the original indexing unchanged.OrderBookSide.merge_snapshot(python/ccxt/async_support/base/ws/order_book_side.py, hand-written) — rebuilt full snapshots one price level at a time via bisect+insert even when the input (a REST snapshot, or hyperliquid's full-book-per-messagel2Book) is already sorted. Now bulk-loads in one linear pass when the input is provably a clean sorted run, falling back to the original per-item path otherwise.parseOrderBookBidAsk/parseOrderBookBidsAsks(ts/src/base/Exchange.ts) — attempted acountOrIdKeylookup on every price level even though almost no exchange sends one. Now checks this once per batch (sampling the first level) instead of once per level. Purely additive to the existing signature (no new parameter), so it's a no-op for every existing caller including exchanges that overrideparseOrderBookBidAsk(kraken, mexc).handleOrderBook— hyperliquid resends its entire order book (typically hundreds of levels) on every message. It now parses raw{"px","sz"}levels directly instead of routing every level through the generic, dict/array-agnosticsafeFloatpipeline meant for arbitrary shapes, falling back to the generic path (for the whole batch) if anything looks malformed (NaN check)._n/__len__/__getitem__override onOrderBookSide—_nwas never set anywhere in the codebase, so this was pure overhead on every list access.Results (this branch vs. unmodified master, offline benchmark)
handleOrderBookhandleTradehandleBidsAskshandleOrderBookhandleTradeshandleBidAskhandleOrderBook(400 levels)handleTradeshyperliquid's order book gets the largest win because it resends the full book every message (hundreds of
safeFloatcalls saved per message); binance/okx only send small deltas, so there's less to save per message.Honest note on the 3x target: I did not reach a uniform 3x. I prototyped two further ideas to close the gap on hyperliquid specifically and rejected both after measuring: (a) a per-symbol level memoization cache — the bookkeeping overhead exceeded the savings even at ~97% cache-hit rates; (b) a new "trusted bulk load" method on
OrderBookSideshared across all 5 language implementations, which would likely close most of the remaining gap but requires touching hand-written WS base classes in JS/PHP/C#/Go/Java that I couldn't fully verify in this session (see Notes). Flagging as a well-scoped follow-up rather than shipping it half-verified.Test plan
npm run lintnpm run tsBuildnpm run transpile(Python + PHP),php -lon all 3 touched PHP filesnpm run transpileCS+npm run buildCS(dotnet build succeeded, 0 errors)test-base-ws-py,test-base-rest-py— both passsafe_string/safe_float/safe_integer/safe_value(+_2): 505,404 checks across dicts/lists/tuples/edge values (None, NaN, bools, empty containers, unhashable-adjacent cases) — 0 mismatchesOrderBookSide: 2000 randomized update sequences incl. adversarial unsorted/duplicate-cancel input — 0 mismatchesparseOrderBookBidsAsks: 333 cases across 7 level shapes (arrays, objects, missing keys, mixed shapes within one batch) — 0 mismatcheshandleOrderBookend-to-end: 300 randomized messages incl. malformed levels, comparing the resulting order book state — 0 mismatchests/src/regardless of what's committed here, so this should not block CI, but flagging since CLAUDE.md asks for 5-language verificationNotes
ts/src/base/Exchange.ts,ts/src/pro/hyperliquid.ts, their generated JS/PHP/C# output, and the two hand-written Python WS base files. Go/Java are left for CI to regenerate.php -lnow passes clean on all three touched PHP files.test_fetch_tickers.py,coinone.cs/coinone.gobehind their current TS source) — left those alone rather than bundling unrelated fixes into this PR.🤖 Generated with Claude Code
https://claude.ai/code/session_013XcB8Z2PGyuzfqpkdTrWZ2