Feat/flow worker#428
Merged
Merged
Conversation
|
All contributors have signed the CLA. Thank you! |
ErKristof
force-pushed
the
feat/flow_worker
branch
from
5bce076 to
1916d87
Compare
Contributor
Author
|
I have read the CLA Document and I hereby sign the CLA |
pigri
added a commit
that referenced
this pull request
Jul 14, 2026
Contributor
|
I have read the CLA Document and I hereby sign the CLA |
pigri
added a commit
that referenced
this pull request
Jul 15, 2026
Mirrors the flow-classifier pull worker: polls the version.json manifest, downloads a new encoder ONNX on version change, verifies sha256, atomically promotes it with rollback, and hot-swaps it into cortex::jepa (which builds the new pool + keeps the local vocab/reference). Adds SSE-push-driven immediate sync with interval backstop. Registered under the classifier feature. Tests (TDD): sha256-mismatch aborts without writing; empty-manifest no-op; and a loopback-HTTP end-to-end pull → verify → hot-swap of the real cortex jepa encoder fixture, asserting current_model()==pulled version + resize-after-swap. NOTE: depends on cortex carrying the jepa module (forward-port on the local cortex feat/jepa-forward-port); synapse's cortex [patch] points at the local checkout for dev and is intentionally not committed.
- cli: Config.jepa + MlConfig.jepa (nested ml: section) + load-time migration and the deprecated-top-level-key warning, mirroring flow_classifier. - app: initialise cortex::jepa from the on-disk model+vocab when present, and register the JepaModelWorker (interval-polling) when a manifest_url + refresh are configured; API-key Bearer auth added to its requests like the other models. - worker: re-export jepa_model. The worker runs on its refresh interval; SSE 'jepa-model-updated' push wiring (config_push arm + with_jepa_push) is a small follow-up. Runtime consumption (triage -> jepa_signals -> ml.jepa.* amygdala fields) is the next phase.
Per cortex docs/model-naming.md: config key ml.jepa -> ml.jepa_embedder (with serde alias "jepa" for one-release back-compat), worker name jepa-model-sync -> jepa-embedder-sync, and all cortex::jepa -> cortex::jepa_embedder references following the module rename. Types keep the Jepa* family prefix; the worker file stays jepa_model.rs (the local *_model.rs pull-worker pattern). Tests green.
The jepa embedder needs three matched files (encoder ONNX + tokenizer vocab + kNN
reference); distributing them as three independently-versioned artifacts risks
skew. This teaches the pull worker to accept a single gzip TAR bundle instead:
- detect a gzip magic prefix on the downloaded, sha256-verified artifact → treat
it as a {model.onnx, vocab.json, reference.json} bundle;
- unpack_bundle() extracts exactly those three names (no path traversal) next to
the promoted artifact and points the swap config at them, so vocab/reference
come from the verified bundle rather than local config;
- a bare ONNX still works unchanged (single-file path).
Because the whole set ships as one artifact with one version + one sha256, the
files can never skew. flate2 + tar added as classifier-scoped deps.
Tests (TDD): a gzip-bundle HTTP e2e with vocab_path/reference_path left EMPTY —
triage only becomes available if all three were unpacked from the bundle; the
existing single-file, sha-mismatch and no-op tests still pass.
Also gates the jepa_embedder field in the manual Config initializer under the
classifier feature (fixes a non-classifier build break from the earlier config
commit).
…in comment The jepa pull worker + app registration consume cortex::jepa_embedder, which isn't in the previously-pinned cortex rev. Point the [patch.gen0sec] cortex rev at cortex feat/jepa-forward-port (adds the module); re-point to the registry release once it lands there. Also correct a stale comment: the deferred SSE push event is jepa-embedder-updated, per the model-naming convention.
The Config struct literal in the upgrade-child path set jepa_embedder unconditionally, but the field is #[cfg(all(feature = "classifier", target_os = "linux"))] like its siblings. Under the default-feature workspace build (classifier off) the field is absent, so clippy/doc failed with E0560. Gate the initializer to match the four sibling classifier fields.
…ig load
The other cortex models (traffic/flow/penalty) get model_path, manifest_url and
refresh_secs defaulted in the classifier-config load path, but jepa_embedder was
skipped, so an enabled jepa embedder with no explicit paths never got a download
target or manifest and the pull worker/boot init had nothing to resolve.
Add the jepa defaults: model_path -> {data_dir}/jepa-embedder.tar.gz (the bundle
the worker unpacks), manifest_url -> {base}/models/jepa/manifest (the platform
aliases the agent route as models/jepa), refresh_secs -> 3600, and the unpacked
vocab_path -> {data_dir}/vocab.json for the tokenizer. reference_path stays empty
(triage optional). Test mirrors ml_flow_classifier_loads_and_applies_distribution_defaults.
jepa's model_path is the tar.gz bundle, so boot init loading it as ONNX always failed on restart and left the agent with no embedder until the pull worker's first poll. Add resolve_boot_config: when model_path is a gzip bundle it unpacks the members beside it (idempotent — skips if the encoder + vocab are already present from a prior pull) and repoints at the unpacked encoder/vocab/reference; a raw encoder path is returned unchanged; returns None when nothing is on disk yet (worker establishes it). synapse-app boot init uses it before load. TDD: resolve_boot_config_unpacks_bundle_to_encoder / _passes_through_raw_encoder / _is_none_when_model_absent.
…nfigs config/config.yaml listed traffic/flow/penalty under ml: but not jepa_embedder. Add it with its bundle-specific notes (model_path is the jepa-embedder.tar.gz bundle, vocab_path is the unpacked vocab, manifest defaults to <base_url>/models/jepa/manifest) and a matching commented stanza in the Windows example.
The jepa embedder was initialized + hot-swapped but never invoked on traffic. Wire it in at the kernel_pump full-JA4+-suite fan-out point (the same site as the traffic classifier, where all five views ja4/ja4h/ja4s/ja4x/ja4t are available — not the earlier JA4T-only enrich point): run kNN triage over the fingerprint embedding and record the nearest reference family + cosine similarity to a new per-source jepa_signals store (mirrors traffic_signals, the cortex -> amygdala wire). Alert-first, external-only (skips HOME_NET like the traffic classifier), and a no-op without a reference set. - security::jepa_signals: latest-verdict-per-source, idle-TTL'd, cortex-agnostic (label + 0..=100 similarity); tests for record/replace + clamp. - record_jepa_triage helper mirrors classify_and_record_traffic. Follow-up (cross-repo, same pending step as the traffic classifier): add ml.jepa.label / ml.jepa.similarity to the amygdala wirefilter scheme + read jepa_signals at the eval site so rules can fuse the embedding verdict.
Bump amygdala 0.1.9 -> 0.1.10 (adds the ml.jepa.{label,similarity} scheme fields
+ Enrichment::ml_jepa_*) and read the per-source jepa_signals verdict at the
kernel_pump eval site, populating the enrichment so a wirefilter rule can fuse
the embedding triage with fingerprint / IDS / threat signals, e.g.
`ml.jepa.label eq "qakbot" and ml.jepa.similarity ge 90`.
Completes the jepa data-plane path: embed+triage at the full-suite fan-out ->
jepa_signals store -> enrichment -> wirefilter eval. Mirrors the traffic
classifier's ml.traffic.* wire.
…patch cortex 0.1.3 (with jepa_embedder) is published to the gen0sec registry, so the [patch.gen0sec] git-rev pin is no longer needed. Bump synapse-core's cortex dep 0.1.2 -> 0.1.3 and remove the patch; cortex now resolves from the registry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added pull worker for the cortex flow classifier model