Feat/model service split - #2390
Closed
ahmedali-80 wants to merge 1389 commits into
Closed
Conversation
[skip ci]
[skip ci]
[skip ci]
[skip ci]
[skip ci]
Bumps [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) from 0.556.0 to 0.562.0. - [Release notes](https://github.com/lucide-icons/lucide/releases) - [Commits](https://github.com/lucide-icons/lucide/commits/0.562.0/packages/lucide-react) --- updated-dependencies: - dependency-name: lucide-react dependency-version: 0.562.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[skip ci]
[skip ci]
Co-authored-by: Motov Yurii <109919500+YuriiMotov@users.noreply.github.com>
[skip ci]
[skip ci]
Co-authored-by: Yurii Motov <yurii.motov.monte@gmail.com>
[skip ci]
[skip ci]
[skip ci]
[skip ci]
…update old passwords using Bcrypt (fastapi#2104) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
[skip ci]
…tacks (fastapi#2105) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
[skip ci]
Bumps traefik from 3.0 to 3.6. --- updated-dependencies: - dependency-name: traefik dependency-version: '3.6' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[skip ci]
[skip ci]
[skip ci]
…tapi#2333) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yurii Motov <yurii.motov.monte@gmail.com> Co-authored-by: Yurii Motov <109919500+YuriiMotov@users.noreply.github.com> Co-authored-by: Alejandra <alejsdev@gmail.com>
[skip ci]
[skip ci]
[skip ci]
…stapi#2379) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[skip ci]
…3.0,<3.0.0 (fastapi#2373) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[skip ci]
…astapi#2380) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[skip ci]
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
[skip ci]
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yurii Motov <yurii.motov.monte@gmail.com>
[skip ci]
[skip ci]
Every torch-backed model now runs in its own container under services/ and is reached over HTTP. The backend keeps its API surface, swagger, auth, faiss and TMX generation, and loads no weights at all. labse LaBSE embeddings + similarity qe CometKiwi (reference-free QE) comet wmt22-comet-da (reference-based) -- same image as qe metricx MetricX-24 hybrid langid lingua restricted to en+fr -- replaces fastText qe and comet share one image: their dependency trees resolve identically, so separate images would duplicate rather than isolate. backend image 7.39 GB -> 1.57 GB backend lock 145 -> 101 packages startup to healthy ~2 min -> ~2 s build context 11 GB -> nil (aimodels now ignored) Parity verified against the in-process models before each cutover: CometKiwi 0.00e+00 bit-identical wmt22-comet-da 0.00e+00 bit-identical LaBSE 4.38e-05 the backend's own 4dp rounding MetricX 5.86e-03 ~3 bf16 ulps on a 0-25 scale Stopping any one service returns 503 + Retry-After on only its own route; everything else keeps serving, and the route recovers unattended. Dependency notes, each found by running it rather than reasoning about it: - setuptools pinned <81 in comet-svc. unbabel-comet pins torchmetrics 0.10.3, which imports pkg_resources at module scope; setuptools 81 removed it. The backend only ever survived this because its lock froze setuptools at 80.7.1. - transformers pinned ==4.51.3 in metricx-svc. 5.x drops a symbol MT5ForRegression needs; 4.57.6 breaks the MT5 attention mask. labse-svc runs 5.14.1 quite happily -- which is the whole argument for per-service locks. - metricx-svc installs gcc: torch's triton backend JIT-compiles its CUDA utils at runtime and the slim base ships no compiler. - sacrebleu declared explicitly; it had been arriving via unbabel-comet. - nptyping kept despite looking unused: hlepor imports it without declaring it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings 7 months of template changes: 5 security fixes, ~70 dependency bumps, the uv workspace restructure, the models refactor and Python 3.14. Structural changes taken from upstream: - uv workspace: root pyproject.toml + root uv.lock; backend/uv.lock removed and the Docker build context moved to the repo root. - Python 3.10 -> 3.14. numpy 1.26.4 -> 2.5.1, faiss-cpu -> 1.14.3. - passlib/bcrypt -> pwdlib[argon2,bcrypt]. - docker-compose*.yml -> compose*.yml (git tracked these as renames, so the five model services carried over intact). - postgres 17 -> 18. The local data volume was initialised by 17 and PG18 refuses to start on it, so app-db-data was dropped and recreated. Other environments still need a real pg_upgrade or dump/restore. Local decisions: - Took upstream's models.py and dropped the local app/models/ package. The class sets were identical, so nothing was lost, and upstream's version adds created_at/get_datetime_utc. The 7 `from app.models.users import` lines are back to `from app.models import`. - Vendored hlepor into app/models_ml/hlepor_vendored.py. The published package imports nptyping, which is unmaintained and hard-caps numpy <2 -- which blocked Python 3.14 for the whole backend. nptyping's NDArray was only ever used in annotations, and PEP 563 meant they were never evaluated, so the import was removed and the annotations now say np.ndarray. hlepor, lepor and nptyping are gone from the dependency list. Verified the metric still discriminates: 1.000 identical, 0.811 near, 0.310 unrelated. Two things the merge would otherwise have broken silently: - sacre_bleu.py loaded libED.so from the hardcoded /app/app/lib/. The workspace layout moves the app to /app/backend/app, so characTER would have failed at import. Now resolved relative to __file__. - No root .dockerignore existed. With the build context at the repo root, the ~11 GB of model weights under backend/aimodels would ship to the daemon on every build. Added. Verified after merge: 60/60 backend tests pass, all 9 metric routes 200, /memory/align and /memory/align-uni 15/15 correct, auth round trip works on pwdlib, and /users/me returns the new created_at field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
labse-svc, metricx-svc and langid-svc move from python:3.11-slim-bookworm to python:3.14-slim-trixie. comet-svc moves to python:3.12-slim-trixie. comet-svc cannot reach 3.14: unbabel-comet pins numpy<2.0.0, and the last numpy 1.x (1.26.4) publishes no wheel past cp312. On 3.13+ the resolver falls back to the sdist and the meson build fails looking for a C compiler; adding one would not help, since numpy 1.26 does not build against the 3.14 C API. The cap is declared in both the Dockerfile and requires-python so `uv lock` cannot resolve something the image could never install. This is the split doing its job. Under one shared lock, unbabel-comet would hold every service — and the backend — at numpy 1.x and Python 3.12. Also: - uv 0.7.5 -> 0.9.26 in all four Dockerfiles. 0.7.5 predates 3.14 and does not resolve cp314 wheels. - Locks regenerated; ~5000 lines lighter, as the raised requires-python floor drops the 3.10-3.13 resolution forks. - lingua 2.1.1 -> 2.2.0 (2.1.1 stops at cp313, so 3.14 forces it). Its bundled n-gram data grew to 292 MB, taking langid-svc from 338 MB to 550 MB. Disk only — it still preloads just en and fr. Documented with the way back. - Base image CVEs: 1 critical + 4 high -> 1 critical + 2 high. The remaining critical is unfixed upstream in Debian. Verified on the rebuilt images: all five services reach /ready inside 40 s and report the intended interpreter (labse/metricx/langid 3.14.6 + numpy 2.5.1, qe/comet 3.12.13 + numpy 1.26.4). All 13 backend routes return 200 with plausible scores, and alignment recovers 12/12 pairs from a shuffled EN/FR corpus. metricx exercises the CUDA forward pass, so triton still JIT-compiles on 3.14. The CPU torch variant also builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/v1/embed had exactly two callers, both inside align_sentences. The backend
pulled every embedding over HTTP — ~4 KB per sentence after base64, so 820 MB
on a 100k+100k corpus — purely to hand them to a local faiss index, then threw
them away. It never kept a vector for anything else.
The vectors are produced in GPU memory a few lines from where the kNN runs, so
the search now lives there too: new POST /v1/align takes text and returns
{src_idx, trg_idx, score} triples. Indices, not text — the caller already holds
the strings, and echoing them would re-inflate the payload this exists to
shrink.
Moved into services/labse/app/aligner.py: normalization, bidirectional faiss
kNN, ratio-margin scoring, greedy one-to-one selection. Left in the backend:
the 10-200 char filter, langid routing for align-uni, TMX, response shaping.
Backend drops faiss-cpu (1.66 GB -> 1.59 GB, lock 101 -> 99 packages). numpy
stays, but only for the vendored hLEPOR, which is array maths throughout —
nothing else in the app touches it now. labse-svc grows 5.51 -> 5.58 GB.
Two deliberate changes to the moved code:
- score_candidates was a Python double loop doing one 768-dim dot per
iteration; it is now a gather plus einsum. Verified against a verbatim
transcription of the original inside the built image: max |delta| 6.0e-07
(float32 reassociation) and *identical* selected pairs at n=30, 200 and 2000
random unit vectors, which tie far more often than real bitext.
- k is clamped to the corpus size. faiss pads with -1 when k exceeds the index
and a -1 index silently wraps to the last row, so a corpus smaller than k
scored against garbage and fell below threshold. /memory/align on two
sentences returned nothing; it now returns the correct pairs. This is the
only behavioural difference, and only for corpora smaller than k.
/v1/embed stays on the service. It has no caller here now, but the raw
primitive is the reusable one and the next consumer may not want alignment.
The search stays on CPU via faiss-cpu even though the vectors are in VRAM: an
index sized for a large corpus would compete with the model for GPU memory, and
that trade is unmeasured. Moving it to the GPU is a separate decision.
Verified: alignment output byte-identical to the pre-move baseline on a
shuffled 30-pair EN/FR corpus, via both /memory/align and /memory/align-uni,
30/30 correct each; metrics/labse unchanged; all 13 routes 200; 60/60 backend
tests pass; faiss confirmed absent from the backend image.
Also: metricx.py now uses the mean the client already computes, which returns
0.0 for an empty batch where np.mean produced a NaN that is not valid JSON.
Two lint errors surfaced by the py314 ruff target fixed (UP037, B905).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
align_sentences_special constructed OpenAI(api_key="sk-proj-...") from a literal committed in 4d4d3e1. It now reads settings.OPENAI_API_KEY and raises if unset, so a missing key fails at the call site instead of a secret living in the tree. .env carries an empty placeholder. That file is committed, so the comment says plainly not to put a real key in it. The literal is purged from history in the following step -- this branch has never been pushed, so the rewrite costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
This PR modifies dependency files ( If you need a dependency change, please open a discussion describing what you need and why. Closing this PR automatically. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Discussion:
Description
AI Disclaimer
AI transcript
Checklist