feat(ml): build Python ML pipeline against FeatureVector and ModelResult contract (#364)

.github/workflows/backend.yml

@@ -11,6 +11,7 @@ on:
       - "apps/backend/**"
       - "packages/**"
       - "tests/integration/**"
+      - "ml/**"
   pull_request:
     paths:
       - "package.json"
@@ -20,6 +21,7 @@ on:
       - "apps/backend/**"
       - "packages/**"
       - "tests/integration/**"
+      - "ml/**"
   workflow_dispatch:
 
 concurrency:
@@ -127,3 +129,41 @@ jobs:
       # so a missing build script must not red-X the whole pipeline.
       - name: Build packages
         run: pnpm --filter "./packages/**" run build --if-present || true
+
+  ml-pipeline:
+    name: Python ML Pipeline
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+        with:
+          version: 9.15.9
+
+      - name: Setup Node.js 22
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: pnpm
+
+      - name: Install JS dependencies and build chenai-mlflow
+        run: |
+          pnpm install --no-frozen-lockfile
+          pnpm --filter @chenaikit/chenai-mlflow run build
+
+      - name: Setup Python 3.11
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: "pip"
+
+      - name: Install Python dependencies
+        run: |
+          pip install -r ml/requirements.txt
+
+      - name: Run ML tests
+        run: |
+          PYTHONPATH=ml pytest ml/tests

ml/README.md

@@ -412,6 +412,66 @@ same change.
 ## Verifying the contract
 
 ```bash
+# Build TypeScript contract package
 pnpm --filter @chenaikit/chenai-mlflow run build   # tsc -p tsconfig.json
 pnpm --filter @chenaikit/chenai-mlflow run test    # vitest run
+
+# Run Python ML test suite (unit and cross-language contract verification)
+pytest ml/tests
+```
+
+## Python ML Pipeline
+
+### Architecture
+
+```
+Stellar Horizon API ──> ingest.py ──> features.py ──> contract.py (validate & impute)
+                                                            │
+                                                            ▼
+                                                        models.py (GBM / IForest)
+                                                            │
+                                                            ▼
+                                                        emitter.py (ModelResult + Soroban Attestation)
+                                                            │
+                                    ┌───────────────────────┼──────────────────────┐
+                                    ▼                       ▼                      ▼
+                            FileDropSink (.jsonl)     SqliteSink (DB)       StdoutSink (CLI)
+```
+
+### Components
+
+1. **Contract Layer (`contract.py`)**: Loads feature specifications dynamically from `packages/chenai-mlflow/src/index.ts`. Provides strict validation and missing-value imputation adhering to `FeatureVector` and `ModelResult` schemas.
+2. **Horizon Ingest (`ingest.py`)**: REST client for Stellar Horizon accounts, operations, payments, and transactions.
+3. **Feature Builder (`features.py`)**: Transforms raw Horizon ledger history into contract-compliant `FeatureVector` payloads.
+4. **Scoring Models (`models.py`)**:
+   - `CreditScoreModel`: Gradient Boosting classifier predicting default risk in `[0, 1]` with risk labels (`low`, `medium`, `high`).
+   - `FraudDetectModel`: Isolation Forest anomaly detector calibrated to anomaly risk in `[0, 1]`.
+   - Computes 32-byte SHA-256 artifact digests matching Soroban `BytesN<32>` in `contracts/model-attestation`.
+5. **ModelResult Emitter (`emitter.py`)**: Validates input vectors, performs inference, tracks `imputedFields`, and formats contract-compliant `ModelResult` records and Soroban attestation payloads.
+6. **Sinks (`sinks.py`)**:
+   - `FileDropSink`: Appends newline-delimited JSON (`.jsonl`) or writes atomic individual JSON files per `{subjectId}_{task}.json`.
+   - `SqliteSink`: Stores structured results in SQLite `model_results` table with indexed fields.
+   - `StdoutSink`: Streams newline-delimited JSON to stdout.
+7. **Training Pipeline (`train.py`)**: Reproducible training on `ml/data/sample_dataset.json` with evaluation reports (`ml/evaluation_report.py`) and artifact manifest (`ml/artifacts/manifest.json`).
+
+### Training Models
+
+```bash
+python3 ml/train.py
+```
+
+### Running Scoring Pipeline
+
+```bash
+# Score single account via Horizon testnet
+python3 ml/pipeline.py --mode single --subject-id GACC123 --task all --sink stdout
+
+# Score single FeatureVector from JSON file
+python3 ml/pipeline.py --mode single --input-file vector.json --task credit-score --sink stdout
+
+# Batch score accounts from JSONL file into SQLite database
+python3 ml/pipeline.py --mode batch --input-file ml/data/sample_vectors.jsonl --sink sqlite --output ml/output/scores.db
+
+# Batch score accounts into newline-delimited JSON
+python3 ml/pipeline.py --mode batch --input-file ml/data/sample_vectors.jsonl --sink file --output ml/output/scores.jsonl
 ```

ml/artifacts/credit_score_report/evaluation_report.json

@@ -0,0 +1 @@
+{"code_version":"1.0.0","dataset":{"identifier":"sample_dataset.json","version":"1.0.0"},"evaluation":{"config_id":"eval-stratified-split","params":{"seed":42,"test_size":0.25}},"generated_at":"2026-09-08T21:18:22Z","metrics":{"accuracy":1.0,"f1_score":1.0,"precision":1.0,"recall":1.0,"roc_auc":1.0},"model":{"name":"credit-score-gbm","version":"1.0.0"},"report_id":"er-dfcce5a2c0283f5e","schema_version":"1.0"}

ml/artifacts/credit_score_report/metrics.json

@@ -0,0 +1 @@
+{"accuracy":1.0,"f1_score":1.0,"precision":1.0,"recall":1.0,"roc_auc":1.0}

ml/artifacts/fraud_detect_report/evaluation_report.json

@@ -0,0 +1 @@
+{"code_version":"1.0.0","dataset":{"identifier":"sample_dataset.json","version":"1.0.0"},"evaluation":{"config_id":"eval-isolation-forest","params":{"contamination":0.1,"seed":42}},"generated_at":"2026-09-08T21:18:22Z","metrics":{"anomaly_rate":0.08,"mean_risk_score":0.4296},"model":{"name":"fraud-detect-iforest","version":"1.0.0"},"report_id":"er-ca764cedf3c9342e","schema_version":"1.0"}

ml/artifacts/fraud_detect_report/metrics.json

@@ -0,0 +1 @@
+{"anomaly_rate":0.08,"mean_risk_score":0.4296}

ml/artifacts/manifest.json

@@ -0,0 +1,20 @@
+{
+  "models": {
+    "credit-score-gbm": {
+      "version": "1.0.0",
+      "artifact": "credit_score_gbm.joblib",
+      "artifact_hash_sha256": "a7108f71280a656e101518cac9257fc3106f84707937d8252f38b2e1f69d53fd",
+      "report_id": "er-dfcce5a2c0283f5e",
+      "attestation_bytes32": "a7108f71280a656e101518cac9257fc3106f84707937d8252f38b2e1f69d53fd"
+    },
+    "fraud-detect-iforest": {
+      "version": "1.0.0",
+      "artifact": "fraud_detect_iforest.joblib",
+      "artifact_hash_sha256": "5d3e9e037b4d32883db2e4e85c8c71ac5e74aff40f5d833de154da7f256a9e76",
+      "report_id": "er-ca764cedf3c9342e",
+      "attestation_bytes32": "5d3e9e037b4d32883db2e4e85c8c71ac5e74aff40f5d833de154da7f256a9e76"
+    }
+  },
+  "random_state": 42,
+  "dataset": "sample_dataset.json"
+}