Add standalone Python 3 / TF 2.15 port (no ROS) + fixed stale weights download URL

.gitignore

@@ -0,0 +1,13 @@
+# Python virtual environments
+.venv/
+standalone_demo/.venv/
+
+# Python caches
+__pycache__/
+*.py[cod]
+
+# Model weights (large; obtained via download, not committed)
+models/*.h5
+
+# macOS
+.DS_Store

download_model.sh

@@ -1,5 +1,5 @@
 #!/bin/sh
 mkdir -p models
 
-wget -O models/lstm_10_50_runsigm_runsigm.h5 'http://lcas.lincoln.ac.uk/owncloud/index.php/s/UHWU1g5qXU9RiAp/download'
+wget -O models/lstm_10_50_runsigm_runsigm.h5 'https://lcas.lincoln.ac.uk/nextcloud/index.php/s/UHWU1g5qXU9RiAp/download'
 

standalone_demo/README.md

@@ -0,0 +1,74 @@
+# Standalone engagement demo (Python 3, no ROS)
+
+A modernised, ROS-free port of the LCAS `engagement_detector` (Del Duchetto,
+Baxter & Hanheide 2020), so it runs on a current Mac (Apple Silicon, Python 3.11,
+TensorFlow 2.15) instead of the original Python 2 / TF 1.14 / ROS stack.
+
+Pipeline: `10 frames -> ResNeXt50 (2048-d features/frame) -> LSTM -> engagement score in [0,1]`.
+
+## Files
+- `engagement.py` — the reusable `EngagementDetector` class (feed it numpy frames).
+- `webcam_demo.py` — live webcam demo with an on-screen score overlay.
+- `test_resnext.py` — checks just the ResNeXt50 feature extractor.
+- `requirements.txt` — exact pins for the fragile TF/Keras combo (see comments inside).
+- `engagement_service.py` — localhost HTTP service exposing the detector (see below).
+- `run_service.sh` — launcher for the service (uses the local venv, works from any cwd).
+- `test_service.py` — round-trip regression test for the service.
+- `.venv/` — the isolated Python 3.11 environment (not committed; recreate as below).
+
+## Setup
+
+Requires **Python 3.11** (tensorflow-macos 2.15 has no 3.12 wheels):
+
+```bash
+# from this folder:
+python3.11 -m venv .venv
+.venv/bin/pip install -r requirements.txt
+```
+
+On **Linux**, replace `tensorflow-macos==2.15.0` in `requirements.txt` with
+`tensorflow==2.15.0` (same version, different distribution name).
+
+## Model weights (required for meaningful output)
+The pretrained LSTM `lstm_10_50_runsigm_runsigm.h5` (~256 MB) must sit in
+`../models/`. Download it with the repo's `download_model.sh`, or directly:
+
+```bash
+curl -L -o ../models/lstm_10_50_runsigm_runsigm.h5 \
+  'https://lcas.lincoln.ac.uk/nextcloud/index.php/s/UHWU1g5qXU9RiAp/download'
+```
+
+(The weights are not committed to git — they are the authors' artefact, served
+from their host.) **Without the file, the demo still runs but falls back to a
+random stand-in LSTM and the scores are not meaningful** (the overlay says so).
+
+## Run it
+
+```bash
+# from this folder:
+.venv/bin/python webcam_demo.py            # live mode, press q to quit
+.venv/bin/python webcam_demo.py --selftest # headless: print a few scores and exit
+```
+
+On macOS the first run triggers a **camera-permission prompt** for your terminal
+(System Settings > Privacy & Security > Camera). Allow it, then re-run.
+
+The first run also downloads the ResNeXt50 ImageNet weights (~90 MB) once.
+
+## Score service (for consuming apps)
+
+`engagement_service.py` exposes the detector to other processes (e.g. apps on a
+different Python version) over localhost HTTP:
+
+```bash
+./run_service.sh                # serves http://127.0.0.1:8100 once models load
+./run_service.sh --port 8200    # alternative port
+```
+
+- `GET /health` → `{"status": "ok"}` (answers only once the model is ready)
+- `POST /score` with `{"frames": [<base64 JPEG> x10, chronological]}` →
+  `{"score": <float 0..1>}`; frames are decoded as BGR (OpenCV convention)
+- Refuses to start without the real weights unless `--allow-standin` (test only)
+
+Regression test: `.venv/bin/python test_service.py` (starts its own instance
+on port 8199, no camera needed).

standalone_demo/engagement.py

@@ -0,0 +1,108 @@
+"""
+Standalone, ROS-free port of the LCAS engagement_detector core.
+
+Pipeline (unchanged from the original paper / repo):
+    10 frames  ->  ResNeXt50 (per-frame 2048-d features)  ->  LSTM  ->  scalar in [0,1]
+
+The only differences from src/engagement_detector/engagement_detector.py:
+  * Python 3 + TF 2.15 (tf.keras) instead of Python 2 + TF 1.14 + classic Keras.
+  * No ROS. You feed it plain numpy frames.
+  * If the pretrained LSTM weights are missing, it falls back to a randomly
+    initialised stand-in LSTM of the same shape, so the pipeline still runs
+    (predictions are meaningless until the real weights are dropped in).
+
+Reference: Del Duchetto, Baxter & Hanheide (2020), "Are You Still With Me?
+Continuous Engagement Assessment From a Robot's Point of View."
+"""
+import os
+os.environ.setdefault("TF_CPP_MIN_LOG_LEVEL", "3")
+
+import numpy as np
+from PIL import Image
+import tensorflow as tf
+from keras_applications.resnext import ResNeXt50, preprocess_input
+
+WINDOW_SIZE = 10          # the model consumes exactly 10 frames
+FEATURE_DIM = 2048        # ResNeXt50(pooling='max') output width
+DEFAULT_WEIGHTS = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.realpath(__file__))),
+    "models", "lstm_10_50_runsigm_runsigm.h5",
+)
+
+# tf.keras module set, injected into the backend-agnostic keras_applications.
+_KA_KWARGS = dict(
+    backend=tf.keras.backend,
+    layers=tf.keras.layers,
+    models=tf.keras.models,
+    utils=tf.keras.utils,
+)
+
+
+class EngagementDetector:
+    def __init__(self, weights_path=DEFAULT_WEIGHTS):
+        self.window_size = WINDOW_SIZE
+        self.using_real_weights = False
+
+        # Front half: ResNeXt50 feature extractor (ImageNet weights).
+        self.resnet = ResNeXt50(
+            include_top=False,
+            input_shape=(224, 224, 3),
+            pooling="max",
+            weights="imagenet",
+            **_KA_KWARGS,
+        )
+
+        # Back half: the engagement LSTM.
+        if weights_path and os.path.exists(weights_path):
+            self.lstm = tf.keras.models.load_model(weights_path, compile=False)
+            self.using_real_weights = True
+            print("[EngagementDetector] Loaded pretrained LSTM:", weights_path)
+        else:
+            self.lstm = self._build_standin_lstm()
+            print("[EngagementDetector] WARNING: pretrained weights not found at")
+            print("   ", weights_path)
+            print("    Using a RANDOM stand-in LSTM. Predictions are NOT meaningful")
+            print("    until the real lstm_10_50_runsigm_runsigm.h5 is in place.")
+
+    def _build_standin_lstm(self):
+        """A shape-compatible placeholder: (10, 2048) -> scalar in [0,1].
+
+        Matches the original naming hints (10 frames, ~50 units, sigmoid output)
+        so the surrounding code path is exercised exactly as it will be for real.
+        """
+        model = tf.keras.Sequential([
+            tf.keras.layers.Input(shape=(WINDOW_SIZE, FEATURE_DIM)),
+            tf.keras.layers.LSTM(50, recurrent_activation="sigmoid"),
+            tf.keras.layers.Dense(1, activation="sigmoid"),
+        ])
+        return model
+
+    def predict(self, frames_seq):
+        """frames_seq: list of exactly WINDOW_SIZE HxWx3 uint8 frames.
+
+        Returns a float engagement score in [0,1], or None if the window is
+        not yet full.
+        """
+        if len(frames_seq) != self.window_size:
+            return None
+
+        # Resize each frame to 224x224 (PIL, same as the original) and preprocess.
+        frames = [np.array(Image.fromarray(f).resize((224, 224))) for f in frames_seq]
+        batch = preprocess_input(np.array(frames, dtype="float32"), **_KA_KWARGS)
+
+        # Per-frame features -> (10, 2048), then LSTM over the sequence.
+        features = self.resnet.predict_on_batch(batch)          # (10, 2048)
+        prediction = self.lstm.predict(features[np.newaxis, :], verbose=0)  # (1, 1)
+        return float(np.squeeze(prediction))
+
+
+if __name__ == "__main__":
+    # Smoke test with dummy frames: proves the full front-to-back path runs.
+    det = EngagementDetector()
+    dummy_window = [
+        np.random.randint(0, 256, size=(480, 640, 3), dtype="uint8")
+        for _ in range(WINDOW_SIZE)
+    ]
+    score = det.predict(dummy_window)
+    print("Engagement score on random frames:", score)
+    print("Using real weights:", det.using_real_weights)

standalone_demo/engagement_service.py

@@ -0,0 +1,131 @@
+"""
+Localhost HTTP service wrapping EngagementDetector (frames in -> score out).
+
+Contract (consumed by the thesis app, reachy_mini_conversation_app):
+  POST /score   body: {"frames": ["<base64 JPEG>", ... exactly 10, chronological]}
+                -> 200 {"score": <float 0..1>}
+                -> 400 {"error": "..."} on bad input
+  GET  /health  -> 200 {"status": "ok"}
+
+Design notes:
+  * The model (ResNeXt50 + LSTM) is loaded ONCE, before the socket opens.
+    /health answering at all therefore implies the model is ready; the
+    contract's 503 "still loading" state cannot occur by construction.
+  * Frames are decoded with cv2.imdecode -> BGR, matching exactly what the
+    validated webcam demo feeds EngagementDetector.predict(). Do NOT switch
+    to PIL (RGB): it would silently flip channels vs. the validated setup.
+  * stdlib http.server handles requests serially -- intentional; predict()
+    is CPU-heavy and the contract requires no concurrency.
+  * Binds 127.0.0.1 only. Default port 8100 (7860/8765 taken by thesis app).
+  * Refuses to serve stand-in (random-LSTM) scores unless --allow-standin.
+"""
+import argparse
+import base64
+import json
+import sys
+import time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+import cv2
+import numpy as np
+
+from engagement import EngagementDetector, WINDOW_SIZE
+
+DETECTOR = None  # set in main() before the server starts
+
+
+def decode_frame(b64_string):
+    """base64 JPEG -> HxWx3 uint8 BGR array, or None if undecodable."""
+    try:
+        raw = base64.b64decode(b64_string, validate=True)
+    except Exception:
+        return None
+    return cv2.imdecode(np.frombuffer(raw, np.uint8), cv2.IMREAD_COLOR)
+
+
+class Handler(BaseHTTPRequestHandler):
+
+    def _send_json(self, status, payload):
+        body = json.dumps(payload).encode("utf-8")
+        self.send_response(status)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, fmt, *args):
+        pass  # silence per-request stderr noise; we log our own line in do_POST
+
+    def do_GET(self):
+        if self.path == "/health":
+            self._send_json(200, {"status": "ok"})
+        else:
+            self._send_json(404, {"error": "unknown path %s" % self.path})
+
+    def do_POST(self):
+        if self.path != "/score":
+            self._send_json(404, {"error": "unknown path %s" % self.path})
+            return
+
+        t0 = time.time()
+        try:
+            length = int(self.headers.get("Content-Length", 0))
+            payload = json.loads(self.rfile.read(length))
+        except Exception:
+            self._send_json(400, {"error": "body is not valid JSON"})
+            return
+
+        frames_b64 = payload.get("frames")
+        if not isinstance(frames_b64, list) or len(frames_b64) != WINDOW_SIZE:
+            self._send_json(400, {
+                "error": "expected 'frames' as a list of exactly %d base64 JPEG strings"
+                         % WINDOW_SIZE})
+            return
+
+        frames = []
+        for i, s in enumerate(frames_b64):
+            img = decode_frame(s)
+            if img is None:
+                self._send_json(400, {"error": "frame %d could not be decoded" % i})
+                return
+            frames.append(img)
+
+        score = DETECTOR.predict(frames)
+        latency_ms = (time.time() - t0) * 1000.0
+        print("[score] frames=%d score=%.4f latency=%.0fms"
+              % (len(frames), score, latency_ms), flush=True)
+        self._send_json(200, {"score": score})
+
+
+def main():
+    global DETECTOR
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=8100)
+    ap.add_argument("--host", default="127.0.0.1",
+                    help="bind address; default localhost-only. Use 0.0.0.0 to "
+                         "serve LAN clients (e.g. the app running on the robot's Pi)")
+    ap.add_argument("--allow-standin", action="store_true",
+                    help="serve even without the real .h5 (testing only)")
+    args = ap.parse_args()
+
+    print("[startup] loading models (this takes a while)...", flush=True)
+    DETECTOR = EngagementDetector()
+    if not DETECTOR.using_real_weights:
+        if not args.allow_standin:
+            print("[startup] FATAL: real weights missing; refusing to serve "
+                  "meaningless scores. Use --allow-standin to override for "
+                  "plumbing tests.", flush=True)
+            sys.exit(1)
+        print("[startup] WARNING: serving STAND-IN scores (--allow-standin).", flush=True)
+
+    server = HTTPServer((args.host, args.port), Handler)
+    print("[startup] ready on http://%s:%d (GET /health, POST /score)"
+          % (args.host, args.port), flush=True)
+    try:
+        server.serve_forever()
+    except KeyboardInterrupt:
+        print("\n[shutdown] bye", flush=True)
+
+
+if __name__ == "__main__":
+    main()

standalone_demo/requirements.txt

@@ -0,0 +1,13 @@
+# Environment for the Python 3 / ROS-free port of the engagement detector.
+# REQUIRES Python 3.11 (tensorflow-macos 2.15 has no 3.12 wheels).
+# These pins are load-bearing — see README. Do not upgrade casually:
+#  - tensorflow-macos 2.15 is the LAST version bundling Keras 2; the pretrained
+#    lstm_10_50_runsigm_runsigm.h5 does not load under Keras 3.
+#  - Keras-Applications 1.0.8 provides ResNeXt50, which modern TF dropped.
+#  - numpy <2 required by TF 2.15.
+tensorflow-macos==2.15.0
+Keras-Applications==1.0.8
+numpy==1.26.4
+opencv-python==4.11.0.86
+pillow==12.3.0
+h5py==3.16.0

standalone_demo/run_service.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+# Launch the engagement scoring service (see engagement_service.py for the API).
+# Works from any cwd; extra args pass through (e.g. ./run_service.sh --port 8200).
+cd "$(dirname "$0")" || exit 1
+exec .venv/bin/python engagement_service.py "$@"