HackerRank Orchestrate — Message Notification Router

Starter repository for the HackerRank Orchestrate 24-hour hackathon.

The system is a deterministic, explainable router for WhatsApp: for every multimodal incoming message (text, image poster/screenshot, voice note) it decides notify (interrupt now), digest (wait), or mute (suppress as low-value, repetitive, unwanted, suspicious, or unsafe), personalized per user/group/business using historical interaction data.

Read problem_statement.md for the full task spec, input/output schema, allowed values, and submission format.

---

Repository Layout

.
├── AGENTS.md                         # Rules for AI coding tools + transcript logging
├── problem_statement.md              # Full challenge statement
├── README.md                         # You are here
├── requirements.txt                  # Runtime dependencies (core = stdlib only)
├── package.py                        # Builds the submission code.zip
├── code/
│   ├── main.py                       # Entry point (also: --debug traces)
│   ├── loader.py                     # Dataset loading + schema validation
│   ├── retrieval.py                  # Personalized evidence retrieval
│   ├── signals.py                    # Per-message signal computation
│   ├── safety.py                     # Deterministic safety triage
│   ├── classify.py                   # Message-type classification
│   ├── router.py                     # Routing engine (precedence rules)
│   ├── confidence.py                 # Confidence calibration
│   ├── reasons.py                    # Explainability + evidence selection
│   ├── media.py                      # OCR/ASR media understanding (optional)
│   ├── writer.py                     # output.csv writer + contract checks
│   ├── models.py, config.py          # Dataclasses and configuration
│   ├── evaluation/main.py            # M8 evaluation harness (guardrail)
│   └── tests/                        # 260 unit + integration tests
└── dataset/                          # Organizer-provided data (not packaged)

---

Quick Start

Requires Python 3.10+. The core pipeline has no third-party runtime dependencies.

# 1. Route every message and write output.csv
python code/main.py

# 2. Run the full evaluation/validation harness (guardrail)
python code/evaluation/main.py

# 3. Run the test suite
python -m unittest discover -s code/tests -p "test_*.py"

Both commands must pass for a valid release. The harness verifies schema conformance, regression checks, confidence bounds, evidence validity, and byte-identical determinism.

Options

Command	Purpose
python code/main.py --debug	Print a per-message execution trace (media quality/warnings, action, type, confidence, evidence) plus loader warnings and elapsed time. Read-only diagnostics; does not change output.csv.
python code/main.py --dataset-dir <path>	Point at a non-default dataset directory.
python code/evaluation/main.py --report <path.json>	Write the structured evaluation report as JSON.
python code/evaluation/main.py --no-sweep	Skip the (report-only) threshold calibration sweep.

---

Architecture

Strictly layered, one-way pipeline. Each stage is a pure, deterministic function of its inputs; no stage inspects or mutates another's internals.

loader -> retrieval -> media -> signals -> safety -> classify -> router
                                                          -> confidence -> reasons -> writer

1. Loader validates every dataset file and builds an immutable Dataset.
2. Retriever finds personalized evidence from message_history for the target user/group/business, ranked by content similarity and behavioral recency, honoring similarity_floor.
3. Media (media.py) optionally runs OCR (images) / ASR (voice). When tools are absent, files are missing, or content cannot be decoded, it returns a degraded MediaResult with structured warnings and never crashes.
4. Signals compute urgency, trust, personalization, engagement, safety, spam likelihood, fatigue, media understanding, business relationship and group importance, plus boolean flags (direct ask, DND, muted group, OTP credential requests, injection attempts, opted-out, repetition).
5. Safety is a priority-ordered rule engine that mutes scams, phishing, fraud payment demands, credential requests, injections, spam and unsafe links — with corroboration.
6. Classifier assigns one of 11 message types, honoring safety overrides and high-confidence media hints.
7. Router applies fixed precedence rules (safety gate > injection gate > DND > muted group > value-vs-cost decision with a notify/digest threshold) and produces an Action.
8. Confidence calibrates a 0.6–0.95 score from signal strength, decision margin, evidence corroboration and media understanding; degraded media caps confidence.
9. Reasons generate human-readable explanations that reference only actually-fired signals/rules, and selects up to two evidence message IDs (or none).
10. Writer enforces the exact output contract and rejects malformed rows.

Determinism

Every stage is deterministic. The evaluation harness runs the pipeline twice and asserts byte-identical output.csv (SHA-256) plus identical decisions; the M9-era production hash is 506ef225…4082. Nothing reads wall-clock time or randomness into predictions.

Media Understanding (optional)

• OCR: Tesseract via pytesseract + Pillow. Detects posters (promotion hint), notices/circulars (business_update hint) and screenshots, computes quality, and normalizes extracted text.
• ASR: faster-whisper (or openai-whisper), with ffprobe or a dependency-free MP3 frame-walk for duration estimation; flags empty/noisy audio and computes quality.
• Enable with MEDIA_TOOLING=true in the environment (default is the degraded, deterministic path that this repo ships and validates).
• Results are cached by file path + mtime + config version + tool signature, so repeated runs are byte-stable.

See requirements.txt for optional installs.

---

Output Contract

output.csv — exactly one row per message_id in dataset/messages.csv:

message_id,action,message_type,reason,confidence,evidence_message_ids

• action: notify | digest | mute
• message_type: one of personal, urgent, event, payment, business_update, promotion, greeting, forward, spam, scam, unknown
• confidence: number in [0.6, 0.95]
• evidence_message_ids: up to two history IDs separated by ;, or none

Secrets are read from environment variables only; none are stored in the repo.

---

Known Limitations and Assumptions

• Media tooling is optional. The validated shipped behavior is the degraded media path (quality 0.0). Enabling OCR/ASR requires the tools above and is not exercised in the default guardrail.
• Confidence is calibrated on 30 coarse sample labels (MAE 0.0464, action accuracy 0.7667); it is a relative signal, not a probability guarantee on unseen messages.
• theta_digest is reserved configuration — the router currently references only theta_notify; it is reported, not tuned.
• Dependency-free duration estimation covers MP3; other audio formats need ffprobe.
• Media type hints (poster/notice/screenshot) are keyword+geometry heuristics and may mislabel ambiguous images; they only fire at high quality.
• Sample labels include four documented overrides (sample_msg_015/042/045/047) where the authored sample disagrees with the classifier's deterministic rules.

---

Submission

1. Code zip — run python package.py to build code.zip (source, README, requirements, evaluation harness, tests; dataset/, venvs, caches, and build artifacts excluded).
2. Predictions CSV — the output.csv regenerated by python code/main.py.
3. Chat transcript — the log.txt described in AGENTS.md.

Before submitting, confirm the guardrail (code/main.py then code/evaluation/main.py) both pass and that output.csv has one row per input message with the exact required columns.