merge: land PRs 3, 4, 5, 6 and 9 of the v7 audit stack in working_branch

CHANGELOG.md

@@ -11,6 +11,46 @@
 - OpenCode edits are no longer gated either: the OpenCode plugin runs the same `pre_edit` hook, which now always allows the edit. The plugin does not yet show the reminder; that rework comes with the edit-bridge removal.
 - `install-claude-hooks` and `uninstall-claude-hooks` refuse to touch a `~/.claude/settings.json` that is not valid JSON, instead of replacing it.
 - The manual `hooks/daem0n_*.py` scripts are deprecated stubs that print a notice and exit 0, and `hooks/settings.json.example` is removed. If you installed them by hand, re-run `python -m daem0nmcp.cli install-claude-hooks`; it also removes the old entries.
+- **Behaviour change — installed profiles now turn themselves on.** An unset (or
+  empty) `DAEM0NMCP_<PROFILE>_ENABLED` means "auto": the profile is ready when
+  its extra is installed and disabled otherwise. `false` still turns it off, and
+  `true` with packages missing is still degraded. An empty value used to mean
+  `false`. What this costs if you installed an extra and never set the flag:
+  `[graph]` imports networkx, igraph and leidenalg at startup, adds a graph
+  provider to recall (so rankings change) and lets dreaming refresh communities;
+  `[local]`/`[models-local]` import sentence-transformers and onnxruntime at
+  startup, download the embedding model on first use, and rebuild the dense
+  projection, which re-embeds the whole workspace once. Opt out per profile with
+  `DAEM0NMCP_<PROFILE>_ENABLED=false`.
+- `sandbox_execute_python` now honours `DAEM0NMCP_AGENCY_E2B_ENABLED=false`; it
+  previously required only `E2B_API_KEY`.
+- `system_health` and `meta.capability_states` report the profile's real
+  remediation (install command or environment variable) instead of "Review the
+  &lt;name&gt; capability profile."
+- `start_daem0nmcp_server.bat` serves the directory you run it from (or its first
+  argument, or an exported `DAEM0NMCP_PROJECT_ROOT`) instead of the Daem0n-MCP
+  checkout, and `install-opencode` installs the four `.opencode/commands/` files.
+- **Migration payload change — re-migrate a store migrated by an earlier
+  v7 development build.** `migrate-v7` no longer copies a v6 host path into the
+  event it writes: the columns that hold one (`file_path`, `project_path`,
+  `database_path`, and a `file_path_relative` that is not workspace-relative)
+  are recorded as a `{"$host_path": {...}}` digest, so an export bundle from a
+  migrated workspace no longer discloses the origin host's directory layout.
+  The raw value stays in the retained v6 tables and in
+  `.daem0nmcp/storage/migrations/v7/<id>/source.snapshot.db`. Because the
+  digest changes the lossless payload, a store migrated before this build fails
+  `verify-v7` with "retained legacy source provenance differs" and must be
+  migrated again from its intact v6 source (`migrate-v7 --rollback`, then
+  `migrate-v7 --apply`). v7 is unreleased, so this affects development stores
+  only.
+- **Migrated v6 workspaces are usable end to end.** `session_brief`, recall,
+  outcomes, the four resources and `workspace_export` all work on a migrated
+  store; `migrate-v7` reports in `warnings` whatever it could not copy
+  faithfully; an un-migrated v6 workspace answers the new non-retryable
+  `MIGRATION_REQUIRED` naming the command instead of a retryable error with no
+  remedy; and `migrate-v7 --rollback` refuses with `ROLLBACK_WOULD_HIDE_WRITES`
+  rather than silently stranding writes made since activation (pass
+  `--discard-v7-writes` to accept that).
 
 ## [6.0.0] - 2026-01-29
 

README.md

@@ -1037,9 +1037,23 @@ python -m daem0nmcp.cli install-opencode --force
 
 This regenerates `opencode.json` and the plugin to match the latest version.
 
-### 5. Migrations Run Automatically
+### 5. Migrations
 
-Database schema migrations are applied automatically when any MCP tool runs. No manual migration step required.
+Within a storage format, schema migrations are applied automatically when any
+MCP tool runs, and no manual step is required.
+
+Moving an existing v6 workspace to the v7 storage format is **not** automatic.
+It is an offline, reversible command, and a v6 workspace answers
+`MIGRATION_REQUIRED` until it has run:
+
+```bash
+# Stop the server first, then:
+python -m daem0nmcp.cli --project-path /path/to/project migrate-v7 --apply
+
+# Undo it; refuses if events were recorded since activation unless
+# --discard-v7-writes is passed.
+python -m daem0nmcp.cli --project-path /path/to/project migrate-v7 --rollback
+```
 
 ### 6. Index Your Codebase
 

daem0nmcp/api/v7/application.py

@@ -15,7 +15,7 @@
 
 from ...covenant import CovenantGate, InvocationScope
 from ...workspace import Workspace
-from .errors import ErrorCode
+from .errors import DATABASE_IN_USE_RETRY_AFTER_MS, ErrorCode
 from .models import CapabilityState
 from .policy import V7_TOOL_LEVELS
 from .responses import ResponseFactory
@@ -119,18 +119,26 @@ def _remedy(
         if code in {"COUNSEL_REQUIRED", "TOKEN_MISSING"} and isinstance(
             workspace_id, str
         ):
-            target_arguments = {
-                key: value
-                for key, value in effective.items()
-                if key not in {"workspace_id", "preflight_token"}
-            }
-            return "memory_preflight", {
-                "workspace_id": workspace_id,
-                "target_tool": tool_name,
-                "target_arguments": target_arguments,
-            }
+            return self._preflight_remedy(workspace_id, tool_name, effective)
         return None, {}
 
+    @staticmethod
+    def _preflight_remedy(
+        workspace_id: str,
+        tool_name: str,
+        effective: Mapping[str, Any],
+    ) -> tuple[str, dict[str, Any]]:
+        target_arguments = {
+            key: value
+            for key, value in effective.items()
+            if key not in {"workspace_id", "preflight_token"}
+        }
+        return "memory_preflight", {
+            "workspace_id": workspace_id,
+            "target_tool": tool_name,
+            "target_arguments": target_arguments,
+        }
+
     def handler(self, tool_name: str) -> Callable[..., Any]:
         try:
             input_model = TOOL_INPUT_MODELS[tool_name]
@@ -294,6 +302,24 @@ async def invoke(**arguments: Any) -> object:
                         states = ()
                     if any(not isinstance(state, CapabilityState) for state in states):
                         states = ()
+                    if error_code == ErrorCode.DATABASE_IN_USE.value:
+                        # The gate already spent any preflight token, so a
+                        # retry needs a fresh one.
+                        workspace_id = effective.get("workspace_id")
+                        remedy_tool, remedy_arguments = (
+                            self._preflight_remedy(workspace_id, tool_name, effective)
+                            if isinstance(token, str) and isinstance(workspace_id, str)
+                            else (None, {})
+                        )
+                        return response.failure(
+                            ErrorCode.DATABASE_IN_USE,
+                            "The workspace database is currently in use.",
+                            retryable=True,
+                            retry_after_ms=DATABASE_IN_USE_RETRY_AFTER_MS,
+                            remedy_tool=remedy_tool,
+                            remedy_arguments=remedy_arguments,
+                            capability_states=tuple(states),
+                        )
                     return response.failure(
                         ErrorCode(error_code),
                         "The operation could not be completed.",

daem0nmcp/api/v7/code_entity_operations.py

@@ -49,7 +49,7 @@
 from .models import (
     EvidenceRef,
     RecordSummary,
-    contains_absolute_filesystem_path,
+    stored_relative_path,
 )
 from .public_ids import (
     PublicObjectIdNotFound,
@@ -79,7 +79,6 @@
 _MAX_ENTITY_EVENTS = 200
 _MAX_IMPACT_ENTITIES = 500
 _MAX_IMPACT_EDGES = 5_000
-_CONTROL_RE = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
 _EVENT_TYPE_RE = re.compile(r"^[a-z][a-z0-9_.-]{2,79}$")
 _EVENT_COLUMNS = (
     "event_id,workspace_id,stream_id,stream_kind,stream_version,event_type,"
@@ -366,7 +365,6 @@ def _validated_record_row(
         or row["stream_version"] != expected_version
         or row["state_hash"] != memory_state_hash(state)
         or row["content_hash"] != memory_content_hash(state)
-        or row["file_path"] is not None
     ):
         raise CodeEntityOperationError("CAPABILITY_DEGRADED")
     return row
@@ -382,17 +380,17 @@ def _event_record_summary(
     content = state["content"]
     if not isinstance(content, str) or not content:
         raise CodeEntityOperationError("CAPABILITY_DEGRADED")
+    # A record's valid time may precede or follow the transaction time it was
+    # written at; ``RecordSummary`` shows the earlier of the two.
     created = _datetime_from_us(occurred_at_us)
     updated = _datetime_from_us(recorded_at_us)
-    if created > updated:
-        raise CodeEntityOperationError("CAPABILITY_DEGRADED")
     try:
         return RecordSummary(
             record_id=record_id,
             record_type=state["record_type"],
             excerpt=content[:4000],
             tags=state["tags"],
-            relative_file_path=state["file_path_relative"],
+            relative_file_path=stored_relative_path(state["file_path_relative"]),
             current_status=(
                 "invalidated"
                 if state["deleted_at_us"] is not None
@@ -935,12 +933,8 @@ def _entity_row(
 
 def _safe_event_content(state: Mapping[str, Any]) -> str:
     content = state.get("content")
-    if (
-        not isinstance(content, str)
-        or not content
-        or _CONTROL_RE.search(content) is not None
-        or contains_absolute_filesystem_path(content)
-    ):
+    # Stored user text reads back as stored (UD-3).
+    if not isinstance(content, str) or not content:
         raise CodeEntityOperationError("CAPABILITY_DEGRADED")
     return content
 

daem0nmcp/api/v7/discovery_operations.py

@@ -16,7 +16,6 @@
 import inspect
 import json
 import os
-import posixpath
 import re
 import secrets
 import sqlite3
@@ -51,7 +50,13 @@
 )
 from .application import AdmittedRequest
 from .errors import STABLE_ERROR_CODE_SET
-from .models import CapabilityState, Page, RecordSummary, RetrievalData
+from .models import (
+    CapabilityState,
+    Page,
+    RecordSummary,
+    RetrievalData,
+    stored_relative_path,
+)
 from .public_ids import (
     PublicObjectIdNotFound,
     PublicObjectIdRepository,
@@ -755,24 +760,6 @@ def _partition_metadata(
     return row_count, content_hash
 
 
-def _safe_code_path(value: object) -> bool:
-    if (
-        not isinstance(value, str)
-        or not 1 <= len(value) <= 1024
-        or "\\" in value
-        or "\x00" in value
-        or value.startswith(("/", "~"))
-        or re.match(r"^[A-Za-z]:", value) is not None
-        or value in {".", ".."}
-    ):
-        return False
-    components = value.split("/")
-    return (
-        all(component not in {"", ".", ".."} for component in components)
-        and posixpath.normpath(value) == value
-    )
-
-
 def _validate_entity_partition(
     connection: sqlite3.Connection,
     workspace_id: str,
@@ -1061,24 +1048,24 @@ def _record_summary(row: sqlite3.Row) -> RecordSummary:
     try:
         content = row["content"]
         tags = json.loads(str(row["tags_json"]))
+        # A migrated v6 row keeps the host-absolute ``file_path`` v6 wrote and
+        # may carry a valid time ahead of its transaction time; neither is
+        # corruption, and ``RecordSummary`` bounds what is emitted.
         if (
             not isinstance(content, str)
             or not content
             or not isinstance(tags, list)
-            or row["file_path"] is not None
             or row["deleted_at_us"] is not None
         ):
             raise ValueError
         created = _datetime_from_us(row["created_at_us"])
         updated = _datetime_from_us(row["updated_at_us"])
-        if updated < created:
-            raise ValueError
         return RecordSummary(
             record_id=row["record_id"],
             record_type=row["record_type"],
             excerpt=content[:4000],
             tags=tags,
-            relative_file_path=row["file_path_relative"],
+            relative_file_path=stored_relative_path(row["file_path_relative"]),
             current_status="archived" if bool(row["archived"]) else "current",
             content_hash=row["content_hash"],
             created_at=created,
@@ -1424,7 +1411,10 @@ def _code_capability_error(
         else "CAPABILITY_DEGRADED"
     )
     remediation = (
-        "Set DAEM0NMCP_APPS_ENABLED=true and install the apps dependency profile."
+        "Set DAEM0NMCP_APPS_ENABLED to true, false, or leave it unset."
+        if capability_status == "failed"
+        else "Install the apps profile with pip install 'daem0nmcp[apps]'; "
+        "it turns on automatically unless DAEM0NMCP_APPS_ENABLED=false."
     )
     return DiscoveryOperationError(
         code,
@@ -2046,6 +2036,56 @@ def reader(connection: sqlite3.Connection) -> _EntitySelection:
         raise _translate_error(error) from None
 
 
+def _lexical_catchup_pending_sync(
+    dependencies: DiscoveryOperationDependencies,
+    workspace: Workspace,
+    record_ids: tuple[str, ...],
+) -> bool:
+    """Return whether the lexical projection still owes these records.
+
+    True only when the records are absent from the active lexical generation
+    *and* a rebuild job is queued or running, i.e. a retry can actually make
+    progress. A dead-lettered rebuild, a permanently unavailable index, or a
+    record that simply does not match the entity name is not retryable.
+    """
+
+    if not record_ids:
+        return False
+
+    def reader(connection: sqlite3.Connection) -> bool:
+        active = connection.execute(
+            "SELECT generation FROM projection_manifests WHERE workspace_id=? "
+            "AND projection_name='lexical' AND status='active'",
+            (workspace.workspace_id,),
+        ).fetchone()
+        indexed = 0
+        if active is not None:
+            placeholders = ",".join("?" for _ in record_ids)
+            indexed = int(
+                connection.execute(
+                    "SELECT COUNT(*) FROM retrieval_documents WHERE workspace_id=? "
+                    f"AND projection_generation=? AND record_id IN ({placeholders})",
+                    (workspace.workspace_id, int(active[0]), *record_ids),
+                ).fetchone()[0]
+            )
+        if indexed >= len(record_ids):
+            return False
+        return (
+            connection.execute(
+                "SELECT 1 FROM background_jobs WHERE workspace_id=? "
+                "AND job_type='retrieval.projection_rebuild' "
+                "AND status IN ('queued','running') LIMIT 1",
+                (workspace.workspace_id,),
+            ).fetchone()
+            is not None
+        )
+
+    try:
+        return bool(_read_snapshot(dependencies, workspace, reader))
+    except Exception:
+        return False
+
+
 def _generation_is_current_sync(
     dependencies: DiscoveryOperationDependencies,
     workspace: Workspace,
@@ -2093,7 +2133,21 @@ async def _memory_recall_entity(
                 raise DiscoveryOperationError("CAPABILITY_DEGRADED")
             indexed[record_id] = item.record
         if set(indexed) != set(selection.record_ids):
-            raise DiscoveryOperationError("CAPABILITY_DEGRADED")
+            missing = tuple(
+                record_id
+                for record_id in selection.record_ids
+                if record_id not in indexed
+            )
+            # Only a projection that has genuinely not caught up yet is worth
+            # retrying. A permanently unavailable or dead-lettered lexical
+            # index stays a terminal CAPABILITY_DEGRADED.
+            pending = await _run_blocking(
+                dependencies,
+                lambda: _lexical_catchup_pending_sync(dependencies, workspace, missing),
+            )
+            raise DiscoveryOperationError(
+                "DATABASE_IN_USE" if pending else "CAPABILITY_DEGRADED"
+            )
         await _run_blocking(
             dependencies,
             lambda: _generation_is_current_sync(