feat(memory): completion-log-reconcile + crontab wiring + PGUSER fix (#561, #562, #564)

ARCHITECTURE.md

@@ -354,11 +354,11 @@ The installer enforces this order automatically.
 
 - **Session‑Aware Caching:** Entity resolver caches per session (30‑minute TTL) to reduce database load.
 - **Embedding Batch Processing:** Embedding scripts run incrementally via cron to avoid overwhelming Ollama.
-- **Connection Pooling:** All PostgreSQL clients use connection pools (default size 5).
+- **Connection Pooling:** PostgreSQL clients use connection pools sized per-service (`turn-context` plugin and the `entity-resolver` library use `max: 5`; the `self-awareness` metacognition plugin uses a smaller `max: 3`) rather than one shared default.
 - **Vector Indexes:** `memory_embeddings` uses PostgreSQL `pgvector` indexes for fast similarity search.
 - **Turn-Context Plugin Context Budget:** The `turn-context` plugin budgets ~1000 tokens for context injection. High-confidence results (>0.7 threshold) get full content injected; lower-confidence results get a summary only. Configurable via `SEMANTIC_RECALL_TOKEN_BUDGET` and `SEMANTIC_RECALL_HIGH_CONFIDENCE` environment variables.
 - **Semantic Recall Priority Weighting:** Results are scored as `vector_similarity × priority_weight` from the `memory_type_priorities` table. Workflows (1.50) and lessons (1.30) surface before entity_facts (1.00) and daily_logs (0.90).
-- **Ghost Embeddings (⚠️ Known Failure Mode):** Orphaned vectors in `memory_embeddings` from deleted source records surface stale information with high confidence. Detection requires manual LEFT JOIN queries. No automatic cleanup exists yet — this is the most dangerous class of memory corruption.
+- **Ghost Embeddings (⚠️ Partial Coverage):** Orphaned vectors in `memory_embeddings` from deleted source records surface stale information with high confidence. `memory-maintenance.py`'s "Clean orphaned embeddings" phase (`clean_orphaned_embeddings()`) automatically deletes orphans for `source_type IN ('entity_fact', 'entity')` on every run — the original ghost-embedding gap for those two types (#216-adjacent) is closed. However, `memory_embeddings` rows for other source types (`lesson`, `library`, and any other `source_type` used by the chunked-file/research embedding paths) have **no automatic orphan cleanup** and still require manual LEFT JOIN queries to detect. This narrower gap remains the most likely residual class of memory corruption.
 
 ## Security Model
 

CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+### Batch: completion-log-reconcile-561 (Issue #561)
+
+#### Added
+- **`completion-log-reconcile.py` — deterministic completion-side daily-log reconcile** (nova-mind#561) — Closes the gap where closed `work_queue` rows and completed `workflow_runs` rows could fail to produce a corresponding daily-log completion line. The script runs LLM-free from cron every few minutes, scans both tables for terminal-status rows whose `completion_logged_at` watermark is NULL, and appends exactly one line per row to the correct `~/.openclaw/workspace/memory/YYYY-MM-DD.md` file (dated by the row's own completion timestamp, not script-run time):
+  - `work_queue`: `- HH:MM wq#<id> closed (<kind>): <description> — <status>` for rows reaching `done`/`failed`/`stale`/`cancelled`.
+  - `workflow_runs`: `- HH:MM workflow run #<id> <status> (workflow <workflow_id>): <trigger_context>` for rows reaching `completed`/`failed`/`cancelled`.
+  - **Two-phase idempotency:** a file-grep pre-check (current day + adjacent days) fires before append, and `completion_logged_at` is set only after the append succeeds, so a crash between append and commit is recovered by the next run with zero duplicate lines.
+  - **Marker anchoring:** the grep pattern uses a word boundary immediately after the numeric id so a shorter id cannot false-match inside a longer prefix id (e.g. `#1` inside `#10`) — fixes the S2/P1 collision found during QA desk review.
+  - **Sanitization:** description/trigger_context whitespace is collapsed to single spaces; long text is truncated to 120 chars (work_queue description) or ~80 chars (trigger_context) with a trailing `…`; Markdown structural characters are preserved verbatim.
+  - **Shared flock:** both `completion-log-reconcile.py` and `generate-daily-log.py` acquire an exclusive advisory `flock` on `~/.openclaw/workspace/memory/.daily-log.lock` during their read/write/rename critical sections, preventing inter-script races.
+  - **Watermark fallback:** `work_queue` uses `COALESCE(completed_at, last_checked_at, created_at)`; `workflow_runs` uses `COALESCE(completed_at, started_at)`. Rows with unusable timestamps are skipped with a stderr warning rather than crashing.
+
+#### Migrations
+- `memory/migrations/087_completion_log_watermark.sql` (nova-mind#561) — Adds `completion_logged_at timestamptz` to both `work_queue` and `workflow_runs`, with `COMMENT ON COLUMN` documenting the watermark semantics, and seeds the column for all already-closed rows at deploy time via `COALESCE(completed_at, now())` guarded by `WHERE completion_logged_at IS NULL`. The migration header documents the operational risk that re-applying against a live system will seed any row that became terminal between applies, permanently excluding it from the reconcile scan; drain pending closures with `completion-log-reconcile.py` before re-applying.
+
+#### Changed
+- **`generate-daily-log.py` companion flock** (nova-mind#561) — Small, surgical change to wrap the existing read/modify/atomic-rename critical section with the same shared advisory lock used by `completion-log-reconcile.py`, closing the deterministic race at 00:05/06:00/12:00/18:00 UTC cron boundaries.
+- **`database/schema.sql` + `database/schema-reference.md`** (nova-mind#561) — Declarative schema and table listing updated to include the new `completion_logged_at` column on both tables, matching the migration.
+- **`agent-install.sh` crontab wiring for `completion-log-reconcile.py`** (nova-mind#562) — Installs/verifies an idempotent `*/5 * * * *` cron entry that runs the LLM-free script every few minutes, appending output to `~/.openclaw/logs/completion-log-reconcile.log`. Mirrors the existing `generate-daily-log.py` idempotent marker-grep pattern and includes `--no-cron` / `--verify-only` support.
+
+#### Fixed
+- **PGUSER cron-environment resolution in `completion-log-reconcile.py` and `generate-daily-log.py`** (nova-mind#564) — Staging integration testing of the #562 cron wiring found every cron invocation of `completion-log-reconcile.py` exiting 1 with `fe_sendauth: no password supplied`. Root cause: `connect()`'s PGUSER fallback was `os.environ.get("USER", str(os.getuid()))`, and Debian cron does not set `USER` — only `LOGNAME`/`HOME`/`SHELL`/`PATH` — so under cron the fallback resolved to the literal numeric UID string (e.g. `"1005"`), which matched no database role and no `.pgpass` entry. Both scripts now resolve PGUSER via `getpass.getuser()` (checks `LOGNAME`/`USER`/`LNAME`/`USERNAME` in order, falling back to `pwd.getpwuid(os.getuid())` only if none are set), which always returns a username string under a stripped cron environment. Test fixtures/helpers in `tests/test_completion_log_reconcile.py` that hardcoded `PGUSER="nova"` were also updated to derive the current OS user via `getpass.getuser()`, fixing a secondary staging failure where the `TestFlockMutualExclusion::test_generate_daily_log_takes_same_lock` test connected as a role lacking grants on staging. Verified on staging with a clean before/after cron repro (08:25 UTC pre-fix run: `fe_sendauth`; 08:30 UTC post-fix run, same cron slot: `Appended 0 line(s)`, exit 0).
+
+#### Tests
+- `tests/test_completion_log_reconcile.py` (nova-mind#561, nova-mind#564) — 64 automated cases covering line formatting, sanitization, watermark fallback, idempotency, crash recovery, midnight-boundary dating, status decision tables, migration idempotency (including the amended TC-561-26 semantics), failure modes (permission errors, unreachable DB, DB permission-denied), flock mutual exclusion, the TC-561-35 numeric-prefix collision regression for both tables, and `TestConnectUserResolution` — a cron-env regression suite that unsets all of `USER`/`LOGNAME`/`LNAME`/`USERNAME` and asserts the resolved PGUSER is a username string (not the numeric UID) for both `completion-log-reconcile.connect()` and `generate-daily-log.connect()`.
+
+#### Issues Closed
+- #561 — completion-log-reconcile: deterministic daily-log completion lines for work_queue + workflow_runs
+- #562 — completion-log-reconcile.py: no crontab/sweeper invocation wiring — feature is inert in production
+- #564 — completion-log-reconcile.py fails on every cron invocation: PGUSER fallback reads USER (unset under Debian cron) → connects as UID string → fe_sendauth
+
 ### Batch: append-run-note-557 (Issue #557)
 
 #### Added

README.md

@@ -29,17 +29,19 @@ All five subsystems share a single PostgreSQL database (`{username}_memory`) and
 
 ### Memory Maintenance
 
-Memory maintenance is handled by a **unified** script `memory/templates/memory-maintenance.py` (deployed to `~/.openclaw/scripts/memory-maintenance.py` by `agent-install.sh`) that replaces the separate embedding scripts (`embed-full-database.py`, `embed-memories.py`, `embed-research.py`, `embed-library.py`) and the previous memory maintenance logic. It runs as a 9-phase pipeline:
+Memory maintenance is handled by a **unified** script `memory/templates/memory-maintenance.py` (deployed to `~/.openclaw/scripts/memory-maintenance.py` by `agent-install.sh`) that replaces the separate embedding scripts (`embed-full-database.py`, `embed-memories.py`, `embed-research.py`, `embed-library.py`) and the previous memory maintenance logic. It runs as a pipeline of these steps, in order:
 
 1. **Cooldown check** — 4-hour gate prevents redundant runs (`--force` to bypass, `--state-file` override)
-2. **Embed** — Generates semantic embeddings across all table types (entities, facts, lessons, events, research, library, tasks, blog posts, etc.); memory files are split with a paragraph/section-boundary-aware chunker before embedding (see `memory/README.md#text-chunking`)
-3. **Cross-key consolidation** — pgvector cosine similarity ≥0.92
-4. **Same-key dedup** — pg_trgm similarity, 3-tier (high/medium/low)
-5. **Confidence decay** — Exponential, durability-based rates
-6. **Ghost entity cleanup** — Pattern-based, zero-fact orphans, low-fact review
-7. **Entity-level dedup** — ≥80% auto-merge via `merge_entities()`, <80% review queue
-8. **Clean orphaned embeddings**
-9. **Archive & purge** low-confidence facts
+2. **Lessons deduplication** — Runs before embedding to avoid wasted embed calls on rows about to be merged: exact duplicates keep the oldest row, near-duplicates (similarity ≥0.80) go to a review report (`--skip-lesson-dedup` to skip)
+3. **Embed** — Generates semantic embeddings across all table types (entities, facts, lessons, events, research, library, tasks, blog posts, etc.); memory files are split with a paragraph/section-boundary-aware chunker before embedding (see `memory/README.md#text-chunking`)
+4. **Cross-key consolidation** — pgvector cosine similarity ≥0.92
+5. **Same-key dedup** — pg_trgm similarity, 3-tier (high/medium/low)
+6. **Confidence decay** — Exponential, durability-based rates
+7. **Ghost entity cleanup** — Pattern-based, zero-fact orphans, low-fact review
+8. **Entity-level dedup** — ≥80% auto-merge via `merge_entities()`, <80% review queue
+9. **Re-embed modified facts** — Facts touched by consolidation/dedup above get stale embeddings deleted and regenerated
+10. **Clean orphaned embeddings**
+11. **Archive & purge** low-confidence facts (archive facts below the confidence floor, then hard-delete archived rows older than 1 year)
 
 **Flags:** `--dry-run`, `--verbose`, `--force`, `--state-file`, `--skip-embed`, `--skip-consolidation`, `--skip-dedup`, `--skip-decay`, `--skip-ghost-cleanup`, `--skip-entity-dedup`, `--skip-lesson-dedup`, `--reindex-files` (force a full re-chunk/re-embed of memory files — see `memory/README.md#text-chunking`)
 
@@ -96,6 +98,8 @@ The installer is **idempotent** — safe to run multiple times. It installs all
 | `--verify-only` | Check installation without modifying anything |
 | `--force` | Force overwrite existing files |
 | `--no-restart` | Skip automatic gateway restart |
+| `--no-cron` | Skip installation of all cron-installed scripts (daily-log generation, D100 roll announcer, Hermes comms-check, completion-log-reconcile) |
+| `--regenerate-agents-json` | Backup and regenerate `~/.openclaw/agents.json` from the database |
 | `--database NAME` / `-d NAME` | Override database name (default: `${USER}_memory`) |
 
 See subsystem READMEs for detailed documentation:

agent-install.sh

@@ -33,6 +33,8 @@ VERIFICATION_ERRORS=0
 ANNOUNCE_D100_CRON_STATUS="not installed"
 # shellcheck disable=SC2034
 HERMES_COMMS_CRON_STATUS="not installed"
+# shellcheck disable=SC2034
+COMPLETION_LOG_RECONCILE_CRON_STATUS="not installed"
 
 # Track if gateway restart is needed
 # shellcheck disable=SC2034
@@ -103,6 +105,11 @@ HERMES_COMMS_CHECK_SCRIPT="hermes-comms-check.sh"
 HERMES_COMMS_CHECK_MARKER="$HOME/.openclaw/scripts/comms/$HERMES_COMMS_CHECK_SCRIPT"
 HERMES_COMMS_CRON_ENTRY='0 */4 * * * '"$HERMES_COMMS_CHECK_MARKER"' >> '"$OPENCLAW_LOGS_DIR"'/hermes-comms-check.log 2>&1'
 
+# Completion log reconcile cron configuration (issue #562)
+COMPLETION_LOG_RECONCILE_SCRIPT="completion-log-reconcile.py"
+COMPLETION_LOG_RECONCILE_CRON_MARKER="$HOME/.openclaw/scripts/$COMPLETION_LOG_RECONCILE_SCRIPT"
+COMPLETION_LOG_RECONCILE_CRON_ENTRY='*/5 * * * * '"$COMPLETION_LOG_RECONCILE_CRON_MARKER"' >> '"$OPENCLAW_LOGS_DIR"'/completion-log-reconcile.log 2>&1'
+
 # Superuser connection helper for DDL operations
 PG_SUPERUSER="${PG_SUPERUSER:-$DB_USER}"
 PG_SUPERUSER_PASSWORD="${PG_SUPERUSER_PASSWORD:-${PGPASSWORD:-}}"
@@ -380,6 +387,59 @@ _install_hermes_comms_check_cron() {
     fi
 }
 
+# Install or verify the completion log reconcile cron entry.
+# Uses globals: COMPLETION_LOG_RECONCILE_SCRIPT, COMPLETION_LOG_RECONCILE_CRON_MARKER,
+#               COMPLETION_LOG_RECONCILE_CRON_ENTRY, VERIFY_ONLY, NO_CRON
+# Sets global: COMPLETION_LOG_RECONCILE_CRON_STATUS
+_install_completion_log_reconcile_cron() {
+    if [ "${NO_CRON:-0}" -eq 1 ]; then
+        echo -e "  ${INFO} Completion log reconcile cron installation skipped (--no-cron)"
+        COMPLETION_LOG_RECONCILE_CRON_STATUS="skipped by --no-cron"
+        return 0
+    fi
+
+    local cron_drift_lines=()
+    local current_crontab
+    current_crontab=$(crontab -l 2>/dev/null || true)
+
+    if echo "$current_crontab" | grep -qF "$COMPLETION_LOG_RECONCILE_CRON_MARKER"; then
+        local line
+        while IFS= read -r line; do
+            case "$line" in
+                *"$COMPLETION_LOG_RECONCILE_CRON_MARKER"*)
+                    if [ "$line" != "$COMPLETION_LOG_RECONCILE_CRON_ENTRY" ]; then
+                        cron_drift_lines+=("$line")
+                    fi
+                    ;;
+            esac
+        done <<< "$current_crontab"
+
+        if [ ${#cron_drift_lines[@]} -gt 0 ]; then
+            echo -e "  ${WARNING} Existing cron entry for $COMPLETION_LOG_RECONCILE_SCRIPT differs from expected schedule (drift detected):"
+            local drift_line
+            for drift_line in "${cron_drift_lines[@]}"; do
+                echo "      $drift_line"
+            done
+            VERIFICATION_WARNINGS=$((VERIFICATION_WARNINGS + 1))
+            COMPLETION_LOG_RECONCILE_CRON_STATUS="drift detected (review required)"
+        elif [ "${VERIFY_ONLY:-0}" -eq 1 ]; then
+            echo -e "  ${CHECK_MARK} Completion log reconcile cron entry installed"
+            COMPLETION_LOG_RECONCILE_CRON_STATUS="installed"
+        else
+            echo -e "  ${CHECK_MARK} Completion log reconcile cron entry verified"
+            COMPLETION_LOG_RECONCILE_CRON_STATUS="verified"
+        fi
+    elif [ "${VERIFY_ONLY:-0}" -eq 1 ]; then
+        echo -e "  ${CROSS_MARK} Completion log reconcile cron entry missing"
+        COMPLETION_LOG_RECONCILE_CRON_STATUS="missing"
+        VERIFICATION_ERRORS=$((VERIFICATION_ERRORS + 1))
+    else
+        (crontab -l 2>/dev/null || true; echo "$COMPLETION_LOG_RECONCILE_CRON_ENTRY") | crontab -
+        echo -e "  ${CHECK_MARK} Installed completion log reconcile cron entry (every 5 minutes)"
+        COMPLETION_LOG_RECONCILE_CRON_STATUS="installed"
+    fi
+}
+
 # Install the PostgreSQL NOTIFY listener as a systemd --user service.
 # Parameters: source_script source_service target_dir service_dir logs_dir
 _install_pg_notify_listener() {
@@ -443,6 +503,7 @@ REGENERATE_AGENTS_JSON=0
 DAILY_LOG_CRON_STATUS="not installed"
 ANNOUNCE_D100_CRON_STATUS="not installed"
 HERMES_COMMS_CRON_STATUS="not installed"
+COMPLETION_LOG_RECONCILE_CRON_STATUS="not installed"
 
 while [[ $# -gt 0 ]]; do
     case $1 in
@@ -1036,6 +1097,7 @@ if [ $VERIFY_ONLY -eq 1 ]; then
     echo "Memory scripts verification..."
     _install_daily_log_cron
     _install_announce_d100_cron
+    _install_completion_log_reconcile_cron
 
     echo ""
     echo "═══════════════════════════════════════════"
@@ -1066,6 +1128,14 @@ if [ $VERIFY_ONLY -eq 1 ]; then
     fi
     echo -e "  ${comms_symbol} Hermes comms-check cron: $HERMES_COMMS_CRON_STATUS"
 
+    completion_symbol="$CHECK_MARK"
+    if [[ "$COMPLETION_LOG_RECONCILE_CRON_STATUS" == missing* ]]; then
+        completion_symbol="$CROSS_MARK"
+    elif [[ "$COMPLETION_LOG_RECONCILE_CRON_STATUS" == drift* ]]; then
+        completion_symbol="$WARNING"
+    fi
+    echo -e "  ${completion_symbol} Completion log reconcile cron: $COMPLETION_LOG_RECONCILE_CRON_STATUS"
+
     if [ $VERIFICATION_ERRORS -gt 0 ]; then
         echo -e "  ${CROSS_MARK} $VERIFICATION_ERRORS errors found"
         exit 1
@@ -1658,6 +1728,9 @@ if [ -d "$SCRIPTS_SOURCE" ]; then
 
     # --- D100 roll announcer cron entry ---
     _install_announce_d100_cron
+
+    # --- Completion log reconcile cron entry (issue #562) ---
+    _install_completion_log_reconcile_cron
 else
     echo -e "  ${WARNING} Scripts directory not found at $SCRIPTS_SOURCE (skipping)"
 fi
@@ -2800,6 +2873,7 @@ echo "    • Schema managed via pgschema (database/schema.sql)"
 echo "    • Daily memory log cron → $DAILY_LOG_CRON_STATUS"
 echo "    • D100 announcer cron → $ANNOUNCE_D100_CRON_STATUS"
 echo "    • Hermes comms-check cron → $HERMES_COMMS_CRON_STATUS"
+echo "    • Completion log reconcile cron → $COMPLETION_LOG_RECONCILE_CRON_STATUS"
 if [ ${#INSTALLED_HOOKS[@]} -gt 0 ]; then
     for hook in "${INSTALLED_HOOKS[@]}"; do
         echo "    • Hook: $hook"

cognition/docs/cross-database-replication.md

@@ -624,7 +624,7 @@ above.
 
 - `cognition/focus/agent_chat/schema.sql` - Contains trigger definitions and replication comments
 - `database/agent-chat/schema.sql` - The dedicated `agent_chat` database's schema as of nova-mind#320 (see the superseded-banner note above) — `agent_chat` no longer lives in `nova_memory`, so replication guidance in this document does not apply to it
-- `agent-install.sh` - Automatic replication detection and configuration
+- `agent-install.sh` - As of #320 no longer auto-detects or configures `agent_chat` logical replication (see the "agent_chat runtime configuration" section, which explicitly notes replication for #64/#67 was superseded by the shared-DB design); the replication mechanics on this page are reference-only for other tables that might use this pattern in the future
 - `memory/database/renames.json` - Declarative rename manifest applied by Step 1.5
 - Database migration scripts in `migrations/`
 - GitHub issue #130 — `ENABLE REPLICA TRIGGER` sets mode R, not ALWAYS