Conversation
Parallel agents/subagents each built their own client with no fleet-wide bound — a provider concurrency limit (e.g. Tencent 'model Concurrency limit 1200') turned into a stampede of 429s across every agent. SetLLMConcurrency (default 6, --max-concurrent-llm-requests) bounds concurrent in-flight LLM requests process-wide; acquisition is cancelable (user stop while queued) and the slot is held for the stream's lifetime.
…failure budget A provider concurrency/rate limit (Tencent 'model Concurrency limit 1200') is pacing, not a failure: under sustained parallel load every one of the 10 infra-retry attempts failed and the turn died. StatusError 429 now maps to a dedicated retryClassThrottle with its own 200-attempt ceiling, full-jitter waits growing to a 120s cap, Retry-After honored via StatusError.RetryAfter, and the global-disable coupling extended (max-stream-retries 0/negative silences all tiers). Tests: survives 12 consecutive 429s with a 2-attempt infra budget; ceiling terminal; global disable; delay curve pinned.
Single-line footer below the status bar, toggled with /infobar and persisted as config show-info-bar. Segments: late version, project folder, focused agent's provider/profile ref + model (config agent_models lookup), context usage bar, running subagent count, discovered skills with estimated token footprint (SkillsInfo plumbed from cmd/late/main.go), tokens to the compaction threshold, and session uptime. The row reserves one layout row, truncates to window width, and hides while the file picker is open. Note: state.go and update.go also carry the /timestamps command def and toggle handler (feature (b)) — the info bar is the dominant feature in both files, so they are committed here per the no-hunk-split rule. Note: config show-info-bar field lands with the compaction commit (config.go is dominantly compaction).
…estamps Session stamps every appended history message with an RFC3339 receive time (appendMessage is the single path; caller-supplied timestamps are preserved, legacy entries stay unprefixed). The TUI renders the time as a muted [HH:MM:SS] row at the start of user and assistant blocks when ShowTimestamps is on; toggling /timestamps invalidates the transcript block cache and persists show-timestamps to config.json (handled by the update.go change in the info-bar commit).
…d tool Staged rollout of the jev-compaction port (design source: Waxmell114514/jev-compaction, MIT). New internal/compaction package: paragraph segmentation, System One scoring backends (resolve via env, key-file lookup, per-backend URLs), a pipeline that scores tool outputs and appends decisions to a shadow log (~/.local/share/late/compaction-shadow.jsonl), and relocation that elides low-scoring segments into [[elided ...]] pointers backed by an in-memory store. Executor integration: SetToolResultCompactor installs the pipeline process-wide (root agent + subagents); results over MinCompactToolResultChars (4000) are offered to the stage. Shadow mode (default) scores and logs without changing any result; enabled mode additionally relocates and registers the expand tool so originals stay retrievable. Everything fails open: backend errors, canceled contexts, and shadow-log failures keep the original tool output. Scoring calls share the process-wide LLM concurrency limiter (client.AcquireLLMSlot). Modes resolve as --compaction-mode > config compaction-mode > shadow; invalid values warn and fall back to shadow. --compaction-threshold (default 0.35) sets the elision score cutoff in enabled mode. compaction-threshold-percent feeds the info bar headroom segment. config.go and config_test.go also carry the show-info-bar / show-timestamps fields and cmd/late/main.go the SkillsInfo plumbing (those files are dominantly compaction, so they land here instead of the info-bar commit, per the no-hunk-split rule). Docs: quickstart sections for context compaction and the info bar & timestamps; README feature bullet + slash-command line.
…ev scoring, elision) Session.CompactContext walks the whole history (not just tool outputs) and relocates low-scoring segments into the elide store: - Frozen prefix: the first max(1, len/4) messages (25%) stay byte-identical, so the prompt-cache anchor - the system prompt at index 0 plus the earliest exchanges - never moves across compactions. - User messages are immune; assistant and tool-result contents are compacted in place with roles, ToolCalls, and timestamps preserved (ToolCalls are structurally required and never touched: only Content shrinks). - Segments are scored against the ongoing task (the last user message); anything scoring strictly below the threshold is replaced by one [[elided ...]] pointer line whose original lives in the store for the expand tool. A nil store disarms relocation: a pointer whose original cannot be stored must never enter history. - Shadow mode (ShadowOnly) runs the full scoring walk and computes the honest would-save report without mutating history or storing originals. - Fail-open mid-walk: a scorer error stops the walk, already-rewritten messages stay valid (their pointers and stored originals resolve), and the error reports how far the walk got. Persistence is the callers job (SaveHistory), so compaction never breaks a session. - Token math: CompactionReport carries EstimateMessageTokens before/after, tokens saved, and the byte size of what was relocated into the store.
Manual command: /jev-compact-context runs one full-history compaction pass on the session behind the TUI. It requires compaction-mode other than off (nil Compactor reports "compaction unavailable"); under shadow mode it runs report-only. The run executes off the TUI update loop (network scoring can take seconds) and reports via compactionResultMsg. Auto-trigger: with jev-autocompact on (config.ResolveAutocompact; default false) and the focused agents context usage crossing jev-autocompact-percent (default 99, valid 1-100) of the context window, the same compaction runs automatically. It fires once per crossing - the agent stays disarmed until usage falls below (percent-9)% (typically right after a compaction shrank the history) or /new starts a fresh conversation. Shared CompactionRunning guard: exactly one CompactContext run (manual or auto) may be in flight at a time; both paths set it on dispatch and the result message clears it. Store id-space unification: the elide-id counter moves into compaction.Store (NextID/Put exported), so tool-output and history compaction mint pointers from one id space and "elide-<n>" always names exactly one original. Pipeline-to-TUI plumbing: Pipeline.HistoryScorer exposes the decision client as the session.HistoryScorer; cmd/late wires historyCompactionRunner (session.CompactContext with the pipelines threshold, persisting mutated history via SaveHistory - skipped for shadow runs, and with a fresh private store there since shadow mints pointer ids without an expand tool to read them). NewModel resolves the autocompact settings; late -h documents the feature under "Context compaction".
quickstart.md: extend the Context Compaction section with /jev-compact-context (manual full-history compaction - frozen prefix, Jev scoring of every post-prefix message, elision into pointers the expand tool resolves; requires compaction-mode other than off, report-only under shadow) and the jev-autocompact / jev-autocompact-percent config keys (auto-trigger at the configured context-usage percent, once per crossing, re-arms after compaction). README.md: one feature bullet next to the tool-output compaction entry.
Emasoft
changed the title
|
The TUI changes should be evaluated separately from the rest. Note that summarization/compaction has been often requested. I am open to adding it but it must be done so with extreme care. That includes for example no mandatory/automatic compaction unless a user opts in. Further note that I have looked at jev and from what I have seen so far its primary use case is enterprise deployment. It seems like a faster, cheaper model trained directly on outputting JSON that answers a question. It seems great for trivial answers ("which department does this go to") and quite awful for anything requiring nuance. I'd suggest some tests first there, specifically considering that eg DeepSeek is basically free. I am open to being proven wrong here but it must be investigated instead of just jumping on a hype train.
Please look at the comment there since I am still not sure I get that PR. |
Yes, you can enable or disable it in config.json. I also thought about making the model's compaction configurable later. Jev is only the beginning.
True, but it is also slow. Jev is very fast. And it should only prune, never rewrite. I will test it.
You are right. In fact, this compaction feature is more of an experiment. But consider that System-One models will definitely decrease in price once some Chinese version of them is available. If DeepSeek is that affordable, you can only imagine how cheap a System-One model will soon be. They might even become local models. However, the real feature should be allowing the user to choose which model to use for compaction and enable or disable it in any way (even based on the subagent type, or letting the orchestrator decide whether to enable it or not when spawning). Anyway, I’ll do as you say. I will try to split the PR into independent parts. |
While I get your line of thought I still think this requires some investigation and thoughts before implementing it. Also note that while I do generally agree with the premise, I disagree with the comparison specifically to DeepSeek. DeepSeek is among the fastest APIs (if not the fastest after Gemini), it plays around the frontier leagues just right below Fable/Astra and it is absurdly cheap. I am going close to 1 billion tokens total usage and still have a substantial part of my $10 left (I exclusively use Late for DeepSeek). I also saw these Jev posts a couple days ago which is why I was investigating it similarly and then specifically came to that conclusion, it seems great in enterprise and generally for simple, repetitive tasks that do not require much nuance and relatively useless otherwise. I agree that locally it might be something worthwhile but this is primarily due to speed. If quality does not degrade I am fine with adding it but again as noted it does require some testing. Also in that case it would be better to abstract it further so users can swap it e.g. using some arbitrary Jev model or using a regular autoregressive LLM. Note that these are just random thoughts not specific architectural decisions to take right now. I'd test it out on harder architecturally complex tasks first to verify. The important part is I want to provide the best out of the box experience possible. Users should never have to fiddle with settings. Also since you noted that this depends on #132, please check out the comment there. I still am not sure I get that PR. |
|
Split as requested — closing this as superseded by the independent PRs below, each one evaluated on its own:
The TUI changes live in #135/#136/#137; the compaction work is isolated in #139, which addresses your points directly:
|
2 participants
Summary
Three features, staged so each is independently revertable:
Info bar (
/infobar) — optional single-line footer below the status bar showing: Late version · project folder · focused agent's provider/profile ref + model (honoringagent_models) · context usage bar · running subagent count · discovered skills with estimated token footprint · tokens to the compaction threshold · session uptime. Reserves one layout row, truncates to window width, hides while the file picker is open. Toggle persists toconfig.json(show-info-bar).Transcript timestamps (
/timestamps) — the session stamps every appended history message with an RFC3339 receive time; the TUI renders a muted[HH:MM:SS]prefix on user and assistant blocks when enabled. Legacy history entries render unprefixed. Toggle persists toconfig.json(show-timestamps).Jev-scored context compaction (staged: off → shadow → enabled) — port of the jev-compaction design:
off— nothing runs.shadow(default) — tool outputs are segmented and Jev-scored; decisions are appended to the shadow log (~/.local/share/late/compaction-shadow.jsonl). No tool result is ever changed.enabled— tool outputs over 4000 chars are segmented, scored, and low-scoring segments are elided into[[elided …]]pointers; the newexpandtool retrieves the original text on demand (shared store, inherited by subagents).--compaction-mode>config.jsoncompaction-mode>shadow; invalid values warn and fall back to the safe default.--compaction-threshold(default0.35) sets the elision score cutoff.Docs: new "Context compaction" and "Info bar & timestamps" sections in
docs/quickstart.md; README feature bullet + slash-command line.New in this update
Full-history context compaction (
/jev-compact-context) — the manual command scores every message after a frozen prefix (the first quarter of history, so the system prompt and earliest exchanges stay byte-identical for prompt caching), elides low-scoring segments in place into[[elided …]]pointers, and moves the originals into the store where theexpandtool retrieves them. Requirescompaction-mode≠off; undershadowit runs report-only (the honest would-save numbers, nothing applied). User messages are never compacted; assistant/tool-result content is rewritten in place with roles, ToolCalls, and timestamps preserved.Automatic compaction (
jev-autocompact+jev-autocompact-percent) — whenjev-autocompactis enabled (bool, defaultfalse) and the focused agent's context usage crossesjev-autocompact-percent(default99, valid 1–100) of the context window, the same compaction runs automatically. One run per crossing: the trigger re-arms once usage falls back below (percent − 9)% — typically right after a compaction shrank the history — or when/newstarts a fresh conversation.session.CompactContextcore — the shared engine both paths call: frozen prefix (25%, the system prompt at index 0 always intact), user-message immunity, in-place content replacement (ToolCalls structurally untouched), fail-open mid-walk (a scorer error stops the walk; already-rewritten messages stay valid with resolvable pointers, and the error reports how far the walk got — compaction never breaks a session), and token before/after/saved accounting inCompactionReport.Store id-space unification — the elide-id counter moved into
compaction.Store(NextID/Putexported), so tool-output and history compaction mint pointers from one id space andelide-<n>always names exactly one original.New tests:
internal/session/compact_test.go(frozen prefix, user immunity, in-place elision, shadow report, fail-open, token math),internal/tui/compaction_cmd_test.go(command, shared guard, auto-trigger + re-arm),internal/config/config_test.go(ResolveAutocompact),cmd/late/compaction_runner_test.go(runner wiring + persistence). Docs: quickstart "Context compaction" extended, README feature bullet,late -hnote under "Context compaction".Test plan
go build ./...— greengo vet ./...— greengo test ./... -race -count=1— all packages green (no environmental failures on this run;internal/pluginpassed in 16s)gofmt -lclean on every changed fileinternal/tui/infobar_test.go— 11 tests: toggle persistence, save-failure surfacing, layout row reservation, rendered segments, omitted-when-unknown segments, hidden-when-disabled, single-line truncation, uptime formatting, compaction headroom math, agent-type mapping, command listinginternal/tui/timestamps_test.go— 3 tests: prefix rendering, toggle persistence, command listinginternal/session/timestamp_test.go— 2 tests: RFC3339 stamping, caller-timestamp preservationinternal/executor/compaction_test.go— 5 tests: enabled relocates + expand retrieves, shadow mode, off mode, fail-open + small results, compactor guards (incl. expand never re-compacted)internal/tool/expand_tool_test.go— 4 tests: metadata, execute round-trip, empty store, call stringinternal/compaction/— 54 tests across client (batching, 429/Retry-After, fail-open, fleet limiter), pipeline (end-to-end, fail-open logging), providers (resolution precedence, key files), relocate (threshold boundaries, elision IDs, store round-trip, concurrency), segment (paragraph splitting, size caps, multibyte), shadow (append/replay, malformed lines, concurrent appends)internal/config/config_test.go— 6 new tests: threshold/mode resolution, load + JSON round-trip for both featuresNotes
install-dev.shnot run in this step (the installed build is local/full; handled in a later step).