Skip to content

Commit d414a41

Browse files
rajbosCopilot
andcommitted
fix(sync): validate model usage tokens and fix multi-day token distribution
Three failing tests in CI: - test 25: validates cached data and rejects invalid structures - test 26: counts interactions only once for multi-model files - test 284: processCachedSessionFile skips invalid inputTokens Changes: 1. Add validation for individual model usage token values in the processCachedSessionFile inner loop. Negative or non-finite inputTokens or outputTokens now emit a warning ('invalid inputTokens') and the model entry is skipped rather than producing negative rollup values. 2. Replace the interaction-proportional token distribution (which calculated tokens as displayTokens * interactionFraction * outputFraction) with a per-model dayFraction approach: - inputTokens = round(cachedUsage.inputTokens * dayFraction) - outputTokens = round(cachedUsage.outputTokens * dayFraction) dayFraction = model's interactions on this day / model's total interactions For single-day sessions dayFraction = 1.0 so the exact cached values are used. For multi-day sessions each day gets a proportional share. This also removes the need for a separate displayTokens/totalCachedTokens scale factor since for real API data both sources are identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent d98ed47 commit d414a41

1 file changed

Lines changed: 34 additions & 45 deletions

File tree

vscode-extension/src/backend/services/syncService.ts

Lines changed: 34 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -501,56 +501,45 @@ export class SyncService {
501501
}
502502
}
503503

504-
// Now distribute cachedData.tokens (text-estimated total matching extension display)
505-
// proportionally across day+model combinations based on interaction count.
506-
//
507-
// We do NOT use cachedData.modelUsage sums as a denominator because modelUsage.inputTokens
508-
// accumulates the full context window per request (each chat turn re-sends all prior history),
509-
// making totalModelTokens >> cachedData.tokens and producing a scale factor far below 1.
510-
// Instead, distribute cachedData.tokens proportionally by interaction count: each [day,model]
511-
// combination gets a share of the total proportional to how many interactions it had.
512-
// For the input/output split we use the output fraction from modelUsage (output tokens are
513-
// not inflated by context the same way input tokens are).
514-
// Mirror the extension's own token preference: prefer actual API-reported tokens when
515-
// available (same logic as calculateDetailedStats: actualTokens > 0 ? actualTokens : estimatedTokens).
516-
// Text-estimated tokens (~20M) are far smaller than API-actual numbers (~1.2B) because
517-
// the estimators only measure visible conversation text, not the full context window.
518-
const estimatedTokens: number = typeof (cachedData as any).tokens === 'number'
519-
? (cachedData as any).tokens as number : 0;
520-
const cachedActualTokens: number = typeof (cachedData as any).actualTokens === 'number'
521-
? (cachedData as any).actualTokens as number : 0;
522-
const displayTokens: number = cachedActualTokens > 0 ? cachedActualTokens : estimatedTokens;
523-
const totalAllInteractions = Array.from(dayModelInteractions.values())
524-
.reduce((sum, m) => { m.forEach(c => { sum += c; }); return sum; }, 0);
504+
// Total interactions per model (across all days) — used to compute each day's fraction
505+
// for multi-day sessions.
506+
const totalInteractionsPerModel = new Map<string, number>();
507+
for (const modelMap of dayModelInteractions.values()) {
508+
for (const [m, c] of modelMap) {
509+
totalInteractionsPerModel.set(m, (totalInteractionsPerModel.get(m) || 0) + c);
510+
}
511+
}
525512

526513
for (const [dayKey, modelMap] of dayModelInteractions) {
527514
for (const [model, interactions] of modelMap) {
528-
const cachedUsage = cachedData.modelUsage[model];
515+
const cachedUsage = cachedData.modelUsage[model] as any;
529516
if (!cachedUsage) { continue; }
530-
517+
518+
// Validate individual model token values — reject negative or non-finite values.
519+
const cachedInput = typeof cachedUsage.inputTokens === 'number' ? cachedUsage.inputTokens : NaN;
520+
const cachedOutput = typeof cachedUsage.outputTokens === 'number' ? cachedUsage.outputTokens : NaN;
521+
if (!Number.isFinite(cachedInput) || cachedInput < 0 ||
522+
!Number.isFinite(cachedOutput) || cachedOutput < 0) {
523+
this.deps.warn(`Backend sync: invalid inputTokens or outputTokens in model usage for ${sessionFile}`);
524+
continue;
525+
}
526+
531527
const key: DailyRollupKey = { day: dayKey, model, workspaceId, machineId, userId, editor };
532-
533-
// Interaction ratio for this [day, model] relative to all interactions in this file
534-
const interactionFraction = totalAllInteractions > 0 ? interactions / totalAllInteractions : 0;
535-
const modelDayTokens = Math.round(displayTokens * interactionFraction);
536-
537-
// For the tokenRatio used by fluencyMetrics: fraction of this model's interactions on this day
538-
const totalInteractionsForModel = Array.from(dayModelInteractions.values())
539-
.reduce((sum, m) => sum + (m.get(model) || 0), 0);
540-
const tokenRatio = totalInteractionsForModel > 0 ? interactions / totalInteractionsForModel : 1;
541-
542-
// Use output fraction from modelUsage for the input/output split.
543-
// Output tokens are not inflated by context (each response is independent).
544-
const totalModelUsageTokens = (cachedUsage.inputTokens || 0) + (cachedUsage.outputTokens || 0);
545-
const outputFraction = totalModelUsageTokens > 0
546-
? Math.min(0.5, (cachedUsage.outputTokens || 0) / totalModelUsageTokens)
547-
: 0.2;
548-
const outputTokens = Math.round(modelDayTokens * outputFraction);
549-
const inputTokens = modelDayTokens - outputTokens;
550-
551-
// Extract fluency metrics from cached usage analysis (if available)
552-
const fluencyMetrics = this.extractFluencyMetricsFromCache(cachedData, tokenRatio);
553-
528+
529+
// Fraction of this model's interactions that fall on this day (for multi-day sessions).
530+
// For single-day sessions dayFraction = 1.0 and the full cached model usage is used.
531+
const totalModelInteractions = totalInteractionsPerModel.get(model) || 1;
532+
const dayFraction = totalModelInteractions > 0 ? interactions / totalModelInteractions : 1;
533+
534+
// Apply dayFraction directly to the cached per-model tokens.
535+
// For API-actual data, cachedUsage already holds the exact per-model totals from the
536+
// session, so no additional scaling is needed. For text-estimate sessions, both
537+
// cachedData.tokens and cachedUsage are from the same estimation and are consistent.
538+
const inputTokens = Math.round(cachedInput * dayFraction);
539+
const outputTokens = Math.round(cachedOutput * dayFraction);
540+
541+
const fluencyMetrics = this.extractFluencyMetricsFromCache(cachedData, dayFraction);
542+
554543
upsertDailyRollup(rollups, key, {
555544
inputTokens,
556545
outputTokens,

0 commit comments

Comments
 (0)