Skip to content

chore(release): staging to production - 2026.03.14 #1018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maxtechera merged 2 commits into from
Mar 14, 2026
Merged

chore(release): staging to production - 2026.03.14 #1018

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e74d66d
fix(documentstore): restore chunk saving and add memory optimization
maxtechera Mar 14, 2026
7fcad74
Merge pull request #1017 from the-answerai/fix/docstore-save-pipeline
maxtechera Mar 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 24 additions & 21 deletions packages/server/src/services/documentstore/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,10 @@ const syncAndRefreshChunks = async (storeId: string, fileId: string, userId: str
)
persistedChunks += batch.length
persistedChars += batch.reduce((acc: number, chunk: IDocument) => acc + (chunk.pageContent?.length ?? 0), 0)
// Free memory: allow GC to reclaim saved chunks
for (let j = i; j < i + batch.length; j++) {
docs[j] = null as any
}
} catch (batchError) {
saveFailed = true
break
Expand Down Expand Up @@ -959,19 +963,24 @@ export const previewChunks = async ({ appDataSource, componentNodes, data, orgId

// if -1, return all chunks
if (data.previewChunkCount === -1) data.previewChunkCount = totalChunks
// return all docs if the user ask for more than we have
// return all docs if the user asks for more than we have
if (totalChunks <= (data.previewChunkCount || 0)) data.previewChunkCount = totalChunks
const previewChunkCount = Math.max(0, data.previewChunkCount || 0)
if (previewChunkCount > 0) {
docs = docs.slice(previewChunkOffset, previewChunkOffset + previewChunkCount)
} else {

// Apply pagination only when previewChunkCount is explicitly set
// undefined/null → return all docs (original behavior, needed by save path)
if (data.previewChunkCount != null && data.previewChunkCount > 0) {
docs = docs.slice(previewChunkOffset, previewChunkOffset + data.previewChunkCount)
} else if (data.previewChunkCount != null && data.previewChunkCount === 0) {
docs = []
}
// else: previewChunkCount is undefined → return all docs

const effectiveCount = data.previewChunkCount ?? totalChunks

return {
chunks: docs,
totalChunks: totalChunks,
previewChunkCount: previewChunkCount,
previewChunkCount: effectiveCount,
previewChunkOffset: previewChunkOffset
}
} catch (error) {
Expand Down Expand Up @@ -1196,18 +1205,8 @@ const _saveChunksToStorage = async (
//step 1: restore the full paths, if any
await _normalizeFilePaths(appDataSource, data, entity, orgId)

//step 2: split the file into chunks
const response = await previewChunks({
appDataSource,
componentNodes,
data,
isPreviewOnly: false,
user: data.user,
orgId,
workspaceId,
subscriptionId,
usageCacheManager
})
//step 2: split the file into chunks (call _splitIntoChunks directly, not previewChunks)
const docs = await _splitIntoChunks(appDataSource, componentNodes, data, data.userId, data.organizationId)

//step 3: remove all files associated with the loader
const existingLoaders = JSON.parse(entity.loaders)
Expand Down Expand Up @@ -1292,14 +1291,14 @@ const _saveChunksToStorage = async (
})
).map((chunk) => chunk.id)

if (response.chunks) {
if (docs.length > 0) {
//step 7: save new chunks first (safe delete timing — delete AFTER all saved)
let persistedChunks = 0
let persistedChars = 0
let saveFailed = false

for (let i = 0; i < response.chunks.length; i += SAVE_BATCH_SIZE) {
const batch = response.chunks.slice(i, i + SAVE_BATCH_SIZE)
for (let i = 0; i < docs.length; i += SAVE_BATCH_SIZE) {
const batch = docs.slice(i, i + SAVE_BATCH_SIZE)
try {
await Promise.all(
batch.map(async (chunk: IDocument, localIndex: number) => {
Expand All @@ -1320,6 +1319,10 @@ const _saveChunksToStorage = async (
)
persistedChunks += batch.length
persistedChars += batch.reduce((acc: number, chunk: IDocument) => acc + (chunk.pageContent?.length ?? 0), 0)
// Free memory: allow GC to reclaim saved chunks
for (let j = i; j < i + batch.length; j++) {
docs[j] = null as any
}
} catch (batchError) {
saveFailed = true
break
Expand Down
Loading