Repository navigation
Stream documents in ExportCollection instead of buffering the whole collection - #170
Open
ChrisJr404 wants to merge 1 commit into
Open
ChrisJr404 wants to merge 1 commit into
ChrisJr404 wants to merge 1 commit into
Conversation
…ollection ExportCollection loaded every document into memory and marshalled them in one shot, which gets expensive for large collections. Iterate the documents and write them to the file one at a time so memory stays flat regardless of collection size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #130.
ExportCollectionwas callingFindAlland building a[]map[string]interface{}of the entire collection before a singlejson.Marshal, so peak memory grew with the collection size. This switches it toIterateDocsand writes each document straight to a buffered writer, opening with[, joining docs with,and closing with], so memory stays flat no matter how big the collection is.The output is the same JSON array as before, so existing exports and
ImportCollectionkeep working. Added a test that exports a populated collection and an empty one and checks both come back as well-formed JSON arrays with the expected element counts.