How to read ndjson file in assets folder

[jesscmoore]

I would like to use ndjson package to read a large ndjson file stored in the assets folder of my flutter app.

As a simple example I have copied your object.ndjson to assets/data folder. Here are relevant code parts:

pubspec.yaml:

flutter:

  assets:
    - assets/data/

main.dart:

import 'package:flutter/services.dart';
import 'package:ndjson/ndjson.dart';

...

void fetchData() async {
  final ndjsonSource = await rootBundle.loadString("assets/data/object.ndjson");

  final Stream<NdjsonLine> ndjson = ndjsonSource.parseNdjson();

  ...

}

This has the issue that "The method 'parseNdjson' isn't defined for the type 'String'..." because parseNdjson expects a stream. How can I use your ndjson flutter package to read a file in the assets folder?

[alexcmgit]

If you are loading the ndjson at once you can just:

  final ndjsonSource = Stream.fromIterable(<String>[await rootBundle.loadString("assets/data/object.ndjson")]);

  final Stream<NdjsonLine> ndjson = ndjsonSource.parseNdjson();

Keep in mind it is loading all the ndjson data at once, which kills the purpose of the ndjson which is to lazy-load JSON entries. But if you just want to make it run, the above should work.

[jesscmoore]

I resolved this by:

1. reading the assets file as ByteData with rootbundle.load(), writing the file to ApplicationDocumentsDirectory (if not already done).
2. the ndjson file with Stream<List> ndjsonSource = File().open() and then parsed with .parseNdjsonWithConverter().

[jesscmoore]

If you are loading the ndjson at once you can just:

final ndjsonSource = Stream.fromIterable([await rootBundle.loadString("assets/data/object.ndjson")]);

final Stream ndjson = ndjsonSource.parseNdjson();
Keep in mind it is loading all the ndjson data at once, which kills the purpose of the ndjson which is to lazy-load JSON entries. But if you just want to make it run, the above should work.

Thanks @alexcmgit I'll try this also and compare the load speed. I call this function in initState() as the app depends on the ndjson file loading.

[jesscmoore]

I run .parseNdjsonWithConverter() within compute() to isolate the process given the file size. However, as isolate() is unable to take a stream argument or access assets, I'll continue with the one-time write of the asset to appDocDir, that way I can run the .parseNdjsonWithConverter() within compute(). Seems like an inefficient hack, but it works.