For smaller systems, interop between python and rust is much slower than encoding/optimisation. One possible method to improve this is to use [Apache Arrow](https://arrow.apache.org/docs/index.html) (`pyarrow` and `arrow-rs`) to exchange data directly. - [ ] Benchmark and profile current setup for `core.encode` - [ ] Duplicate `lib.rs` function using arrow - [ ] Compare bench and profile - [ ] Find out more about numpy interop with arrow (usability & performance).
For smaller systems, interop between python and rust is much slower than encoding/optimisation.
One possible method to improve this is to use Apache Arrow (
pyarrowandarrow-rs) to exchange data directly.core.encodelib.rsfunction using arrow