At them moment ella has no concept of exclusivity for a datastore. This means that you can start two servers pointing at the same location and they will overwrite each other's data.
A straightforward solution would be some kind of file lock, but that becomes impractical if the instances are pointing at a remote object store rather than a local filesystem.
An alternative would be to put the lock files in a dedicated location regardless of datastore path. This wouldn't prevent instances on two separate machines from both modifying a cloud store, but it would at least prevent two sessions on the same machine from conflicting.
Once this exclusivity is implemented, ella should also include a "read-only" mode to allow access to a datastore without having to make sure no instances are running.
The challenge there is that with the way the system is currently implemented, if there were a simultaneous "writer" and "reader" on the same datastore, the reader could end up with outdated state (e.g. trying to read a parquet file that no longer exists).
As such, it may be necessary to update the engine to verify the state on disk before certain operations.
At them moment ella has no concept of exclusivity for a datastore. This means that you can start two servers pointing at the same location and they will overwrite each other's data.
A straightforward solution would be some kind of file lock, but that becomes impractical if the instances are pointing at a remote object store rather than a local filesystem.
An alternative would be to put the lock files in a dedicated location regardless of datastore path. This wouldn't prevent instances on two separate machines from both modifying a cloud store, but it would at least prevent two sessions on the same machine from conflicting.
Once this exclusivity is implemented, ella should also include a "read-only" mode to allow access to a datastore without having to make sure no instances are running.
The challenge there is that with the way the system is currently implemented, if there were a simultaneous "writer" and "reader" on the same datastore, the reader could end up with outdated state (e.g. trying to read a parquet file that no longer exists).
As such, it may be necessary to update the engine to verify the state on disk before certain operations.