Off-by-one-thousand errors with valid time columns

[refset]

Sometimes when programming it's tempting to handle epoch timestamps using simple number types, but be aware there are pitfalls!

For example, when using plain numbers with XTDB's _valid_from and _valid_to columns, it can be quite confusing to hit an off-by-one off-by-one-thousand error. Eventually you might discover that some of your data was only valid in early 1970, or sometime after the year 57000AD. Best not to confuse your millis and your seconds 🙂

XTDB might eventually offer guardrails here - e.g. by throwing errors or returning warnings (suppressable via configuration) whenever it encounters valid-from/to before 1971 / after 57K respectively - but for now I hope this is a useful pointer.

[seancorfield]

I would have thought there were several domains where data would be valid prior to 1971?

[jarohen]

XT shouldn't be guessing about what is and what isn't user error, IMO - if a user's asked us to do something (assuming it's a valid request), we should respect that request, even if it seems strange to us.

You wouldn't want a compiler asking you whether you really wanted to execute that code, for example 🙂

[refset]

We'll have to wait to see how many people shoot themselves in the foot.

It's worth mentioning also that, currently, data inserted into the distant future may slow down the performance of as-of-now queries (as it has to be scanned and filtered out each time), even if you DELETE it from system-time.