[Hackerberg43]

Description

Data export page:

• Added loading indicators that are activated while retrieving table data and while the export request is running.
  Before, it was not visible whether the app was loading, becoming unresponsive or simply not doing anything. (closes Data export: improve performance and feedback of data export page #1396)
• Added a list input to select alternative export formats.

asset/datapoint/export resource :

• Added an extra csv export format that was previously suggested in Enhancement/data export #1557 (closes Enhancement/alternative data export format ( postgres update req. ) #1574)

• Added 1 minute average aggregation option, potentially saves even more time post-processing the csv.

• Fix resource leak using try-with-resources on FileInputStream (a leak was observed while we tried multiple times extracting data that resulted in database timeouts, not sure if this is the direct cause though)

• Added migration script to add tablefunc extension to postgres, required for the new csv formatting.

Checklist

• 1. I've made "format" an optional query parameter in asset/datapoint/export to avoid an API breaking change. However, I'm not fully confident if this is the correct way. I did not find any usage of @DefaultValue in any of the existing api resources, while it seems a straightforward solution. I'm trying to create something similar to the optional parameters used in the datapoint query (gapfill, formula,..), but not sure if that is the best way either, because those did not get picked up by the automated API documentation.

• 2. Tested on a local deployment only with a 2 million datapoint database.

• 3. The format names are little lengthy, not sure how to improve.

[Copilot]

Pull Request Overview

This PR enhances the data export functionality by adding loading indicators to improve user experience and introducing multiple CSV export formats with database optimization. The changes address user feedback about unclear loading states and provide more flexible data export options.

• Added loading indicators to show export progress and prevent user confusion
• Introduced three CSV export formats: default, column-per-attribute, and column-per-attribute with 1-minute averages
• Fixed resource leak using try-with-resources pattern and added database migration for PostgreSQL tablefunc extension

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ui/app/shared/locales/nl/or.json	Added Dutch translation for "Export format" label
ui/app/shared/locales/en/or.json	Added English translation for "Export format" label
ui/app/manager/src/pages/page-export.ts	Enhanced UI with loading indicators and export format selection dropdown
model/src/main/java/org/openremote/model/datapoint/AssetDatapointResource.java	Added optional format parameter to export API endpoint
manager/src/main/resources/org/openremote/manager/setup/database/V20250724_01__addTableFunc.sql	Database migration to enable PostgreSQL tablefunc extension
manager/src/main/java/org/openremote/manager/datapoint/AssetDatapointService.java	Implemented multiple CSV export formats with SQL crosstab functionality
manager/src/main/java/org/openremote/manager/datapoint/AssetDatapointResourceImpl.java	Updated export endpoint to handle format parameter and fixed resource leak

[Hackerberg43]

https://github.com/openremote/vdl-es/issues/161

[Hackerberg43]

Intial comments/insights have been addressed, this should be ready for proper review.

[richturner]

As discussed @Hackerberg43 a frontend test would be great to get this finished @MartinaeyNL can you review the frontend code changes please.

[Ekhorn]

I have this commit running on our testrig right now with 200m datapoints. It is unable to export data, only gives timeouts. I'm currently narrowing down if this is caused by the DB bloat and not the export itself. Will keep you updated.

@Hackerberg43 Just checking that you're aware of the fact that @openremote/core sets the request timeout to 20 seconds (on the frontend). Increasing rest.setTimeout(20000); avoids these timeouts and it might make sense to increase this timeout limit (for datapoint exports), because downloading hundreds of megabytes or gigabytes of data at this scale starts to get throttled by connection speeds.

[Hackerberg43]

Ah good to know! I'm trying some of the queries directly into the testrig database right now, which take a couple of minutes to complete. I am currently trying to optimize/clean the database a bit to see if it can be brought down to a reasonable time. Otherwise it might need to be considered to move back to disk storage and decouple the exporting/downloading process.

[Hackerberg43]

So after some DB tuning our testrig is just able to export 2m datapoints before the timeout. While still not sufficient for our use case, I think for most projects this is completely fine? We might diverge to a separate solution in our custom project that performs export jobs in the background and exposes files in an export folder as downloadable.

So this is ready to test @Ekhorn.

[Ekhorn]

There seems to be an issue with the GitHub actions runner timing out on certain PRs. I'll check if I can get this resolved on my own first.

[Ekhorn]

Nice work @Hackerberg43!

I will approve this when the threads are resolved.

See #2020 (comment) and #2020 (comment)

[Hackerberg43]

@Ekhorn Done.
The sql injection fix required some code updating but done now. db migration file updated for today.

[Ekhorn]

With these changes the normal CSV export runs a little faster again (after the recent SQL injection defensive measure) and prevents filesystem issues, also the crosstab formats are a nice addition.

Thanks @Hackerberg43!