Telemetry Service: Logs

[tanmoysrt]

• Evaluate Quickwit if it's enough for us. Maybe some stress testing needed to for ingestion + search capability.
  • If fine, then we can copy the implementation from https://github.com/frappe/press-containers/tree/develop/telemetry_manager
  • If not, need to figure out a better solution. Elasticsearch is anyway the proven one, but becomes heavy with more an more historical data.
• Rebuild this service for log collection.
• Integrate it with pilot.
  • Should be able to ship all vm logs, pilot specific logs, pilot's audit log, crash log, all bench, site logs. Can we write our own parser, as part of admin process. If that's heavy, better to use a exporter.
  • If need daemon even, lets use fluentibit , filebeat is too heavy. This one also should work as non-root user.
  • Ship MariaDB, Postgres's slow query log, error log
  • Ship all NGINX logs
• Integrate with Atlas / Central (Need to get the requirements)
• Indexing : Quickwit has the limitation on data indexing. Elasticsearch also has similiar issue. Can we do something magical that we can change schema definition easily without much ops effort.
• It will be mutli-tenant. But, can be simple api token based. Central already do JWT minting with jwks, so that can be reused.
• Add support for retention config per client.

[tanmoysrt]

• https://github.com/VictoriaMetrics/VictoriaLogs
• https://github.com/VictoriaMetrics/sql-to-logsql

[tanmoysrt]

For Grove (LLM Service), Central, most things will be handled by pilot.