Opentelemetry integration with Grafana Mimir

[Chaitan1991]

Hi Folks,

I am working on Grafana mimir and opentelemetry, when I am trying to send metrics from postman to mimir via opentelemetry I am putting below configs in opentelemtry,

config:
  exporters:
    logging: {}
    otlphttp/mimir:
      endpoint: http://mimir-distributor-url.namespace:4318/otlp



service:
    telemetry:
      metrics:
        address: 0.0.0.0:8888
    extensions:
      - health_check
      - memory_ballast
    pipelines:
      logs:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
      metrics:
        exporters:
          - otlphttp/mimir
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp

but I am getting below error on opentelemtry logs,

2023-05-17T16:08:21+05:30 2023-05-17T10:38:21.543Z error exporterhelper/queued_retry.go:176 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/mimir", "error": "max elapsed time expired failed to make an HTTP request: Post "(http://mimir-distributor:4318/v1/metrics)\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 1

could you please help me configure it on open telemetry and mimir to receive metrics on Grafana Mimir?

[jpkrohling]

@Chaitan1991, could you please double-check the endpoint configuration? The use of [] seems odd there:

      endpoint: [http://mimir-distributor-url.namespace:4318/otlp]

This reflects in the error message:

Post "http:///v1/metrics": context

[Chaitan1991]

@jpkrohling , sorry the things were misplaced, I have corrected the url, do you think otlphttp/mimir is configurable with on open telemetry and mimir.

As I see the for mimir distributor otlp port 4318 is not enabled as theres no config added in the mimir helm chart, did you ever got chance to test out this?

Also, what configurations should needed for mimir for otlphttp?

I have seen in tempo helm chart, some configuration has added to enable otlp traces but same is not on mimir.

otlp:
    http:
      # -- Enable Tempo to ingest Open Telemetry HTTP traces
      enabled: true
      # -- HTTP receiver advanced config
      receiverConfig: {}=

[jpkrohling]

We use the OTLP with Grafana Cloud Metrics, which is backed by Mimir. That said, I haven't configured Mimir myself to get this working, so I would need input from someone else. Perhaps @kovrus or @gouthamve would be able to help you?

[Chaitan1991]

Sure, Thanks, I will wait for there response.

[Chaitan1991]

Hi @kovrus and @gouthamve, could you please test this out if its working on your end "otlphttp/mimir" ?

[mcabrito]

@Chaitan1991 I'm on the some page.
Did you progress on this issue?

[Ca-moes]

Answering for similar error output, but different cause.

On my side it was a network error. When using the otlphttp exporter to send metrics to Mimir in a different cluster, Mimir was also in a different AWS VPC, and there was no VPC peering established for connection between private subnets.