Bug: Some metrics return no data when querying time periods smaller than 24 hours

[juliajohannesen]

What is the bug?

This may be a configuration issue, but I'm experiencing an issue where querying certain metrics for time periods smaller than 24 hours returns no data, but other metrics show complete data.

Screenshot of the last 24 hours:

Screenshot of the last hour:

How to reproduce it?

1. Start up Mimir with the attached config.
2. Set up alloy or similar to push node-exporter metrics to Mimir.
3. Query metrics.
4. Data for time periods less than 24 hours will return nothing.

What did you think would happen?

I would expect to be able to query metrics regardless of the time range.

What was your environment?

mimir --version:

Mimir, version 2.17.1 (branch: HEAD, revision: c5ebb8f3)
  go version:       go1.24.6
  platform:         linux/amd64

compose.yml:

services:
  consul:
    image: docker.io/consul:1.15
    container_name: mimir_consul
    command: ["agent", "-dev" ,"-client=0.0.0.0", "-log-level=info"]
    networks:
      - network

  mimir:
    image: docker.io/grafana/mimir:latest
    container_name: mimir
    command: ["-config.file=/etc/mimir.yaml"]
    networks:
      - network
    ports:
      - "0.0.0.0:6100:8080"
    volumes:
      - ./config/mimir.yaml:/etc/mimir.yaml
      - ./storage/compactor:/data/compactor
      - ./storage/ingester:/data/ingester
      - ./storage/querier:/data/querier

networks:
  network:
    name: mimir_network
    driver_opts:
      com.docker.network.driver.mtu: 1420

config/mimir.yaml:

target: all,alertmanager,overrides-exporter

common:
  storage:
    backend: s3
    s3:
      endpoint: <...>
      bucket_name: <...>
      access_key_id: <...>
      secret_access_key: <...>

alertmanager_storage:
  storage_prefix: alertmanager

blocks_storage:
  storage_prefix: blocks
  tsdb:
    dir: /data/ingester
    flush_blocks_on_shutdown: true
    ship_interval: 5m
    retention_period: 336h
  bucket_store:
    sync_dir: /data/querier

compactor:
  compaction_interval: 30m
  data_dir: /data/compactor
  deletion_delay: 2h
  first_level_compaction_wait_period: 25m

ingester:
  ring:
    final_sleep: 0s
    num_tokens: 512
    kvstore:
      store: consul
      consul:
        host: mimir_consul:8500
    replication_factor: 1

limits:
  compactor_blocks_retention_period: 1y
  max_global_exemplars_per_user: 100000
  native_histograms_ingestion_enabled: true
  otel_metric_suffixes_enabled: true

multitenancy_enabled: true

ruler:
  poll_interval: 2s
  ring:
    kvstore:
      store: consul
      consul:
        host: mimir_consul:8500

ruler_storage:
  storage_prefix: ruler

store_gateway:
  sharding_ring:
    kvstore:
      store: consul
      consul:
        host: mimir_consul:8500
    replication_factor: 1

Any additional context to share?

No response

[LuxTheDude]

I am also experiencing this issue.

Specifically, this hits for the mixin dashboards for Mimir itself.

The 'Mimir / Writes' dashboard shows no data for 'Distributore - Requests / sec' among some, when the timeframe is less that 24h

[56quarters]

This is most likely an issue with how the dashboard is configured in Grafana, not Mimir itself. Can you get the query being run along with the step being used?

[juliajohannesen]

Query is sum(irate(node_cpu_seconds_total{instance="icarus",job="node-exporter", mode="system"}[$__rate_interval])) / scalar(count(count(node_cpu_seconds_total{instance="icarus",job="node-exporter"}) by (cpu))).

For a working query, inspect element reports a step of 240, Grafana logs reports 2m0s gets submitted to the datasource.
For a broken query, inspect element reports a step of 240, Grafana logs reports 20s gets submitted to the datasource.

Curl requests for both:

$ curl -H 'X-Scope-OrgID: personal' 'http://10.0.0.2:6100/prometheus/api/v1/query_range?query=sum%28irate%28node_cpu_seconds_total%7Binstance%3D%22icarus%22%2Cjob%3D%22node-exporter%22%2C%20mode%3D%22system%22%7D%5B2m15s%5D%29%29%20%2F%20scalar%28count%28count%28node_cpu_seconds_total%7Binstance%3D%22icarus%22%2Cjob%3D%22node-exporter%22%7D%29%20by%20%28cpu%29%29%29&start=2025-09-30T17:12:46.188Z&end=2025-10-02T14:38:17.222Z&step=2m0s&limit=1372
{"status":"success","data":{ ... }}
$ curl -H 'X-Scope-OrgID: personal' 'http://10.0.0.2:6100/prometheus/api/v1/query_range?query=sum%28irate%28node_cpu_seconds_total%7Binstance%3D%22icarus%22%2Cjob%3D%22node-exporter%22%2C%20mode%3D%22system%22%7D%5B1m0s%5D%29%29%20%2F%20scalar%28count%28count%28node_cpu_seconds_total%7Binstance%3D%22icarus%22%2Cjob%3D%22node-exporter%22%7D%29%20by%20%28cpu%29%29%29&start=2025-10-02T02:41:55.501Z&end=2025-10-02T09:54:14.907Z&step=20s&limit=1372'
{"status":"success","data":{"resultType":"matrix","result":[]}}

[jdeastwood]

I've also found the same behaviour it seems to be something to do with when the counter values are being updated, which looks like every 60 seconds (even if the metric is collected every 15s). rate shows nothing unless it actually sees a change. Example queries:

• rate(cortex_prometheus_rule_evaluations_total{}[60s]) <- will show no results
• rate(cortex_prometheus_rule_evaluations_total{}[62s]) <- will show results as expected.

Is it possible that mimir is coincidentally updating the counter values in perfect alignment with the intervals being used in grafana?