Grafana Mimir in-memory series sudden large spike

[KaloyanHr]

Hello,

Yesterday I encountered the following issue where my in-memory series of one of my Mimir instances spiked suddenly reaching the limit of max_series_per_user:

err="user=anonymous: per-user series limit of 20000000 exceeded (err-mimir-max-series-per-user).

Snippet of my config:

block_builder:
    scheduler_config:
        grpc_client_config:
            connect_backoff_base_delay: 5s
            connect_backoff_max_delay: 10s
            connect_timeout: 15s
            max_recv_msg_size: 419430400
            max_send_msg_size: 419430400
blocks_storage:
    backend: s3
    bucket_store:
        chunks_cache:
            backend: memcached
            memcached:
                connect_timeout: 15s
                max_idle_connections: 300
                timeout: 30s
        index_cache:
            backend: memcached
            memcached:
                connect_timeout: 15s
                max_idle_connections: 300
                timeout: 30s
        index_header:
            lazy_loading_concurrency: 8
            lazy_loading_concurrency_queue_timeout: 10s
            max_idle_file_handles: 2
        max_concurrent: 500
        max_concurrent_queue_timeout: 2m0s
        metadata_cache:
            backend: memcached
            memcached:
                connect_timeout: 15s
                max_idle_connections: 300
                timeout: 30s
    tsdb:
        flush_blocks_on_shutdown: true
        head_compaction_interval: 15m0s
        wal_compression_enabled: true
        wal_replay_concurrency: 3
compactor:
    compaction_interval: 5m0s
    deletion_delay: 2h0m0s
    first_level_compaction_wait_period: 5m0s
    max_closing_blocks_concurrency: 2
    max_opening_blocks_concurrency: 4
    sharding_ring:
        heartbeat_timeout: 2m0s
    symbols_flushers_concurrency: 4
distributor:
    max_recv_msg_size: 419430400
    remote_timeout: 1m0s
    ring:
        heartbeat_timeout: 2m0s
frontend:
    cache_results: true
    grpc_client_config:
        connect_backoff_base_delay: 5s
        connect_backoff_max_delay: 10s
        connect_timeout: 15s
        grpc_compression: snappy
        max_recv_msg_size: 419430400
        max_send_msg_size: 419430400
    max_outstanding_per_tenant: 500
    query_sharding_target_series_per_shard: 10000
    results_cache:
        backend: memcached
        memcached:
            connect_timeout: 15s
            max_idle_connections: 300
            timeout: 30s
    scheduler_worker_concurrency: 16
frontend_worker:
    grpc_client_config:
        connect_backoff_base_delay: 5s
        connect_backoff_max_delay: 10s
        connect_timeout: 15s
        grpc_compression: snappy
        max_recv_msg_size: 419430400
        max_send_msg_size: 419430400
    query_scheduler_grpc_client_config:
        connect_backoff_base_delay: 5s
        connect_backoff_max_delay: 10s
        connect_timeout: 15s
        grpc_compression: snappy
        max_recv_msg_size: 419430400
        max_send_msg_size: 419430400
ingester:
    ring:
        heartbeat_timeout: 2m0s
        num_tokens: 512
        zone_awareness_enabled: true
ingester_client:
    grpc_client_config:
        connect_backoff_base_delay: 5s
        connect_backoff_max_delay: 10s
        connect_timeout: 15s
        grpc_compression: snappy
        max_recv_msg_size: 419430400
        max_send_msg_size: 419430400
limits:
    align_queries_with_step: true
    compactor_blocks_retention_period: 15d
    ingestion_burst_size: 3000000
    ingestion_rate: 2e+06
    max_global_series_per_user: 20000000
    max_label_names_per_series: 60
    max_query_lookback: 15d
    max_query_parallelism: 240
    max_total_query_length: 15d
    query_sharding_total_shards: 32
querier:
    max_concurrent: 32
query_scheduler:
    grpc_client_config:
        connect_backoff_base_delay: 5s
        connect_backoff_max_delay: 10s
        connect_timeout: 15s
        grpc_compression: snappy
        max_recv_msg_size: 419430400
        max_send_msg_size: 419430400
    max_outstanding_requests_per_tenant: 5000
ruler:
    query_frontend:
        grpc_client_config:
            grpc_compression: snappy
            max_recv_msg_size: 419430400
            max_send_msg_size: 419430400
    ruler_client:
        connect_backoff_base_delay: 3s
        connect_backoff_max_delay: 10s
        connect_timeout: 15s
        grpc_compression: snappy
        max_recv_msg_size: 419430400
        max_send_msg_size: 419430400
server:
    grpc_server_keepalive_timeout: 1m0s
    grpc_server_max_concurrent_streams: 0
    grpc_server_max_connection_age: 1m0s
    grpc_server_max_connection_age_grace: 5m0s
    grpc_server_max_connection_idle: 1m0s
    grpc_server_max_recv_msg_size: 419430400
    grpc_server_max_send_msg_size: 419430400
    grpc_server_min_time_between_pings: 30s
    grpc_server_num_workers: 200
    http_server_idle_timeout: 5m10s
    http_server_read_timeout: 5m10s
    http_server_write_timeout: 5m10s

Attaching screenshots of graphs from that period:

The way I fixed it was by rollout restarting my ingesters zone per zone. Since doing that on all ingesters it has been stable.

Additionally, this instance sees less traffic than another instance I'm running and I haven't seen such an issue on it so far.

Grafana Mimir app version - 2.16.0
Deployed in distributed mode and running as single-tenant.

Can you please advise why this happened and if there's any config options which could avoid such issues in the future as I couldn't find any ?

Let me know if you need additional info.

Thank you

[deniszh]

Hi @KaloyanHr
Issue lies not in Mimir but in your data. You can use cardinality dashboards https://github.com/cerndb/grafana-mimir-cardinality-dashboards to find out which metrics and/or labels causing this.