MaxMind databases not propagated correctly within the Search Head Cluster

[NicolasIannuzzi]

Ciao Team,

I configured the Maxmind app within a Splunk Cloud environment, and what I found is that on the Search Head where I made the configurations, the databases were downloaded correctly and the geoip command works as expected, while on the other cluster members, no configurations are present and the geoip command cannot find the databases:

“Error in ‘geoip’ command: FileNotFoundError at ”/opt/splunk/etc/apps/geoip/bin/geoip_command.py“, line 164 : Database not found: /opt/splunk/etc/apps/geoip/local/data/GeoLite2-Country.mmdb”

It appears that the configurations are not being propagated across the cluster and, as a result, the databases are not being downloaded and updated regularly.

Thank you,
Nicolà

[horgh]

Are you able to confirm whether the config is populated on the failing cluster member? I think the app should be replicating the configs already. I believe you could check this by looking at the app's config page from a failing cluster member and seeing whether the database list is populated there.

Do you still see the error after some time has passed? We currently run the updater once an hour, so I'm wondering if it started working on all cluster members.

I think what may be happening is that the config replicated but the database updater hasn't triggered yet on cluster members other than the one where the config was entered. We trigger the updater on save but that doesn't apply to cluster members, only the one that receives the initial save. I'm not sure there is a way to trigger it on all cluster members immediately unfortunately.

If that is the problem, potentially we could reduce the time between updater checks. Another option would be to provide a search command that would run the updater which could potentially be triggered either on save or manually if necessary. I'm also considering whether we could replicate the databases with Splunk's knowledge bundle support, which may also help with this.

[NicolasIannuzzi]

Ciao Horg,
Even though it’s in the cloud, I was able to access each node in the cluster individually via the UI, and the configurations were indeed propagated (account and database configured). What isn’t working is the download and update of the databases.

Given how this is designed, my guess is that the upgrade in inputs.conf isn’t working because Splunk delegates the operation to only one member at a time rather than to every node in the cluster, and I believe this behavior is expected.
I understand that the run_only_one option is essentially ignored by Splunk Cloud, and it’s not fully supported on-premises either.

BR,
Nicolà

[horgh]

Hmm, yeah I see in the spec that run_only_one is documented as not currently supported. That could potentially be the cause, but I'm not sure. I see some evidence that this setting does work though, e.g.

• https://community.splunk.com/t5/Deployment-Architecture/DB-inputs-only-running-on-Search-Head-Cluster-Captain/m-p/671501 / https://splunkbase.splunk.com/app/2686 seems to use it (it's set true/false in some of its inputs)
• A Splunk employee suggests using it here: https://community.splunk.com/t5/Monitoring-Splunk/How-to-run-a-script-only-on-one-search-head-in-a-SH-cluster/m-p/642409
• https://splunkbase.splunk.com/app/5579 seems to use it too

I wonder if the spec docs are out of date.

To confirm whether the database update input is running on all search heads, I think you could search in the _internal index for geoip.log or for database update, which I believe will show what hosts it is running on. It will log when it succeeds or fails.

Another possible cause: When you see the error, does it report anything about the hostname(s) it's failing on? It occurred to me that it could be indexer nodes erroring rather than search heads. Previously when I encountered this, the error showed hostnames like "idx-*" in it. The modular input doesn't run on indexers right now. I am planning to add support for indexers but it's not complete yet.

If it is the indexers causing the issue, you can add | localop before the geoip command which will make it execute on a search head rather than an indexer and should avoid the error.

By the way, thank you for trying out the app and your interest! And for helping me understand what's going on with this error :-).

[NicolasIannuzzi]

Ciao Horgh,
I checked the internal logs and didn't see any errors. Basically, a single search head handles the update request and runs the script.
This would confirm that Splunk Cloud does not parallelize the activity.

As for the add-ons you gave me as examples, they actually seem to use this methodology. In particular, in the db_inputs.conf for db-connect and inputs.conf in the other.

I noticed in the second link you shared with me that it asks what is meant for “script.” In your case, you’re running a command from inputs.conf and don’t have a [script: ] stanza:

• https://help.splunk.com/en/data-management/splunk-enterprise-admin-manual/10.2/configuration-file-reference/10.2.4-configuration-file-reference/inputs.conf#scripted-input-0

I don't know if this might affect the behavior of the parameter

[horgh]

Regarding the script stanza difference: We're using a modular input which from my understanding is roughly equivalent to a scripted input (docs about the custom inputs). I can't find in the docs exactly how modular/scripted inputs are supposed to behave in a SHC in terms of running on just one SH or all. My understanding was that each SH would run the input but clearly you're seeing the opposite!

Some questions for you:

• When you searched your internal logs, did you see logs from the app's updater at all? I would expect you to see logs about successful updates from one SH at least. This section has a sample search but I think you'd need to expand it beyond stderr. If you can't see any logs from the updater at all, I wonder if your search could be adjusted to show the logs, and maybe that would tell us more. There should be log lines like Database update completed successfully from whichever SH is running it.
• Is there anything more to show in the error message? I'm just wondering if there's a hostname showing anywhere and want to rule out indexers being involved.
• Do you know if you're on Splunk Cloud Victoria or Classic? I assume Victoria, but just want to be sure.
• When you see the error, does adding | localop fix it? e.g. | localop | geoip databases=GeoLite2-ASN. Probably not, but I just want to rule it out.

One thing I've noticed is that I don't have run_only_once = false in the inputs.conf.spec. I'm wondering if the lack of that could be causing Splunk Cloud to default to run_only_once = true. It seems pretty unlikely, but I'm thinking of doing a release adding it to the spec file to try it.

[horgh]

#80 has the spec change. I'll probably do a release tomorrow with it.

[horgh]

I released v1.1.1. Would you be able to try that out and see if it runs the updates on all the search heads please?

[NicolasIannuzzi]

Hi, I performed the update, but it seems that the DB update job is still running on only one search head. This is because I see that the geoip.log file is generated by only one node in the cluster, while for the others we have neither logs nor errors of any kind.
To answer your questions, yes we are on Splunk Victoria Experience and I don't see any changes in using | localop

[NicolasIannuzzi]

I was noticing that the add-on proposed as possibly conflicting with yours: TA-geoip2 and our old add-on seckit_sa_geolocation have a configuration file distsearch.conf.

In this configuration it seems that there are some regex to control the replication for .mmdb via replicationAllowlist stanza: https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.4/configuration-file-reference/10.4.0-configuration-file-reference/distsearch.conf#replication-allow-list-options-0

So the strategy is not "I run the update on all SH at the same time" but I make sure they are distributed.
Do you think it could be a valid alternative?

BR,
Nicolà

[horgh]

Thank you for trying it and for the additional info! Sorry that it didn't work 😢.

The distsearch.conf idea is interesting. From what I can tell, it may only control files we replicate from search heads to indexers though. However it may be worth looking into further.

I decided to ask on the Splunk community slack about the problem. I've been told that modular inputs indeed won't run on all SHC members, matching up with what you're seeing. I was given a couple suggestions for making this work. First, scripted inputs may run on all SHC members. Second, scheduling a search to run a command periodically might run on all SHC members. I'm thinking of trying converting the modular input to a scripted input first, as I believe that will be a fairly small change. I'll try out these ideas and let you know when I have a possible solution for you.

Thank you for helping debug this and for the ideas!

[horgh]

I found something else to try other than those things I mentioned in my last comment: There is another way to declare run_only_once = false. I released v1.1.2 with that change. Would you be able to try that one please? Thank you!

[NicolasIannuzzi]

No problem, Horgh—this is a really interesting topic.
I wasn't able to update the add-on because Splunk isn't notifying me about it, and I can't upload it manually. I'm not sure if this might be because the app.manifest lists version 1.1.1 instead of 1.1.2.

I'll try to figure out if there's another way to update it

[horgh]

Hmm, the app.manifest looks ok to me. I wonder if something on Splunk's side hadn't/hasn't refreshed yet. Maybe it will show up now or soon?

[NicolasIannuzzi]

Now I see it, I'll proceed with the add-on update

[NicolasIannuzzi]

Hi Horgh, it now seems that the update add-on is running erratically, as it doesn't follow a regular interval. Sometimes it runs every hour, and sometimes every 15 to 30 minutes.

In addition, even though it runs more frequently, it always runs on the same Search Heads and doesn't update the database on the others 😢