ISiloStatusOracle service stops sending Silo Status Events to ISiloStatusListener once a silo has joined cluster (status Active)

[olegbsky]

Listener:

public class SiloStatusListener : ISiloStatusListener
{
	public void SiloStatusChangeNotification(SiloAddress updatedSilo, SiloStatus status)
	{
		Logger.LogWarning($"Silo '{updatedSilo.ToParsableString()}' status: {status}");
	}
}

Startup code:

ISiloHost siloHost = BuildSilo(); // Using ADO.NET provider for Clustering (MySql)
var oracle = siloHost.Services.GetService<ISiloStatusOracle>();                
var result = oracle.SubscribeToSiloStatusEvents(new SiloStatusListener()); // Returns true

Steps to reproduce the issue:

1. Start Silo Fixing typo in README #1.

warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310110425' status: Created
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310110425' status: Joining
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310110425' status: Active

2. Start Silo Exclude code-gen files from repo #2.

warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310110425' status: Active
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310110682' status: Created
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310110682' status: Joining
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310110682' status: Active

3. Note that Silo Fixing typo in README #1 does not get Status updates about Silo Exclude code-gen files from repo #2.

4. Wait for some time and stop Silo Fixing typo in README #1 or Silo Exclude code-gen files from repo #2.

5. Neither Silo Fixing typo in README #1 nor Silo Exclude code-gen files from repo #2 receive new Silo Status Events.

Reproducible on Version 2.4.3.

[veikkoeeva]

@olegbsky Can you tell if this is a new problem that started to happen after the update to 2.4.3? If so, what was your previous version and is it possible to double-check by going back to that version?

Do you use reminders and do you observe an increased call frequency to the database?

How do you host your deployment currently? Kubernetes? Linux or Windows?

[olegbsky]

Unfortunately, I did not test it on any previous version. No, I do not use reminders in my app code.
Both silos run on my local Windows machine. I configured different sets of ports for them, but the same Clustering options.

[veikkoeeva]

@olegbsky Can you update to the latest 3.0 so we can see if the problem persists?

[veikkoeeva]

@olegbsky Oh, also, can you try LocalDb or SQL Server too? If you don't have SQL Server installed but have Visual Studio, it installs LocalDb too. The connection string is something like "Data Source=(localdb)\\MSSQLLocalDB;Initial Catalog=TestDatabase;Integrated Security=True;Persist Security Info=False;Pooling=True;MultipleActiveResultSets=True;Connect Timeout=60;Encrypt=False;TrustServerCertificate=False;".

[olegbsky]

I have checked configuration with SQL Server on 2.4.3 - no difference, I can see the same behavior.

With version 3.0.0 and SQL Server the behavior slightly changed, but still is not as desired:

1. Start Silo Fixing typo in README #1:

// Silo #1 output
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310213708' status: Created
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310213708' status: Joining
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310213708' status: Active

2. Start Silo Exclude code-gen files from repo #2:

// Silo #2 output
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11111@310213708' status: Active
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310213720' status: Created
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310213720' status: Joining
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310213720' status: Active

3. Silo Fixing typo in README #1 shows new events about Silo Exclude code-gen files from repo #2:

// Silo #1 output
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310213720' status: Joining
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310213720' status: Active

4. Stop Silo Exclude code-gen files from repo #2:

// Silo #2 output
warn: MyApp.Silo.Program[0]
Silo '10.0.75.1:11112@310213720' status: ShuttingDown

Note: Silo #1 does not show status updates anymore, even if I restart Silo #2 several times.

[sergeybykov]

@olegbsky This sounds very strange. Can you do the same with the Azure Storage clustering provider for a comparison?

[ReubenBond]

Closing due to inactivity - feel free to comment/open a new issue if this is still a problem