The target silo became unavailable for message. The silo appears to be running.

[buddy-james]

I've upgraded to Orleans 2.0 and I'm able to successfully run the silo and client web application on my dev machine. The client application is able to connect to the silo, obtain grains, all seems to work fine. When i deploy the silo and client application to our dev server, I'm getting the "The target silo became unavailable for message" exception when trying to connect to the silo. I know that this can happen if the silo isn't yet ready for connections so we have a retry mechanism when connecting, however, the silo appears to be running and doing work (reminders are firing and such). I've read through https://dotnet.github.io/orleans/Documentation/Advanced-Concepts/Troubleshooting-Deployments.html to no avail.

Here is the code used to configure the silo
`private static ISiloHost CreateSiloHost()
{
var siloHostBuilder = new SiloHostBuilder();
var invariant = ConfigurationManager.ConnectionStrings["BackingStore"].ProviderName;
var connectionString = ConfigurationManager.ConnectionStrings["BackingStore"].ConnectionString;

        var appSettings = BuildApplicationSettings();

        Log.Logger = Startup.CreateLogger();

        siloHostBuilder.UseServiceProviderFactory(services =>
        {
            var builder = new ContainerBuilder();

            builder.RegisterLogger(Log.Logger, true);
            builder.RegisterModule<DataModule>();
            builder.RegisterModule<ServiceModule>();
            builder.Populate(services);
            var container = builder.Build();
            return container.Resolve<IServiceProvider>();
        })
        .AddSimpleMessageStreamProvider("SMSProvider")
        .Configure<ClusterOptions>(options =>
        {
            options.ClusterId = "ClusterID";
            options.ServiceId = "ServiceID";
        })
        .ConfigureEndpoints(siloPort: 11111, gatewayPort: 30000)
        .UseAdoNetClustering(options =>
        {
            options.Invariant = invariant;
            options.ConnectionString = connectionString;
        })
        .UseAdoNetReminderService(options =>
        {
            options.Invariant = invariant;
            options.ConnectionString = connectionString;
        })
        .AddAdoNetGrainStorage("PubSubStore", op =>
        {
            op.ConnectionString = connectionString;
            op.Invariant = invariant;
        })
        .AddAdoNetGrainStorageAsDefault(op =>
        {
            op.ConnectionString = connectionString;
            op.Invariant = invariant;
        })
        .ConfigureLogging(lb =>
        {
            lb.AddConsole();
            lb.AddSerilog(Log.Logger);
        })
        .ConfigureApplicationParts(c =>
            c.AddApplicationPart(typeof(IFeatureManagementStartupGrain).Assembly).WithReferences())
        .Configure<MessagingOptions>(options =>
        {
            options.ResponseTimeout = appSettings.ResponseTimeout;
            options.ResendOnTimeout = appSettings.ResendOnTimeout;
            options.MaxResendCount = appSettings.MaxResendCount;
        })
        //This will bootstrap all of the grains.  The logs show that the reminders are firing
        .AddStartupTask<GrainStartupTask>();

        return siloHostBuilder.Build();
    }`

Here is the code in the client:
`public static IGrainFactory BuildGrainFactory()
{
IClusterClient client = null;

        var invariant = ConfigurationManager.ConnectionStrings["BackingStore"].ProviderName;
        var connectionString = ConfigurationManager.ConnectionStrings["BackingStore"].ConnectionString;

        var result = Policy.Handle<Orleans.Runtime.SiloUnavailableException>().Or<System.AggregateException>()
            .WaitAndRetry(10, r => TimeSpan.FromSeconds(Math.Pow(2, r)))
            .ExecuteAndCapture(() =>
            {
                try
                {
                    client = new ClientBuilder()
                    .Configure<ClusterOptions>(options =>
                    {
                        options.ClusterId = "ClusterID";
                        options.ServiceId = "ServiceID";
                    })
                    .UseAdoNetClustering(options =>
                    {
                        options.Invariant = invariant;
                        options.ConnectionString = connectionString;
                    })
                    .AddSimpleMessageStreamProvider("SMSProvider")
                    .ConfigureApplicationParts(parts => parts.AddApplicationPart(typeof(IFeatureManagementGrain).Assembly))
                    .Build();

                    client.Connect().Wait();
                }
                catch (AggregateException ex)
                {
                    Log.Logger.Error(ex, "Failed to initialize orleans client");

                    if (ex.InnerException is SiloUnavailableException siloUnavailableException)
                    {
                        Log.Logger.Error(ex.InnerException, "Failed to initialize orleans client");
                        client.Dispose();
                        throw;
                    }
                }
                catch (Exception ex)
                {
                    Log.Logger.Error(ex, "Failed to initialize orleans client");
                }
            });

        if (result.Outcome == OutcomeType.Failure)
        {
            Log.Logger.Error("Failed to initialize the Orleans client");
        }

        return client;
    }`

Please note, when I revert back to our orleans 1.5 application on the DEV server, everything works so it shouldn't be a network connectivity issue on the server. Here are the logs of the silo starting.
OrleansDebugLog.xlsx

Here is the exception from the client:

"The target silo became unavailable for message: Request cli/d0621ab9@97ba6db4->S10.10.92.113:30000:265920396stg/17/00000011@S00000011 #10: . Target History is: S10.10.92.113:30000:265920396:*stg/17/00000011:@S00000011. See https://aka.ms/orleans-troubleshooting for troubleshooting help.", Data: {…}, InnerException: null, TargetSite: "Void Throw()", StackTrace: " at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Orleans.OutsideRuntimeClient.<b__58_2>d.MoveNext() in D:\build\agent\_work\18\s\src\Orleans.Core\Runtime\OutsideRuntimeClient.cs:line 249\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Orleans.OutsideRuntimeClient.<g__ExecuteWithRetries58_3>d.MoveNext() in D:\build\agent\_work\18\s\src\Orleans.Core\Runtime\OutsideRuntimeClient.cs:line 269\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Orleans.OutsideRuntimeClient.d__58.MoveNext() in D:\build\agent\_work\18\s\src\Orleans.Core\Runtime\OutsideRuntimeClient.cs:line 252\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Orleans.OutsideRuntimeClient.d__57.MoveNext() in D:\build\agent\_work\18\s\src\Orleans.Core\Runtime\OutsideRuntimeClient.cs:line 193\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Orleans.ClusterClient.d__19.MoveNext() in D:\build\agent\_work\18\s\src\Orleans.Core\Core\ClusterClient.cs:line 120", HelpLink: null, Source: "mscorlib", HResult: -2146233088, Type: "Orleans.Runtime.SiloUnavailableException"}

Any suggestions on how I can debug this issue? Thanks!

[buddy-james]

Installing .NET 4.7.2 on the server fixed the issue. I was able to track this down because I ran into this issue: https://stackoverflow.com/questions/42867434/could-not-load-file-or-assembly-system-valuetuple and I used an assembly redirect in the config file to get rid of the error (System.ValueTuple), however, I received the SiloUnavailableException exception. We decided to install .NET 4.7.2 on the server and the client can now communicate with the silo.

[benjaminpetit]

Closing this issue since it's resolved with .NET 4.7.2