Client shutdown problem

[debarisi]

Hi,

We noticed that if we kill the client without calling method .Close, next time when we tried to establish a connection with silo host client connection hangs with the state "Starting".

From console log, we can see all active gateways, TCP connection is alive but code simply hangs on

Has anyone else noticed the same issue before?

Best,

Dean

[benjaminpetit]

What do you mean by kill the client? Do you reuse the same object or do you create a new one with the builder?

[debarisi]

I mean shutdown application in your IDE.

When we are debugging our client application most of the time we are connecting to the SiloHost that is deployed on dev servers.

Every time when we close the application without calling .close() on the client we encounter the same problem.

Just one notice - application does not 'hang' on a picture I send you in my previous post because we can see in debug output live gateway list. The only thread that is calling Connect 'hangs'

Best,

Dean

[debarisi]

We tried with server version 2.0.3 and 2.0.4, client version 2.0.3

Best,

Dean

[debarisi]

And one more info - when running SiloHost and client local on PC - everything works ok.

[debarisi]

Client logs after running client for second time

---------- Starting OutsideRuntimeClient with runtime Version='2.0.3. Commit Hash: c5432c1 (Release).' in AppDomain=<AppDomain.Id=1, AppDomain.FriendlyName=SportFeed.DataProvider.API>
trce: Orleans.Runtime.Membership.AdoNetGatewayListProvider[0]
AdoNetClusteringTable.InitializeGatewayListProvider called.
trce: Orleans.Runtime.Membership.AdoNetGatewayListProvider[0]
AdoNetClusteringTable.GetGateways called.
trce: Orleans.Runtime.Membership.AdoNetGatewayListProvider[0]
AdoNetClusteringTable.GetGateways called.
info: Orleans.Messaging.GatewayManager[101309]
Found 4 knownGateways from Gateway listProvider [gwy.tcp://10.0.0.18:5020/271713289, gwy.tcp://10.0.0.26:5020/271713291, gwy.tcp://10.0.0.24:5020/271713296, gwy.tcp://10.0.0.25:5020/271713321]
dbug: Orleans.Runtime.SafeTimer[101404]
Creating timer Orleans.Runtime.sync.SafeTimerBase with dueTime=00:01:00 period=00:01:00
dbug: Orleans.Messaging.ClientMessageCenter[0]
Proxy grain client constructed
dbug: Orleans.Messaging.ClientMessageCenter[0]
Proxy grain client started
dbug: Orleans.Runtime.SafeTimer[101404]
Creating timer Orleans.Runtime.sync.SafeTimerBase with dueTime=00:30:00 period=-00:00:00.0010000
trce: Orleans.OutsideRuntimeClient[0]
Send Request cli/c04bf192@ad775ab1->S10.0.0.25:5020:271713321stg/17/00000011@S00000011 #2:
dbug: Orleans.Messaging.ClientMessageCenter[0]
Creating gateway to gwy.tcp://10.0.0.25:5020/271713321 for pre-addressed message
dbug: Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321[100914]
Starting gateway connection for gateway gwy.tcp://10.0.0.25:5020/271713321
info: Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321[100908]
Connected to gateway at address gwy.tcp://10.0.0.25:5020/271713321 on trial 0.
dbug: Messaging.GatewayClientReceiver/gwy.tcp://10.0.0.25:5020/271713321[0]
Started asynch agent Messaging.GatewayClientReceiver/gwy.tcp://10.0.0.25:5020/271713321
dbug: Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321[0]
Started asynch agent Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321
trce: Orleans.Messaging.ClientMessageCenter[100918]
Sending message Request cli/c04bf192@ad775ab1->S10.0.0.25:5020:271713321stg/17/00000011@S00000011 #2: via gateway gwy.tcp://10.0.0.25:5020/271713321
info: Messaging.GatewayClientReceiver/gwy.tcp://10.0.0.25:5020/271713321[0]
Starting AsyncAgent Messaging.GatewayClientReceiver/gwy.tcp://10.0.0.25:5020/271713321 on managed thread 13
info: Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321[0]
Starting AsyncAgent Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321 on managed thread 14
trce: Messaging.GatewayConnection/GatewayClientSender_gwy.tcp://10.0.0.25:5020/271713321[0]
Got a Request message to send: Request cli/c04bf192@ad775ab1->S10.0.0.25:5020:271713321stg/17/00000011@S00000011 #2:

[debarisi]

If I leave the client connected, gateway list will be refreshed.
To be honest we got first demo schedule soon so we are fixing this issue by restarting all servers in the cluster.

We will try to dig deeper soon!

[debarisi]

client Builder

_client = new ClientBuilder()
.Configure(ClusterOptions)
.UseAdoNetClustering(ClusterMembership)
//.ConfigureApplicationParts(parts => parts.AddFromAppDomain().AddFromApplicationBaseDirectory())
.ConfigureApplicationParts(parts => parts
.AddApplicationPart(typeof(IEventGrain).Assembly)
.AddApplicationPart(typeof(EventGrain).Assembly)
)
.ConfigureLogging(logging =>
{
logging.SetMinimumLevel(LogLevel.Trace);
logging.AddConsole();
})
.Build();

                await _client.Connect(RetryMethod);

 private static readonly Action<AdoNetClusteringClientOptions> ClusterMembership = options =>
    {
        options.ConnectionString = _appSettings.ConnectionString;
        options.Invariant = "Npgsql";
    };

Best,

Dean

[debarisi]

Hi,

I tried to dig a little.

This is what I have found so far (stack trace)

On client connect

1. _clusterClient.Connect(RetryMethod).Wait();
2. method StartInternal is called in class OutsideRuntimeClient
3. Inside StartInternal line of code 256

await ExecuteWithRetries(
async () => this.GrainTypeResolver = await transport.GetGrainTypeResolver(this.InternalGrainFactory),
retryFilter);

calls ExecuteWithRetries that after await task(); never continues.

[debarisi]

Hi,

I will need more time to fully understand what is all happening in the code.
But the last line of code that 'hangs' is TaskExtensions.cs method ConvertAsync.

Best,

Dean

[debarisi]

Updated everything to 2.0.5 night build and everything works!

[sergeybykov]

@debarisi Thanks for validating! Can you also check with 2.1.0-beta that we published to nuget.org yesterday?

[debarisi]

Sure!