Error after deploying to azure.

[loneshark99]

After deploying application to Azure,I sometimes see these errors and the role recycles.

Can someone give some guidance on why this is happening. I see this multiple times in the logs. Will this cause the role to recycle?

[2016-01-12 00:10:20.185 GMT 8 WARNING 100837 StatsTableDataManager 100.108.82.161:0] Intermediate error bulk inserting 17 entries in the table OrleansClientStatistics
Exc level 0: Microsoft.WindowsAzure.Storage.StorageException: 0:The specified entity already exists.
RequestId:c46b4a63-0002-00c6-3acd-4c1bb9000000
Time:2016-01-12T00:10:20.4352956Z
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.AzureUtils.AzureTableDataManager`1.d__4c.MoveNext()
[2016-01-12 00:10:20.674 GMT 22 WARNING 100157 CallbackData 100.109.82.163:0] Response did not arrive on time in 00:05:00 for message: Request _cli/fb36a694@ef00c1b0->S100.108.16.57:30000:190253047_grn/42F4CACE/00000000+whatever-0-0 #4: global::Orleans.MethodName(). Target History is: S100.108.16.57:30000:190253047:*grn/42F4CACE/00000000+whatever-0-0:. About to break its promise.
[2016-01-12 00:10:22.021 GMT 10 WARNING 100011 OutsideRuntimeClient 100.108.82.161:0] No callback for response message: Response S100.108.112.40:30000:0_grn/42F4CACE/00000000+whatever-0-0@94108a91->_cli/fb36a694@ef00c1b0 #4: global::Orleans.MethodName()

[sergeybykov]

There are seem to be 2 different errors here.

1. StatsTableDataManager is failing to write statistics to the table. This should be harmless but it's failing with a curious error "The specified entity already exists." Are you reusing same deployment IDs for multiple deployments by chance?
2. Grain call timeout: "Response did not arrive on time in 00:05:00 for message". Without more details of what this call is trying to do or full logs it's hard to guess why the call is timing out. There can be one of many reasons, e.g. blocking calls in grain methods, deadlocks in grain call cycles, silo failing, etc.

[loneshark99]

@sergeybykov

1. I just publish to azure from my solution, I am sure that creates a new deploymentId everytime, but if its harmless I am ok for now, will look into that later.

One basic question, If I do .Result in the client code is that ok? meaning in my GrainCode I always use async/await but when I call the Grain.

var result = SomeGrain.GetDataAsync("id").Result;

And I am calling the above code multiple times. Do you think code like this can cause a issue.

[gabikliot]

1. Can you please share a full client log? I would like to investigate it a bit more. We should not be getting "The specified entity already exists." for that table. Every row key has a client name, which is unique and incremented counter. See: https://github.com/dotnet/orleans/blob/master/src/OrleansAzureUtils/Storage/StatsTableDataManager.cs#L155.

Unless maybe we corrupt the counter? How frequently are you publishing to this table? Please share a client log where this happens.

2. Yes, it is OK to call .Result in the client.

[loneshark99]

@gabikliot I have attached the logs till the first exception came. I am doing the deployment pretty frequently, like 30 mins-1 hour duration because of the thing I am doing.

Logs.txt

Can I tell from the orleans silo logs that there is some threading issues, The reason I am asking this is, for 5 mins test I get 1000 calls to grains but if I run the same test for 1 hour I only get 3000 calls, I do see these timeout errors but there is nothing other than that in the orleans logs. Can you provide some guidance on this.

I can share with you my solution if you want to take a look or provide some guidance, have been stuck on this for a while.

[gabikliot]

There are a couple of things in the log that look suspicions.

1. This is only 1 core machine. We usually recommend at least 2 or 4 cores. It should be OK tom use 1 as well, especially for clients, but still not our usual setup.
2. GC Type=Client GCLatencyMode=Interactive - unless it is desktop application, with live UI, you should configure the client machine also to use server GC. If the client is a front end in a service, it is effectively a server, so should use server GC and non concurrent GC. See: http://dotnet.github.io/orleans/Advanced-Concepts/Configuring-.NET-Garbage-Collection.
3. you are using Orleans.Serialization.BondSerializer.
4. Response timeout: 00:05:00 - we recommend keeping it at the default 30 sec.

None of those could explain, as far as I can tell, "The specified entity already exists" on bulk insert.
So I am puzzled.

As for the 1000 vs 3000 calls - for that we definitely should look at silo logs and into the grain code. Maybe grains are doing something funcky.

[gabikliot]

@loneshark99 , is this still an issue? Have you been able to resolve it? Do you have any more data to share?