AzureWebSample: worker role exception.

[maiorfi]

Hi.

Running AzureWebSample I always get AzureSilos worker role recycling, with following trace.

Any suggestion?

Microsoft.WindowsAzure.ServiceRuntime Information: 100 : Role environment . INITIALIZING
Microsoft.WindowsAzure.ServiceRuntime Verbose: 500 : Role instance status check starting
Microsoft.WindowsAzure.ServiceRuntime Verbose: 502 : Role instance status check succeeded: Ready
Microsoft.WindowsAzure.ServiceRuntime Information: 100 : Role environment . INITIALED RETURNED. HResult=0
Microsoft.WindowsAzure.ServiceRuntime Information: 101 : Role environment . INITIALIZED
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\WindowsAzureTelemetryEvents\v4.0_2.5.0.0__31bf3856ad364e35\WindowsAzureTelemetryEvents.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\WindowsAzureEventSource\v4.0_2.5.0.0__31bf3856ad364e35\WindowsAzureEventSource.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\Progetti\orleans\Samples\AzureWebSample\OrleansAzureSample\csx\Debug\roles\OrleansAzureSilos\approot\Orleans.Azure.Silos.WorkerRole.dll'. Symbols loaded.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.WindowsAzure.ServiceRuntime\v4.0_2.4.0.0__31bf3856ad364e35\Microsoft.WindowsAzure.ServiceRuntime.dll'. Cannot find or open the PDB file.
Microsoft.WindowsAzure.ServiceRuntime Information: 200 : Role entrypoint . CALLING OnStart(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Information: 202 : Role entrypoint . COMPLETED OnStart(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
The thread 0x33a8 has exited with code 259 (0x103).
A first chance exception of type 'System.Threading.WaitHandleCannotBeOpenedException' occurred in mscorlib.dll
Microsoft.WindowsAzure.ServiceRuntime Information: 203 : Role entrypoint . CALLING Run(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Warning: 204 : Role entrypoint . COMPLETED Run() ==> ROLE RECYCLING INITIATED: Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Information: 503 : Role instance recycling is starting
The thread 0x3954 has exited with code 259 (0x103).
Microsoft.WindowsAzure.ServiceRuntime Information: 205 : Role entrypoint . CALLING OnStop(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Information: 205 : Role entrypoint . CALLING OnStopping()
Microsoft.WindowsAzure.ServiceRuntime Information: 206 : Role entrypoint . COMPLETED OnStop(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
The thread 0x2db8 has exited with code 259 (0x103).
'WaWorkerHost.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\Progetti\orleans\Samples\AzureWebSample\OrleansAzureSample\csx\Debug\roles\OrleansAzureSilos\base\x64\WaWorkerHost.exe'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: WaWorkerHost.exe): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_64\msshrtmi\v4.0_2.5.0.0__31bf3856ad364e35\msshrtmi.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: WaWorkerHost.exe): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.WindowsAzure.ServiceRuntime\v4.0_2.5.0.0__31bf3856ad364e35\Microsoft.WindowsAzure.ServiceRuntime.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: Domain 2): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.WindowsAzure.ServiceRuntime\v4.0_2.5.0.0__31bf3856ad364e35\Microsoft.WindowsAzure.ServiceRuntime.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: WaWorkerHost.exe): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\mscorlib.resources\v4.0_4.0.0.0_it_b77a5c561934e089\mscorlib.resources.dll'. Module was built without symbols.
'WaWorkerHost.exe' (CLR v4.0.30319: WaWorkerHost.exe): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\Microsoft.ServiceHosting.Tools.DevelopmentFabric.Runtime\1.0.0.0__31bf3856ad364e35\Microsoft.ServiceHosting.Tools.DevelopmentFabric.Runtime.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Xml.Linq\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.Linq.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_64\System.Web\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Web.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_64\msshrtmi\v4.0_2.5.0.0__31bf3856ad364e35\msshrtmi.dll'. Cannot find or open the PDB file.
Microsoft.WindowsAzure.ServiceRuntime Information: 100 : Role environment . INITIALIZING
Microsoft.WindowsAzure.ServiceRuntime Information: 100 : Role environment . INITIALED RETURNED. HResult=0
Microsoft.WindowsAzure.ServiceRuntime Information: 101 : Role environment . INITIALIZED
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\WindowsAzureTelemetryEvents\v4.0_2.5.0.0__31bf3856ad364e35\WindowsAzureTelemetryEvents.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\WindowsAzureEventSource\v4.0_2.5.0.0__31bf3856ad364e35\WindowsAzureEventSource.dll'. Cannot find or open the PDB file.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\Progetti\orleans\Samples\AzureWebSample\OrleansAzureSample\csx\Debug\roles\OrleansAzureSilos\approot\Orleans.Azure.Silos.WorkerRole.dll'. Symbols loaded.
'WaWorkerHost.exe' (CLR v4.0.30319: RdRuntime): Loaded 'C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.WindowsAzure.ServiceRuntime\v4.0_2.4.0.0__31bf3856ad364e35\Microsoft.WindowsAzure.ServiceRuntime.dll'. Cannot find or open the PDB file.
Microsoft.WindowsAzure.ServiceRuntime Information: 200 : Role entrypoint . CALLING OnStart(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Information: 202 : Role entrypoint . COMPLETED OnStart(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
The thread 0x3290 has exited with code 259 (0x103).
A first chance exception of type 'System.Threading.WaitHandleCannotBeOpenedException' occurred in mscorlib.dll
Microsoft.WindowsAzure.ServiceRuntime Information: 203 : Role entrypoint . CALLING Run(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Warning: 204 : Role entrypoint . COMPLETED Run() ==> ROLE RECYCLING INITIATED: Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Information: 503 : Role instance recycling is starting
The thread 0x2a7c has exited with code 259 (0x103).
Microsoft.WindowsAzure.ServiceRuntime Information: 205 : Role entrypoint . CALLING OnStop(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint
Microsoft.WindowsAzure.ServiceRuntime Information: 205 : Role entrypoint . CALLING OnStopping()
Microsoft.WindowsAzure.ServiceRuntime Information: 206 : Role entrypoint . COMPLETED OnStop(): Microsoft.WindowsAzure.ServiceRuntime.DefaultEntryPoint

[ReubenBond]

That first chance exception is in the Azure ServiceRuntime, not Orleans, so it's probably something else. The Azure UserVoice team marked this issue as 'Completed', but I'm not sure they actually fixed it: http://feedback.azure.com/forums/169386-cloud-services-web-and-worker-role/suggestions/6193103-reduce-expected-exceptions-thrown-on-role-initia

Have you correctly set the Azure Table Storage connection strings inside OrleansConfiguration.xml?

[maiorfi]

Creating a brand new cloud service project containing only "stock" WorkerRole work fine, despite first chance exception "System.Threading.WaitHandleCannotBeOpenedException" throws anyway. So that seems not to be real cause of role recycling.

Also, I'm working with unmodified source AzureWebSample project, where OrleansCOnfiguration.xml is simply:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <Liveness LivenessType="AzureTable" />
  </Globals>
    <Defaults>
    <Tracing DefaultTraceLevel="Info" TraceToConsole="true" TraceToFile="{0}-{1}.log" >
    </Tracing>
    </Defaults>
</OrleansConfiguration>

[ReubenBond]

Thanks for the info. You need a <SystemStore> element, like so:

<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <ReminderService ReminderServiceType="AzureTable" />
    <SystemStore SystemStoreType="AzureTable" DataConnectionString="UseDevelopmentStorage=true"/>
<!-- snippp -->

The <Liveness> element is unnecessary now.
Hope that helps!

[maiorfi]

Thank you very much for your support!

Sadly, behavior is exactly the same :(

But I just observed that setting a breakpoint on WorkerRole Start or Run methods doesn't cause debugger to halt, so WorkerRole object is never created.

After updating "Microsoft.WindowsAzure.ServiceRuntime" to 2.5 Azure SDK WorkerRole is instantiated as expected, but soonafter AzureSilo.cs (in OrleansAzureUtils) complains about a non existent member (System.EntryPointNotFoundException) been called.

It seems something tied to SDK assemblies dependencies...now I'm going to recompile Orleans binaries after updating references...

[maiorfi]

Bingo.

Updating references in projects:

• Orleans
• OrleansAzureUtils
• OrleansRuntime
• OrleansProviders

and redeploying related updated binaries into SDK Binaries folder makes AzureWebSample running fine, and precisely:

• OrleansAzureUtils.* both into SDK\Binaries\OrleansServer and SDK\Binaries\OrleansClient
• OrleansProviders.* both into SDK\Binaries\OrleansServer and SDK\Binaries\OrleansClient
• ClientGenerator.* both into SDK\Binaries\OrleansServer and SDK\Binaries\OrleansClient
• Orleans.* both into SDK\Binaries\OrleansServer and SDK\Binaries\OrleansClient
• OrleansRuntime.* only into SDK\Binaries\OrleansServer

Also, I needed to copy following Azure SDK 2.5 assemblies into Binaries folder:

• Microsoft.WindowsAzure.ServiceRuntime.dll
• Microsoft.WindowsAzure.StorageClient.dll

and finally removed:

• Microsoft.WindowsAzure.StorageClient.dll (since it misses in Azure SDK 2.5)

[sergeybykov]

I submitter a PR to upgrade Microsoft.WindowsAzure.ServiceRuntime.dll to 2.5 - #72. Microsoft.WindowsAzure.StorageClient.dll is needed because system store is still written against the 1.7 version of the API (see #2 in https://github.com/dotnet/orleans/wiki/Ideas-for-Contributions).

[gabikliot]

@ReubenBond There is no need to specify SystemStore in the config file when using AzureHost (you can, it's benign). AzureHost figures out it needs to use Azure table automatically and takes the connection string from csdef file. Only if you run on prem, not on Azure, you have to specify SystemStore.