Subtle but important behavior change in orleans streaming coming with 1.5

[xiazen]

We made several changes to Orleans streaming system in 1.5, including new features and bug fixes, some of which are included in Orleans 1.5 beta, remaining of which will be included in future 1.5 official release.

Aside from obvious changes you can see in our beta release note:

1. Added extension points to EventHubAdapterFactory Add extension points to EventHubAdapterFactory #2930
2. Added SlowConsumingPressureMonitor for EventHub streams Add SlowConsumingPressureMonitor #2873
3. Programmable stream subscribe API Programmable stream subscribe API #2741 Programmatic streaming subscribe API #2796 refactor Programmatic Subscribe API #2909

There's a subtle behavior change in EventHubStreamProvider eco system's cache purge logic, which may not be obvious to end users, but may bring confusion. Let me recap 1.4 and compare it to 1.5 to demonstrate.

The behavior change

In EventHubStreamProvider, different cache can be configured to use its own block pool or share block pool with other caches. When messages are added to the cache, cache allocate blocks from block pool to store those messages. When messages expired, there's purging logic kicking in and return purged blocks back to block pool. How messages are purged and how blocks are freed to block pool is what it is subtle different between 1.4 and 1.5.

In 1.4, when the fixed size block pool is full, it send a purge request to the cache (name it cache1) which owns its oldest block (block0), requesting it return block 0 back. This "purge block 0" request will enqueue cache1's purgeQueue and wait there. So overtime, cache1 will have a purgeQueue which is full of "purge block x" requests. Whenever new messages are added to cache1, it will dequeue a "purge block x" request from its purgeQueue and do a purge action. This purge action includes two steps: 1. mark messages as purged if they expired, 2 is return block x to the block pool. But because this purge action would return block x to the pool whether it has valid messages (messages which haven't expired yet) or not, it produces cache miss bug. This behavior is fine historically to its initial users. But when more and more users started to use EHStreamProvider, this became a problem , because it violates its time based purge promise, which is configured using DataMaxAgeInCache and DataMinTimeInCache in its EHStreamProviderSettings. This bug is fixed in #2934 , which was included in 1.5beta.

After that fix, in 1.5beta, messages would only be purged if it expired. Cache now will do a time based purge every time when pulling agent try to ReadFromQueue, which is more frequent compared to 1.4. In its time based purge action, it will do two things, 1 is mark messages as purged if they expired, 2 is put purged blocks into a purgedBlock queue if its messages are all purged. When the block pool send a "purge block x" request to the cache, the cache will free every blocks in its purgedBlock queue up to this block x. In this way, we won't be freeing blocks which still contains valid messages.

What users need to do

This fix brings up a dilemma, which is why I want to write this post. Since even when the block pool is full and it sends out the "purge oldest block" request, the oldest block is not always purged anymore in 1.5. The promise on "fixed size" is not enforced. Size of block pool depends more on DataMaxAgeInCache and DataMinTimeInCache settings, less on CacheSizeInMb. And since CacheSizeInMb is be essentially based on DataMinTimeInCache and DataMaxAgeInCache Settings, we think it is a good direction to go. And we may remove CacheSizeInMb related settings and infra in the future.

And to help end users tune their settings to avoid memory allocation related issues, we will be releasing statistic monitoring feature (#3014) in 1.5, which will help end users to understand their cache's behavior and tune their settings to a optimal value.

Last thing I want to bring up, there could be a bad behavior in 1.5 beta streaming if you used EHStreamProvider under a massive delivery scenario (for example, 15190 TPS per silo). The cause of it is : in massive delivery scenario, purge block x request from block pool can be always too new to purge, hence "free block" action will not be performed even when the cache's purgeBlocks queue is not empty. This results in purged blocks not able to be freed to the pool, which leads to unnecessary memory allocation and potentially out of memory exception from your host. If you see memory issues in streaming when migrating to 1.5 beta, this might be the cause. And we will be including a fix for that in 1.5 .

[kogir]

I'm relatively new to Orleans (~1yr use) and completely new to streams but I'm currently evaluating the EventHubStreamProvider.

My takeaway from what was written:

• Prior to 1.5, cache eviction was primarily size based. When needed, DataMaxAgeInCache and DataMinTimeInCache were violated to ensure CacheSizeInMb was respected.
• Post 1.5 cache eviction is entirely based on age. CacheSizeInMb will be violated when needed to respect DataMaxAgeInCache and DataMinTimeInCache. This will lead to fewer cache misses, but (potentially) more memory use. New statistics are being added to better monitor memory use, and CacheSizeInMb may be removed entirely in later versions.

If the above is what you were trying to convey, then you succeeded! Otherwise it's at least not clear enough for me to understand. That may be ok anyway since the change only affects anyone already using EventHubStreamProvider - people who should have more background than me.

[xiazen]

@kogir , yes those are precisely what I want to explain to users, which mostly lie on the section of "What users need to do". People can read more of this post if they want more context. Thanks for your feedback !