Entity Requeuing and .net 10

[treyhendon]

Hi all,

I wanted to start a discussion (not Issue yet) because I found a problem, but it's possibly not a dotnet-operator-skd issue - it might be a ZiggyCreatures.Caching or .net 10 issue. Also, I understand that .net 10 is not listed as fully supported and I'm running a little ahead.

TLDR; Using dotnet-operator-sdk 9.x or 10.x and updating to .net 10 will cause a memory leak when entity requeuing is used.

Using gcdump I did a lot of digging and found that cache objects (including JsonSerializerOptions in the tree of cache objects) were the biggest offenders.
For this image, memory started around 100MiB for the baseline gcdump and was above 600MiB for the comparison (after 16 hours of running).

I am calling out the Controller Requeue because it is critical for my operator and can understand that it's probably a less commonly used feature. I performed a round of testing on .net 10 and removed all requeue logic from my code. Console logging showed the initial entity reads from k8s were added to the memory cache, then the app resources remained stable as it ran for hours. I am assuming this was because there were no new writes to the cache like the requeue feature does.

I should also point out that I tested both forms of requeuing - the original EntityRequeue<T> and the new ReconciliationResult requeueAfter option. There was no difference in the outcome.

Ultimately, I found that I could use the latest version of dotnet-operator-sdk (10.5.0) and simply switch between .net 8 and .net 10 in my container to invoke (or resolve) the memory leak.

Does anyone else use the requeuing feature heavily? Has anyone else tried upgrading to .net 10 yet?

[kimpenhaus]

hey @treyhendon,

thanks for starting this discussion and sorry for asking - but just to get it right as I am no native speaker:

you are using v10.5.0?

• when using it with .NET 10 there is a memory leak?
• when using it with .NET 8 there is not?

we are using the operator with .NET 10 in production, staging and development - with no heavy retrying but with queued retries. actually we don't see the meory increasing that much. and just out of curiosity: where did you read that .NET 10 isn't fully supported? from my perspective it should be 😄 if there is still a gap somewhere we should address that.

thanks,
marcus

[treyhendon]

I also saw the leak on v9.11.8 when using .NET 10.

There is not a memory leak on v9.11.8 using .NET 8.

[kimpenhaus]

@treyhendon I've tried to get a sample in my minkube running which should hopefully showed the memory leak. but unfortunately I can't get a sample that leaks...

Its just simple as it could be:

    public Task<ReconciliationResult<V1Pod>> ReconcileAsync(V1Pod entity, CancellationToken cancellationToken)
    {
        _logger.LogInformation(
            "Reconciling pod {Namespace}/{Name} (phase: {Phase}, node: {Node}). Requeuing in {Interval}.",
            entity.Namespace(),
            entity.Name(),
            entity.Status?.Phase ?? "unknown",
            entity.Spec?.NodeName ?? "unscheduled",
            RequeueInterval);

        return Task.FromResult(ReconciliationResult<V1Pod>.Success(entity, RequeueInterval));
    }

where RequeueInterval is 5 seconds.

I than recorded the kubectl top pod stats from the .NET 9 and .NET 10 versions of the operator.

the results after nearly 3 hours are:

which looks pretty stable.

maybe you could attach a stripped sample which reproduces the leak? that would help a lot I guess 😅

thanks!

ps: I've updated the README with .NET 10 support.

[treyhendon]

Thanks for clarifying about .NET versions.

I'm working to get an example together for you now. :)

[kimpenhaus]

I'll updated my sample to do status writes on each reconcile - so that reads and writes with json transformation are ensured. I'll attach it here.
PodWatchOperator.zip

The outcome was pretty much the same

[treyhendon]

Update - To prepare a sample, I removed all my custom service classes as well as a Prometheus histogram metric from the Controllers yesterday and did not see any sign of the memory leak. Today I added back the Prometheus histogram with a simple 500ms wait time and there might be a sign the leak again. I'm going to continue testing with this very slimmed down code base and will let you know.

[treyhendon]

Ok, official answer on this - it was not Prometheus.net.

The issue was with database connections. The operator does various database management tasks for us in our environments and we specifically want to use Pooling=false in our connection strings.

[kimpenhaus]

Thanks for the heads up and glad you have found the issues.