Add initial draft of ownership for C#

[agocke]

No description provided.

[timcassell]

This is the only part I don't like. It's awkward compared to an instance method. I think we should be able to add an attribute to an instance method that declares it takes its own ownership, like how an extension method would take ownership.

[agocke]

I proposed BorrowedReceiver to allow adapting to borrows in classes. It would make sense to allow inverting that for owned types

[timcassell]

Also how will a collection of resources be handled?

[AaronRobinsonMSFT]

There is something missing in this section around transferring ownership via move semantics. I don't know where yet, but that notion seems missing after getting to the ImmutableArray discussion below.

[AaronRobinsonMSFT]

or on the stack?

[AaronRobinsonMSFT]

I assume it cannot be a global static.

[agocke]

Correct -- this was meant as a restriction on fields of non resource types. But also, they cannot be stored in static fields

[AaronRobinsonMSFT]

This would imply that every method with one of these types now must contain a try block, correct? Additionally, it means there could now be many try blocks.

[agocke]

unfortunately, yes

[AaronRobinsonMSFT]

"should" or "must"? This seems important for the previous case note above where a method can have multiple try blocks. If this is "should" it then means there could be no optimization for collapsing try blocks for the same scope and each one of these variables would require its own try block.

[agocke]

i'm actually not sure what breaks in all cases. it may be possible to recover from an exception with only moderately negative consquences. Definitely needs more thought

[jaredpar]

should is a four letter word.

[agocke]

I’m fine with making this “must” and specifying that an uncaught exception in Drop aborts the process

[AaronRobinsonMSFT]

For structs it implies that borrowed references can never be "normal" struct values — borrowed struct references are always true managed by-ref variables.

I assume this would limit use of these types to non-async methods. At least async v1.

[agocke]

Correct, this is one of the main reasons to allow owned reference types -- by-refs are too limiting for many scenarios.

[AaronRobinsonMSFT]

Similar to the Drop rules above, I assume that using reflection here has some implication worth discussing
.

[MattParkerDev]

Love this. While not strictly lang related and likely not v1, I could see this allowing the runtime to intrinsify IResource and Drop(), to immediately/more efficiently dealloc reference types, outside of GC

[hez2010]

What about deferred ownership? This is quite common in ECS/async programming.

For example,

var buffer = pool.Rent(...);

Task A: ReadWrite(buffer);
Task B: ReadWrite(buffer); // depends on A

In Rust we can have

let mut buffer = vec![0; 1024];

let a = tokio::spawn(async move {
    process_a(&mut buffer).await;
    buffer // ownership comes out of A
});

let b = tokio::spawn(async move {
    // B can be scheduled immediately, but cannot access buffer yet.
    let mut buffer = a.await.unwrap();

    process_b(&mut buffer).await;
    buffer
});

let buffer = b.await.unwrap();

where in C# it's simliar to something like

var buffer = pool.Rent(...);
var a = Schedule(buffer, static buffer => { ProcessA(buffer); return buffer; });
var b = Then(a, static buffer => { ProcessB(buffer); return buffer; });
var buffer = await b;

Note that B can be created/scheduled before A has completed, but B does not actually receive a borrow of buffer until A has finished using it. Conceptually, the task owns the resource and represents the future right to access it, rather than both A and B holding overlapping mutable borrows. This seems especially relevant to ECS/job systems, where the entire dependency graph is commonly constructed before any of the jobs execute.

With the current rule that resource types cannot be substituted for generic type parameters, it would seem to prevent representations such as Task<R> or an ownership-aware JobHandle<R> from carrying the resource through the dependency chain.

[jjonescz]

Suggested change

All of these problems are solved using a new language construct: _conditional implementation_. In existing C#, all types either implement an interface or not. This proposal would add a new special case, just for arrays: for a substituted type `T[]`, `T[]` implements the `IResource` interface if and only if `T` implements `IResource`. Correspondly, `T[]` is a resource if and only if `T` is a resource. The implementation of `Drop` is also specified as walking the array from beginning to end and calling `Drop` on each element.

All of these problems are solved using a new language construct: _conditional implementation_. In existing C#, all types either implement an interface or not. This proposal would add a new special case, just for arrays: for a substituted type `T[]`, `T[]` implements the `IResource` interface if and only if `T` implements `IResource`. Correspondingly, `T[]` is a resource if and only if `T` is a resource. The implementation of `Drop` is also specified as walking the array from beginning to end and calling `Drop` on each element.

[jjonescz]

Suggested change

	However, converting just GC types to borrows is not particularly useful on its own. We would like to be able to treat a class as either GC-tracked or owned, _per instantiation_. To do so we will need another instrinsic type:
	However, converting just GC types to borrows is not particularly useful on its own. We would like to be able to treat a class as either GC-tracked or owned, _per instantiation_. To do so we will need another intrinsic type:

[jjonescz]

There is already ImmutableCollectionsMarshal.AsImmutableArray

[jjonescz]

I wonder how BCL will migrate existing types like File. Making them into resources would be breaking. Perhaps we need some opt in flag to allow migration (like nullable or unsafe evolution).

[agocke]

Flags only work when your code has no behavioral changes -- this does. Adding IResource will be a severe breaking change to an existing type. It will likely require a new API in all cases.

[jjonescz]

So I can't have List<R> nor Action<R>? That seems very limiting. What would it take to allow this?

[agocke]

Extend the conditional implementation feature as described for arrays to arbitrary types

[timcassell]

What about an anti-constraint like what was done for ref structs?

[agocke]

I don't think you can write code that works for both owned types and unowned types in the same method. For the simplest problem, the owned type needs to Drop at the end and it has to have a bunch of exception handling to make that happen. In contrast, a GC type can't drop its parent.

I think the way you share code instead is writing it against a borrow. That would allow you to write the same code for both modes and the rules around borrows would make sure that you aren't doing anything illegal. As long as both GC and resources can be converted to borrows, the code is sound in both cases.

[CeliaCasanovas]

Maybe if we had Roles...

[jjonescz]

Should we support async Drop too? I imagine many IAsyncDisposables might want to be converted to IResource.

[agocke]

Drop shouldn't throw, arbitrary I/O likely should throw

[jjonescz]

Does this apply to interior references, i.e., can I have two mutable references to fields of a resource?

[agocke]

Current lifetime spec doesn't describe a difference for interior borrow, so that's not possible

[jjonescz]

I wonder if we could just say that a resource must be dropped via Dispose but we left it to the user. That would connect more naturally to existing C# IDisposable, would be less breaking to migrate to, would be more explicit where disposal vs ownership transfer happens (but still easy to use via using), would make it possible to not need try/finally everywhere (leaving a resources undropped after exception isn't a memory safety issue, it's just a memory leak, right?).

[agocke]

Just because leaking isn't a memory safety issue doesn't mean that leaking cannot be catastrophic for system performance and stability.

There are also many other problems:

• Dispose had the wrong semantics (can be called twice)
• using has the wrong structure (structure-based, as opposed to data flow)
• existing dispose implementations may not be valid drop implementations
• you can't use the interface implementation to imply ownership tracking

[jjonescz]

• Dispose had the wrong semantics (can be called twice)
• existing dispose implementations may not be valid drop implementations

I meant this would still be opt-in via IResource (or some attribute).

• using has the wrong structure (structure-based, as opposed to data flow)

using on a resource could work differently.

• you can't use the interface implementation to imply ownership tracking

That can be disallowed.

Just because leaking isn't a memory safety issue doesn't mean that leaking cannot be catastrophic for system performance and stability.

If you use using and your Dispose doesn't throw, there are no leaks.

[MattParkerDev]

I think the problem is, a using statement can be omitted, accidentally or otherwise. And while memory leak is not an immediate problem, it is still a critical failure in an application - consider a game that allocates every frame. It feels like a pit of failure. I think having to use a using statement to drop an IResource muddies the big win of this proposal - that the lifetime/scope of the object dictates when it is dropped, not the user calling using on it.

Also consider a world where lots/most types are IResources - all the usings would just be noise.

[jjonescz]

I think the problem is, a using statement can be omitted, accidentally or otherwise.

Not with this feature, the compiler would complain about it.

[jcouv]

Like Jan, I'm wondering about how IResource relates to IDisposable/using:

• IResource offers deterministic cleanup for "resources" and ownership discipline
• IDisposable/using offers deterministic cleanup for "resources"

Is there a way to rationalize them together? Can we re-use/reduce concepts?

Just spitballing:

• Drop vs. Dispose: similar concept of deterministic cleanup, but also some differences. we can generate part of it for owned resource, and there's also the problems that Andy pointed out including Dispose semantics. We can just tighten the expectations for Dispose when it comes to owned resources.
• Would it make sense to separate concerns by having the marker interface (perhaps IOwned) only mean "ownership discipline"?
  Then you could have permutations: class C : IOwned (only ownership discipline), class C : IOwned, IDisposable (ownership discipline plus deterministic cleanup) and possibly even class C : IOwned, IAsyncDisposable.
• Should usages with deterministic cleanup involve explicit using syntax? It's a bit more syntax, but blends with existing C# (we'd extend the meaning of using on owned resources) and surfaces the implicit try/catch.
  The lowering for owned resources would differ from today, to be aware of whether the ownership was transferred.
  You could say there are two ways of discharging the cleanup obligation: 1. you transfer ownership, or 2. you use using to cleanup when ownership could remain.
  In that view { File f = new File(); f.ReadBytes(); } would be an error because the obligation to clean up this owned resource is not discharged (either by using or transfer).

[333fred]

It does somewhat feel like ownership without a drop/dispose/free is a potentially useful thing; for example, an array builder type might want to opt-into ownership so that it isn't reused after a call to ToArray(), but doesn't actually have anything to do in that drop/dispose step, since ownership of the underlying array was transfered by ToArray in one case, and in the case that the builder was abandoned, just needs the GC to come and free both it and its underlying resources.

[jjonescz]

I've been toying with some ideas in my extension of this proposal: jjonescz#1

[VasiliyNovikov]

Also think about beginners learning C# and seeing both IDisposable and IResource - this will create a lot of confusion

And think about experienced engineers learning/switching to C# - they see IDisposable and IResource and I'd guess immediate thought would be - "bad design"

[EgorBo]

This looks like a solid step forward!!

• I'm curious if we can learn from Rust and Swift mistakes, e.g. Swift shipped their initial version without noncopyable generics (this proposal bans generics too) and it took them many releases to undo that. Rust seems to invest into CFG-aware borrow checking (project Polonius) while this spec starts with "all lifetimes combine to the shorter lifetime" e.g.

static Borrow<Node> GetOrAdd(ref Map map, int key)
{
    if (map.TryGetBorrow(key, out var value))
        return value;                 // This path extends the borrow to the caller.

    map.Add(key, new Node());         // No borrow exists on this path.
    return map.GetBorrow(key);
}

• Is value-type's default(R) an initialized resource whose Drop must be harmless, or can resource types be non-defaultable? Same question to just array of such structs (similar to nullability analysis hole).

• Rather than permanently banning resource captures, could we eventually have a noncopyable, invoke-once delegate similar to Rust's FnOnce?

• In general, I think I would prefer language-level ownership modifiers over exposing the model primarily through IResource, Owned<T>, Borrow<T>, and ReadOnlyBorrow<T>. For example, purely as illustrative syntax:

noncopyable struct RentedBuffer<T>
{
    // The returned Span borrows this. The relationship is inferred.
    public readonly Span<T> Span { get; }

    public consuming void Dispose();
}

static void Fill(borrowing RentedBuffer<byte> buffer);
static byte[] Freeze(consuming RentedBuffer<byte> buffer);

var buffer = Rent();

Fill(buffer);                    // Shared borrow
Span<byte> view = buffer.Span;   // Keeps buffer borrowed
Use(view);                       // Last use ends the borrow

byte[] result = Freeze(move buffer);
Use(buffer);                     // Error: use after move

This makes three otherwise hidden facts fairly clear:

• Fill borrows.
• Freeze consumes.
• The span's lifetime depends on buffer.

[jaredpar]

The document needs to do a better job of outling the restrictions around resource types. For example, the inability to call ToString or participate in any patterns that convert values to string representations which are prevalent throughout the .NET BCL and ecosystem.

[jaredpar]

what is mutable here

[jaredpar]

Can't this be simplified to R may not be converted to a non-resource type? Also I would separate out the conversions and capture into different rules here cause they can be addressed differently.

[jaredpar]

I worry that implicit move operations will be a net negative. As structured it means the reader, or agent, needs context to understand when values are usable or not. It needs to know what interfaces R implement, what overloads of Helper are being called, etc ... to know if the code will compile. I would consider a more explicit form of transfer here.

[jaredpar]

In this section you should discuss how this code sample works:

R local = new R();
local.M();
local.M();

Is this legal or illegal? It's important to understand how the ref semantics work before getting into borrowing.

[jaredpar]

should is a four letter word.

[jaredpar]

may want to rewrite that paragraph in english :)

[333fred]

Hey, at least no brought up a lattice.

[jaredpar]

Please define _rented so readers can better understand what behavior is being demonstrated here. As written it's unclear what the source is here hence the borrowing / owner relationship is unclear.

[jaredpar]

Do you plan to allow class types to take on readonly members? Otherwise, not much point in a ReadOnlyBorrow<T> of a class type

[jaredpar]

Should discuss here whether R[] is legal or not.

[jaredpar]

It's not really a new language or language construct. Arrays already conditionally implement IEnumerable<T> based on the type of T. This doesn't seem that different.

[agocke]

Oh because of pointers? Yeah I forgot about that feature

[jaredpar]

Yep it's the pointer / __makeref type issue.

[tannergooding]

I'm not sure ImmutableArray is the best example

Namely because I'm not sure any version of this feature would allow ToImmutable itself to change behavior here. It's more an optimization but one that would require passing info into ToImmutable itself to relay that this becomes dead on return and so it is safe to hand back the underlying array directly. -- It's also not always possible because Capacity may not equal Count, in which case the size would be incorrect.

Instead we have MoveToImmutable (throws if capacity != count) and DrainToImmutable (copies if capacity != count) where these concepts were designed up front, but also designed in a way so-as to avoid the ownership issue in the case a move does occur by setting _elements to empty after, allowing the builder to still be safely used.

So I think this is rather an area where it would have had to be designed with that intent up front and where its very different from say ArrayPool where it is instead a correctness concern and one we'd be able to take a change or break around (since its not changing behavior so much as enforcing correctness)

[tannergooding]

This is a very generic name and highly likely to be conflicting, both for our own code and 3rd party. We may want to consider something that is clearer on semantics and less likely to be problematic.

Existing types shouldn't have an issue opting-in to it, but generic methods would since it requires a constraint change. You'd at best get if (T is IResource ...) { } on existing generic methods

[tannergooding]

We may want to consider how this can be used on existing types, especially ones where we know there is effectively ownership or "move only" semantics and where copying can be catastrophic (most mutable structs, for example).

SpinLock is a trivial case that has this issue and likely should be opted into whatever tracking is available both from the "resource" perspective (its a lock with one holder, that may or may not be re-entrant) and that copying the struct represents a failure (because future mutations won't be touching what you think it should be touching).

[agocke]

• In general, I think I would prefer language-level ownership modifiers over exposing the model primarily through IResource, Owned<T>, Borrow<T>, and ReadOnlyBorrow<T>.

There's plenty of time for debating syntax. This is not that time. 🙂

However there's one thing worth calling out:

noncopyable struct RentedBuffer<T>
    public consuming void Dispose();
}

This is not how ownership works. Noncopyable must mean non-aliasable as well. And Dispose shouldn't be optional -- the language should do its best to prevent you from leaking or in some other way losing track of the object. In fact, one of the primary benefits of this system is that it very hard to lose track of types.

[333fred]

Have you thought about dedicated syntax for borrowing? Even if it's a "we can think about it after initial debate", I do think it's at least worth a mention.

[agocke]

& and &mut prefixes are technically available. Maybe riff on that?

[333fred]

I was actually thinking more borrow or mut borrow, but my general concern is just making sure that readers understand that we aren't saying that we won't have a syntax, we just need to consider it.

[jcouv]

I'm unsure about the ergonomics of types over syntax as well. I'd throw borrow and readonly borrow into the hat. This could also help avoid the asymmetry between classes and structs

[jcouv]

This rule seems incomplete. As stated, it implies that Drop is called in this code:

{
    R local = new R();
    Helper(local);
} // leave local's lexical declaring scope

Is the intent that Drop will only be called at most once on a resource (when its ownership ends) or that we skip calling it in some cases or that it can be called more than once?

[jcouv]

Also, consider including some examples showing when Drop is called. What happens with this:

R local = new R();
if (b)
{
  Helper(local);
}

[CeliaCasanovas]

I'm repeating myself but I think Roles would help massively here. Maybe Linear and/or Affine Roles, for instance, if borrowing is to be conceptualised in terms of typing.

[jcouv]

Is there a benefit to keeping these intrinsic types in metadata for the runtime? Maybe metadata erasure would be better since we want all those types to behave like the underlying anyways: modreq(Borrow) T, modreq(ReadOnlyBorrow) T, modreq(Own) T.

That raises a question though: do we want to allow overloads to differ by such annotations?
M(ref StructResource) vs. M(ref readonly StructResource) is disallowed already, would we want to similarly disallow M(Borrow<Resource>) vs M(ReadOnlyBorrow<Resource>)?

[jcouv]

Do we have some motivating examples for ownership and borrowing of ordinary classes? The worked examples below only illustrate resource/IResource cases.

[Sergio0694]

This is technically a safety issue, given that Drop() would be called from a finally block, and it would be returning the array to the pool from there. My understanding is that the guidance is to explicitly not return arrays to the pool from a finally block, because (as @GrabYourPitchforks said), if there's an exception you cannot make assumptions on the state of the process, so it's safer to just throw that rented array away. Basically you should only be returning it in the non-exception case, otherwise let it be collected.

Should we have a way to define a resource that should be dropped without a finally block? This has come up before multiple times already for similar scenarios. E.g. completely spitballing here, say you had a [SkipFinally] or [SkipDropOnFault] attribute (or whatever name) that you could add to the resource type. In that case the compiler could see that and just emit the call to Drop() inline after the other statements of the scope that just finished, without injecting a try/finally.

Otherwise e.g. this shouldn't really be used for array pooling at all 😅

[hamarb123]

I am thinking the logical way is to have 2x Drop - 1 for exceptional and one for non-exceptional - this can then work well with the auto-impl aspect & generics. JIT should hopefully be able to opt

try
{
    ...
    x.DropNonCritical();
}
fault
{
    x.DropCritical();
}

when DropNonCritical and/or DropCritical are no-ops (or if not, then it could be fixed).

[jkotas]

My understanding is that the guidance is to explicitly not return arrays to the pool from a finally block

It is "consider" guidance. It only applies when the exceptions are expected to be very rare on the given code path.

It is defense in depth for bugs in manually managed lifetime. It is not relevant when the correctness of lifetime management is enforced by the language.

[hamarb123]

In the case of array pools this might be fine, but for other objects they may end up in a broken state in some exceptional cases - imo having a way to have a non-exceptional/non-critical drop would be good - I know I would certainly use it.

[jjonescz]

Suggested change

	* Direct access to `Value` is prohibited. Instead, all `BorrowedReceiver` methods on `T` will re-exported as theough on the `Owned<T>` instance and the compiler is responsible for translating all such invocations through `Value`.
	* Direct access to `Value` is prohibited. Instead, all `BorrowedReceiver` methods on `T` will re-exported as though on the `Owned<T>` instance and the compiler is responsible for translating all such invocations through `Value`.

[jjonescz]

Suggested change

	3. Special thanks to [boats](https://without.boats/)), who's writing has heavily influenced my framing of how to think of a GC hybrid language
	3. Special thanks to [boats](https://without.boats/), who's writing has heavily influenced my framing of how to think of a GC hybrid language

[jjonescz]

Is it allowed to have RentedArray a = default; or RentedArray[] a = new[5];? Then presumably this Drop implementation should handle null _pool or it will crash.