xds: application receives CANCELLED (grpc-status=1, grpc-message="CANCELLED") on in-flight streams during EDS locality-weight updates (1.82.2)

[sushanb]

Environment

• grpc-java 1.82.2, grpc-xds 1.82.2 (Traffic Director as xDS control plane)
• grpc-opentelemetry 1.82.2
• Client library: google-cloud-bigtable 2.81.0 / gax-grpc 2.83.0

Symptom

When Traffic Director sends an EDS ClusterLoadAssignment update that changes only locality weights (localities not added/removed), a subset of in-flight streams close with:

io.grpc.StatusRuntimeException: CANCELLED: CANCELLED

The description string is literally the 9 characters CANCELLED — a null description would render as CANCELLED alone with no colon-space-word. Application layers above grpc-java (google-cloud-bigtable / gax retry) do not retry CANCELLED, so this surfaces to the app.

Stacktrace

io.grpc.StatusRuntimeException: CANCELLED: CANCELLED
    at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:150) ~[gax-grpc-2.83.0.jar:2.83.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.cloud.bigtable.data.v2.stub.MetadataExtractorInterceptor$1$1.onClose(MetadataExtractorInterceptor.java:82) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.cloud.bigtable.gaxx.grpc.BigtableChannelPool$ReleasingClientCall$1.onClose(BigtableChannelPool.java:782) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:653) ~[gax-grpc-2.83.0.jar:2.83.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.opentelemetry.OpenTelemetryMetricsModule$MetricsClientInterceptor$1$1.onClose(OpenTelemetryMetricsModule.java:760) ~[grpc-opentelemetry-1.82.2.jar:1.82.2]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.cloud.bigtable.data.v2.stub.CookiesInterceptor$UpdateCookieListener.onClose(CookiesInterceptor.java:92) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.82.2.jar:1.82.2]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.82.2.jar:1.82.2]
    at com.google.api.gax.grpc.GrpcLoggingInterceptor$1$1.onClose(GrpcLoggingInterceptor.java:100) ~[gax-grpc-2.83.0.jar:2.83.0]
    at io.grpc.xds.XdsNameResolver$ConfigSelector$1ClusterSelectionInterceptor$1$1.onClose(XdsNameResolver.java:527) ~[grpc-xds-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:566) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:734) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:715) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.82.2.jar:1.82.2]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.82.2.jar:1.82.2]
    ... 3 more
Wrapped by: com.google.api.gax.rpc.CancelledException: io.grpc.StatusRuntimeException: CANCELLED: CANCELLED
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:108) ~[gax-2.83.0.jar:2.83.0]
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:88) ~[gax-grpc-2.83.0.jar:2.83.0]
    at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:84) ~[gax-grpc-2.83.0.jar:2.83.0]
    at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86) ~[gax-2.83.0.jar:2.83.0]
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:150) ~[gax-grpc-2.83.0.jar:2.83.0]
    ... (identical grpc interceptor chain as above, ending at XdsNameResolver$ClusterSelectionInterceptor:527 → ClientCallImpl.closeObserver:566)
Wrapped by: com.google.cloud.bigtable.data.v2.models.MutateRowsException: Some mutations failed to apply
    at com.google.cloud.bigtable.data.v2.models.MutateRowsException.create(MutateRowsException.java:53) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at com.google.cloud.bigtable.data.v2.stub.mutaterows.MutateRowsAttemptCallable.handleAttemptError(MutateRowsAttemptCallable.java:259) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at com.google.cloud.bigtable.data.v2.stub.mutaterows.MutateRowsAttemptCallable$3.apply(MutateRowsAttemptCallable.java:133) ~[google-cloud-bigtable-2.81.0.jar:2.81.0]
    at com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction.apply(ApiFutures.java:396) ~[api-common-2.66.0.jar:2.66.0]
    at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:238) ~[guava-33.6.0-jre.jar:?]
    ... [application frames elided] ...

Investigation

I traced the producer of this status inside grpc-java v1.82.2:

core/src/main/java/io/grpc/internal/Http2ClientStreamTransportState.java:196 — statusFromTrailers:

private Status statusFromTrailers(Metadata trailers) {
  Status status = trailers.get(InternalStatus.CODE_KEY);
  if (status != null) {
    return status.withDescription(trailers.get(InternalStatus.MESSAGE_KEY));
  }
  ...

This is the only path in v1.82.2 that yields Status.CANCELLED with description exactly "CANCELLED", which means the server is sending HTTP/2 trailers with grpc-status: 1 and grpc-message: CANCELLED.

Questions

1. I though grpc java does not deliver CANCELLED error. is this intended?
2. the customer started seeing once they adopted directpath.

[shivaspeaks]

grpc-java does emit CANCELLED in some paths, but all of them carry a specific description. A bare "CANCELLED" description can only come from statusFromTrailers copying grpc-message off the wire, so the peer sent grpc-status: 1 + grpc-message: CANCELLED. I'd look at the backend/DirectPath data path, not the client.

WeightedTargetLoadBalancer only shuts down children whose target names are removed and then the weights are updated in updateOverallBalancingState(). And since childConfig is unchanged, the child GracefulSwitchLoadBalancer short-circuits instead of swapping policies. Net effect would just be a new WeightedRandomPicker, nothing else, so a weight-only update changes nothing.

Also subchannel shutdown yields UNAVAILABLE: Subchannel shutdown invoked or something, not CANCELLED.

[sushanb]

Thanks. @shivaspeaks

We use c++ server. Is it a valid pattern for server to send peer sent grpc-status: 1 + grpc-message: CANCELLED