CritterWatch with HttpTransport not working

[penenkel]

To be honest I'm not entirely sure this isn't a configuration error on my end, but I followed the WebService.Http sample pretty closely.

The telemetry channel (service->watch) seems to be working, but the control channel (watch->service) does not.
I'm getting the following errors in the watch log:

fail: Wolverine.Transports.Sending.BufferedSendingAgent[201]
Failed to send outgoing envelopes batch to https://localhost:7045/_wolverine/invoke
System.InvalidOperationException: An invalid request URI was provided. Either the request URI must be an absolute URI or BaseAddress must be set.
at System.Net.Http.HttpClient.PrepareRequestMessage(HttpRequestMessage request)
at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Wolverine.Http.Transport.WolverineHttpTransportClient.SendBatchAsync(String uri, OutgoingMessageBatch batch) in /home/runner/work/wolverine/wolverine/src/Http/Wolverine.Http/Transport/WolverineHttpTransportClient.cs:line 16
at Wolverine.Http.Transport.HttpSenderProtocol.SendBatchAsync(ISenderCallback callback, OutgoingMessageBatch batch) in /home/runner/work/wolverine/wolverine/src/Http/Wolverine.Http/Transport/HttpSenderProtocol.cs:line 26
fail: Wolverine.Transports.Sending.BufferedSendingAgent[0]
Failure trying to send a message batch to https://localhost:7045/_wolverine/invoke
System.InvalidOperationException: An invalid request URI was provided. Either the request URI must be an absolute URI or BaseAddress must be set.
at System.Net.Http.HttpClient.PrepareRequestMessage(HttpRequestMessage request)
at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Wolverine.Http.Transport.WolverineHttpTransportClient.SendBatchAsync(String uri, OutgoingMessageBatch batch) in /home/runner/work/wolverine/wolverine/src/Http/Wolverine.Http/Transport/WolverineHttpTransportClient.cs:line 16
at Wolverine.Http.Transport.HttpSenderProtocol.SendBatchAsync(ISenderCallback callback, OutgoingMessageBatch batch) in /home/runner/work/wolverine/wolverine/src/Http/Wolverine.Http/Transport/HttpSenderProtocol.cs:line 26

https://localhost:7045 is the address of the service that's being monitored, so the resolution of the target address seems to work propperly.

I think the issue lies with the HttpClient that the watch console uses to send messages to the service. I debugged into the WolverineHttpTransportClient and noticed the following

As you can see, the URI is used as the key for fetching a named HttpClient (which i assume is intentional) and then that client is used to send the message to the BaseAddress of the client. Which would work fine if said BaseAddress was not null for the resolved client.

I think this ties back to how the HttpClient is registered in the Sample

// The console can also push operator commands (PauseProjection / RebuildProjection / DLQ ops / …) BACK to
// the OrderService over the same HTTP transport. Sending over the transport needs the transport client;
// register it so those sends work when an operator triggers them. (The named HttpClient that targets a
// specific service's control URL is created on demand from the service's reported control URI.)
builder.Services.AddScoped<IWolverineHttpTransportClient, WolverineHttpTransportClient>();
builder.Services.AddHttpClient();

To me it looks like on-demand creation that's mentioned in the comment either never happens or is broken.

Please advice me if my understanding or configuration is faulty.

Also I got the impression that HttpTransport might not be the recommended setup here. I'm trying to build an event sourcing application that (for the moment at least) does not need an external message bus, so from its description HttpTransport sounded like a suitable approach, please advice if that is not the case.

[jeremydmiller]

Confirmed — your reading is exactly right, and it's a Wolverine bug. Fixed in wolverine#3681.

Nothing wrong with your configuration. You followed the sample; the sample documents a contract that
cannot be satisfied for the control direction, and the sample's own comment was misleading about it.

What's happening

WolverineHttpTransportClient uses the outbound URI purely as an IHttpClientFactory client name,
then posts to that client's BaseAddress — as you spotted in the debugger:

var client = clientFactory.CreateClient(uri);
var response = await client.PostAsync(client.BaseAddress, content);

CreateClient never fails on an unknown name; it returns a default client whose BaseAddress is null,
and PostAsync(null, …) throws exactly the error you're seeing. The transport had your absolute URL
(https://localhost:7045/_wolverine/invoke) in hand and declined to use it — which is why the address in
the log message looks correct while the send still fails.

Your telemetry direction works because the monitored service knows its target at startup and the sample
registers a named client for it:

builder.Services.AddHttpClient(telemetryUri.ToString(), c => c.BaseAddress = …);

The console cannot do the same thing. It only learns a service's control URL at runtime, from that
service's own registration, so there is no point at which it could have registered a named client keyed by
that URL. The AddHttpClient() in the console's Program.cs registers the factory, not a named client — and
the comment next to it claiming one is "created on demand" was simply wrong. Sorry; that comment sent you
looking for a configuration mistake that wasn't there.

The fix

wolverine#3681:

• Resolution is now most-specific-first. A named client registered for the destination still wins
  (including the sample's deliberate "https key, http BaseAddress" trick); otherwise the send falls
  through to the transport's own isolated client and posts to the absolute outbound URI.
• AddWolverineHttp() now registers IWolverineHttpTransportClient and a named client of the transport's
  own (HttpTransport.HttpClientName), so neither line has to be added by hand — and there's one place to
  hang a timeout/handler/proxy for transport traffic instead of it inheriting your default HttpClient.

Once that ships you can delete both manual registration lines from your console.

Sample side: CritterStackSamples#7 corrects the
comment and README, and adds the regression test the battery was missing — it only ever covered the
telemetry direction, which is how a permanently-broken control path shipped as a reference sample.

On your last question — is HttpTransport the right choice here?

For "event-sourced app, no external broker, service already exposes HTTP", it is a reasonable fit and is
what this sample exists to show. Two things worth knowing:

• It is a push model — the monitored service is the HTTP client, the console the HTTP server — so the
  console's address has to be reachable from the service, and vice versa for control commands.
• If you later want durability across restarts for the monitoring link, or you grow past one console, the
  Postgres database-queue transport is the lighter step up from here (still no broker to operate, since
  you already have Postgres for the event store).

No need to switch on account of this bug.

[jeremydmiller]

@penenkel The main reason the HTTP transport is even in the mix for me is specifically to enable exactly what you're doing:)

Changes coming in the Wolverine 6.24 for this and another early report. CritterWatch RC.2 will pin the right versions

[penenkel]

Thanks for an other super quick fix. And for the confirmation that I'm not totally of track. 👍