Voice PE using external Home-Assistant Cloud URL instead of local - no voice response and onboarding issues with https and local-only access

[whitty]

Hi - I'll apologise up front if this is a little incoherent, I'm still trying to understand the issue itself, and only just got to part of the crux of the issue.

Background:

1. I've had voice PE working on my network since its initial release
2. I've explicitly avoided adopting the devices
3. All of my esphome devices, including Voice PEs (and other IOT devices) are segregated on a dedicated IOT WiFi network running on a VLAN which has no internet access and no access to local networking except the HA server itself (and DNS).
4. Things were working fine until recently when I deployed HTTPS via lets encrypt across my whole local network, including setting up the HA server via the Lets Encrypt add-on.

The issue:

After enabling HTTPS I eventually noticed

1. Music Assistant playback is fine
2. I can use wake-words, send commands to perform tasks to HA and have them take effect ("ok Nabu, turn on the bedroom fan")
3. There is no longer an acknowledgement voice, instead the ring just spins the anti-clockwise for about 30 seconds
4. There haven't been any firmware updates (one was still running 2024 era firmware)

Everything else is fine about HA - app works locally and remotely, local browsers etc,...

Investigations

I tried resetting and re-onboarding the devices, however it fails with the error "The voice assistant is unable to connect to Home Assistant" with no further information. I've tried fiddling with the network URL but I'm sure its correct since it works in all other aspects. I've verified that DNS and certificate validation works fine from computers connected to the IOT WiFi network, the router is providing the local addresses and the full certificate chain is provided. I've looked through #258 and #536 but wasn't sure it applied.

Eventually I installed chromium so I could download up-to-date ish firmware (25.12.4) to check in case there were certificate errors. This gives access to the serial console so I looked for errors. There was an error trying to access the esphome site

[14:11:34][E][http_request.update:052][update_task]: Failed to fetch manifest from https://firmware.esphome.io/home-assistant-voice-pe/home-assistant-voice/manifest.json

But that seems fair enough (it would be nice if Home Assistant could check, instead of the device itself, since Home Assistant does have network access).

The only other interesting messages referred to a media URL which was sited on my nabu-casa public URL - I'm not sure if that's intentional or not, but it had me suspicious that it was not my internal https URL.

Eventually it occurred to move the device off the IOT WiFi Network in case it was trying to use my external URL, and that's when the device started working, I was able to go through the onboarding and now voice response is working.

Summary

Having passed though above and now reverted back to the IOT WiFi network (things are failing - after not having worked properly at all (spinning white disk at boot until unplugging from power) I think I know what the issue is, but not sure how to proceed.

I get a successful local connection to HA - this must be via the local address because of the local IP. I don't remember these messages prior to resetting the network and onboarding, and the browser loves to lose log history, so I can't tell yet.

[14:15:26][W][component:373]: wifi cleared Warning flag
[14:15:27][D][api:136]: Accept 192.168.1.46
[14:15:27][W][component:373]: api cleared Warning flag
[14:15:27][D][api.connection:1398]: aioesphomeapi (192.168.1.46) connected
[14:15:27][D][api.connection:264]: aioesphomeapi (192.168.1.46) disconnected
[14:15:27][W][api.connection:1978]: aioesphomeapi (192.168.1.46): Reading failed CONNECTION_CLOSED errno=128

I don't know if the reading failed is a normal transaction or not

A speech recognition cycle now looks like

[14:17:03][D][voice_assistant:478]: State changed from STOP_MICROPHONE to STOPPING_MICROPHONE
[14:17:03][D][voice_assistant:478]: State changed from STOPPING_MICROPHONE to AWAITING_RESPONSE
[14:17:03][D][voice_assistant:624]: Event Type: 4
[14:17:03][D][voice_assistant:663]: Speech recognised as: "Turn off the bedroom fan."
[14:17:03][D][voice_assistant:624]: Event Type: 5
[14:17:03][D][voice_assistant:668]: Intent started
[14:17:03][D][voice_assistant:624]: Event Type: 6
[14:17:03][D][voice_assistant:624]: Event Type: 7
[14:17:03][D][voice_assistant:721]: Response: "Turned off the switch"
[14:17:03][D][voice_assistant:624]: Event Type: 8
[14:17:03][D][voice_assistant:743]: Response URL: "https://mylongprivateid.ui.nabu.casa/api/tts_proxy/ZqbK0hcg4dLe9AafPuIqbg.flac"
[14:17:03][D][voice_assistant:478]: State changed from AWAITING_RESPONSE to STREAMING_RESPONSE
[14:17:03][D][voice_assistant:485]: Desired state set to STREAMING_RESPONSE
[14:17:03][D][voice_assistant:624]: Event Type: 2
[14:17:03][D][voice_assistant:766]: Assist Pipeline ended
[14:17:03][D][light:090]: 'voice_assistant_leds' Setting:
[14:17:03][D][light:078]:   Brightness: 100%
[14:17:03][D][light:164]:   Effect: 'Replying'
[14:17:03][D][media_player:096]: 'Media Player' - Setting
[14:17:03][D][media_player:103]:   Media URL: https://mylongprivateid.ui.nabu.casa/api/tts_proxy/ZqbK0hcg4dLe9AafPuIqbg.flac
[14:17:03][D][media_player:109]:  Announcement: yes
[14:17:03][D][http_media_source:244]: Started read and decode tasks for pipeline 1
[14:17:03][D][http_media_source:253]: Pipeline 1 starting
[14:17:05][D][voice_assistant:478]: State changed from STREAMING_RESPONSE to RESPONSE_FINISHED
[14:17:05][D][voice_assistant:485]: Desired state set to RESPONSE_FINISHED
[14:17:05][D][voice_assistant:478]: State changed from RESPONSE_FINISHED to IDLE
[14:17:05][D][voice_assistant:485]: Desired state set to IDLE
[14:17:05][D][light:090]: 'voice_assistant_leds' Setting:
[14:17:05][D][light:103]:   State: OFF
[14:17:05][D][light:164]:   Effect: 'None'
[14:17:08][E][http_media_source:264]: Pipeline 1 error occurred during playback
[14:17:08][D][http_media_source:271]: Pipeline 1 both tasks finished
[14:17:08][D][esp-idf:000][HTTPRead_1]: E (120398) esp-tls: [sock=59] select() timeout
[14:17:08][D][esp-idf:000][HTTPRead_1]: E (120399) esp-tls: Failed to open new connection
[14:17:08][D][esp-idf:000][HTTPRead_1]: E (120399) transport_base: Failed to open a new connection
[14:17:08][D][esp-idf:000][HTTPRead_1]: E (120399) HTTP_CLIENT: Connection failed, sock < 0
[14:17:08][E][component:362][HTTPRead_1]: http_request set Error flag: unspecified
[14:17:08][E][http_request.idf:131][HTTPRead_1]: HTTP Request failed: ESP_ERR_HTTP_CONNECT
[14:17:08][E][http_media_source:391][HTTPRead_1]: Pipeline 1: Failed to start HTTP request
[14:17:09][E][component:379]: http_request cleared Error flag

This demonstrates why the voice response fails - its using the remote URL (which is not accessible on the IOT WiFi), not the local URL.

I'll follow up separately to see if I can get details out of the onboarding failure.

[whitty]

OK - here's a log from onboarding - looks like its essentially the same problem(s) as above. Completion of the onboarding requires the response URL record to work, but its using the external URL

[15:01:11][D][esp-idf:000][update_task]: E (86576) esp-tls: [sock=59] select() timeout
[15:01:11][D][esp-idf:000][update_task]: E (86576) esp-tls: Failed to open new connection
[15:01:11][D][esp-idf:000][update_task]: E (86576) transport_base: Failed to open a new connection
[15:01:11][D][esp-idf:000][update_task]: E (86576) HTTP_CLIENT: Connection failed, sock < 0
[15:01:11][E][component:362][update_task]: http_request set Error flag: unspecified
[15:01:11][E][http_request.idf:131][update_task]: HTTP Request failed: ESP_ERR_HTTP_CONNECT
[15:01:11][E][http_request.update:052][update_task]: Failed to fetch manifest from https://firmware.esphome.io/home-assistant-voice-pe/home-assistant-voice/manifest.json
[15:01:12][E][component:379]: http_request cleared Error flag
[15:01:16][D][light:090]: 'voice_assistant_leds' Setting:
[15:01:16][D][light:103]:   State: ON
[15:01:16][D][light:078]:   Brightness: 100%
[15:01:16][D][light:164]:   Effect: 'Replying'
[15:01:16][D][media_player:096]: 'Media Player' - Setting
[15:01:16][D][media_player:100]:   Command: ENQUEUE
[15:01:16][D][media_player:103]:   Media URL: https://mylongprivateid.ui.nabu.casa/api/esphome/ffmpeg_proxy/ca2b62c21b55a5364d080a96a9569508/Hjd5t-KiqhpYNTWGI-ncCQ.flac
[15:01:16][D][media_player:109]:  Announcement: yes
[15:01:16][D][voice_assistant:478]: State changed from IDLE to STREAMING_RESPONSE
[15:01:16][D][voice_assistant:485]: Desired state set to STREAMING_RESPONSE
[15:01:16][D][http_media_source:244]: Started read and decode tasks for pipeline 1
[15:01:16][D][http_media_source:253]: Pipeline 1 starting
[15:01:18][D][voice_assistant:478]: State changed from STREAMING_RESPONSE to RESPONSE_FINISHED
[15:01:18][D][voice_assistant:485]: Desired state set to RESPONSE_FINISHED
[15:01:18][D][voice_assistant:478]: State changed from RESPONSE_FINISHED to IDLE
[15:01:18][D][voice_assistant:485]: Desired state set to IDLE
[15:01:18][D][light:090]: 'voice_assistant_leds' Setting:
[15:01:18][D][light:103]:   State: OFF
[15:01:18][D][light:164]:   Effect: 'None'
[15:01:21][E][http_media_source:264]: Pipeline 1 error occurred during playback
[15:01:21][D][http_media_source:271]: Pipeline 1 both tasks finished
[15:01:21][D][esp-idf:000][HTTPRead_1]: E (96631) esp-tls: [sock=59] select() timeout
[15:01:21][D][esp-idf:000][HTTPRead_1]: E (96631) esp-tls: Failed to open new connection
[15:01:21][D][esp-idf:000][HTTPRead_1]: E (96631) transport_base: Failed to open a new connection
[15:01:21][D][esp-idf:000][HTTPRead_1]: E (96631) HTTP_CLIENT: Connection failed, sock < 0
[15:01:21][E][component:362][HTTPRead_1]: http_request set Error flag: unspecified
[15:01:21][E][http_request.idf:131][HTTPRead_1]: HTTP Request failed: ESP_ERR_HTTP_CONNECT
[15:01:21][E][http_media_source:391][HTTPRead_1]: Pipeline 1: Failed to start HTTP request
[15:01:22][E][component:379]: http_request cleared Error flag

[whitty]

Looks like the reason this tripped might be here on the host side when called from tts

    if prefer_external is None:
        prefer_external = hass.config.api is not None and hass.config.api.use_ssl

That might make sense if SSL is being used to expose ourselves externally, but SSL is being used internally in my case.