Script hangs indefinitely

[FMG-Jay]

Hi,

While trying to use the script, after Tailscale starts and it prints the IP & MagicDNS name the script appears to hang until interrupted, I have left it for about 30mins and nothing is happening. This occurs both with & without the --staging flag. *

I am using a fresh Tailscale account/Tailnet, and a fresh install of Proxmox - However, it is a "pre-configured" base from Ionos on one of their machines, so I don't know if something they have done may be affecting it.

If I use Ctrl+C to interrupt, the script then prints Tailscale serve is now running. & Tailscale service started for tailmox. But the Service shows no hosts in the Tailscale Admin Console.

*Initially I had also tried to run the script on three internal Nodes which are already clustered, and these hung in the same place as well.

[willjasen]

HI! Are you using the main or dev branch? The dev branch has come a long way from main and adds many features while addressing some bugs. I can't say for sure that you won't run into the same thing, but give it a try and let me know your results please!

[FMG-Jay]

I did a direct git clone, so I assume its the main branch - I'll try dev now.

[willjasen]

Take note that dev sets up a web service for managing tailmox. You should see a Tailscale MagicDNS URL at the end if all is successful.

The first host shoudln't have much troubles in terms of tailmox because it would be the first cluster member (thus creating a new cluster). If a subsequent host hangs (and the cluster connection process shouldn't take longer than a minute), then feel free to post (sanitized) output from around where it seems to hang, and I'll help diagnose.

[FMG-Jay]

Hi,

I seem to have the same issue - If I don't provide an auth key and use the link, it hangs after 'Success.' Using an auth key it hangs after printing the MagicDNS name.

━━━ TAILMOX SETUP ━━━
Staging mode enabled.
Using auth key for Tailscale...
Checking if Proxmox v8 or v9 is installed...
Proxmox VE 9.x detected.
Running from the correct directory: /opt/tailmox
Checking for required dependencies...
Checking the latest age release with post-quantum identity support...
Latest age release 1.3.2 is already installed.
Starting Tailscale...
Waiting for Tailscale to come online...
This host's Tailscale IPv4 address: 100.XX.XX.57 
This host's Tailscale MagicDNS name: [][].ts.net 

[willjasen]

Update to the latest on the dev branch, then try a staging again. The script will prompt for a Tailscale auth key if one is not available, and that auth key must be set within the Tailscale admin console to set "tag:tailmox". I'm removing the link feature for now, so an auth key must be used.

I just ran it myself on a host deployed from the project's image, and I got back:

root@tailmox1:/opt/tailmox# cd /opt/tailmox; git pull --quiet; tailmox stage;
━━━ TAILMOX SETUP ━━━
Staging mode enabled.
Checking if Proxmox v8 or v9 is installed...
Proxmox VE 9.x detected.
Running from the correct directory: /opt/tailmox
Checking for required dependencies...
Checking the latest age release with post-quantum identity support...
Latest age release 1.3.2 is already installed.
Tailscale is already connected with tag:tailmox.
Tailscale DNS is enabled.
Waiting for Tailscale to come online...
This host's Tailscale IPv4 address: 100.95.182.117 
This host's Tailscale MagicDNS name: tailmox1.risk-mermaid.ts.net 
Tailscale serve is now running.
Installing the Tailmox monitoring interface...
Tailmox monitoring interface is running locally on port 8088.
Verified Tailmox monitoring at https://tailmox1.risk-mermaid.ts.net:8088/monitor
Staging mode enabled. Exiting after `tailscale serve` setup.

If it's also helpful, I'm having GitHub Actions run tests with each commit to the dev branch, and you're welcome to peruse that for reference. The actions aren't testing or interacting with an actual setup, just checking for internal consistency.

If it still doesn't work on the image you're using, let me know.

[FMG-Jay]

Hi, I appreciate your help.

The change to the dev branch allowed me to get further! It died on the first run through but prompted me to enable Serve and succeeded the second time.

After a few issues involving ttyd not being present (nor available via apt??) & resolvconf wiping the nameserver settings on a fresh Proxmox VM I'm now having some issues during tailmox cluster:
During step 3; it seems 50/50 on whether or not it will succeed with the 80% tailscale pings, despite manual pings (tailscale and regular) being 100% successful.
If it progresses to Step 5; It is 'Unable to determine Proxmox Cluster Status'.

Reading through the script at these parts and running the commands manually appears to show no issues
Although I am wondering if the difference in route show below is what is causing Step 3 to fail?

root@pve-test:~# tailscale ping --c 1 --timeout=200ms "38965C6"
pong from 38965c6 (100.68.113.41) via DERP(lhr) in 27ms
direct connection not established
root@pve-test:~# tailscale ping --c 1 --timeout=200ms "38965C6"
pong from 38965c6 (100.68.113.41) via 109.228.XXX.XXX:41641 in 29ms
root@pve-test:~# tailscale ping --c 1 --timeout=200ms "38965C6"
pong from 38965c6 (100.68.113.41) via 109.228.XXX.XXX:41641 in 29ms

root@pve-test:~# pvecm status
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?

If it helps as well running tailmox test results in all 37 test files passing.

[willjasen]

@FMG-Jay the issue here is that pve-host is trying to connect to another host via DERP, which is suppressing and preventing the rest of the cluster creation/join:

root@pve-test:~# tailscale ping --c 1 --timeout=200ms "38965C6"
pong from 38965c6 (100.68.113.41) via DERP(lhr) in 27ms

because of the latency considerations with corosync, with some people saying that even a latency greater than 5ms is very bad. before the cluster is allowed to be created, or any settings related to proxmox clustering and tailscale, a slew of tests like pings and tcp checks must all pass. i chose 200ms as threshold for these tests, and if a host can't connect directly via tailscale to all other hosts (and not through derp), then the script will refuse to continue.

what is probably happening here is that a direct connection via tailscale is possible but not quite ready, thus the script sees the initial DERP relay and is failing.

i'm right now busy tending to some other things, but i can eventually account for this and run it through tests.

in full disclosure (but also natürlich), i'm using ai coding tools to improve the project further, so if anyone has that and isn't a total doomer, then iterating would be a lot quicker. an idea i have for the future is to have github actions spin up a temporary worker that can connect to a proxmox host over tailscale, deploy a vm template, create 3 linked clones, and run the project against those to ensure an entire suite test. baby steps...

[willjasen]

i have removed the setup of resolvconf and merged it into dev - #35

this was meant for the virtualized tailmox testing environment, not the actual tailmox code itself.

[willjasen]

i've done some further work still, but noticed way too late that the changes pushed to the dev branch - i think it should be mostly okay because most of those changes involve the virtualized proxmox testing environment and not to the tailmox scripts themselves.