Skip to content

added redirect to pull non-existant local assets from live_endpoint url - #70

Open
sjuerges wants to merge 1 commit into
netbootxyz:masterfrom
sjuerges:master
Open

added redirect to pull non-existant local assets from live_endpoint url#70
sjuerges wants to merge 1 commit into
netbootxyz:masterfrom
sjuerges:master

Conversation

@sjuerges

@sjuerges sjuerges commented Aug 4, 2024

Copy link
Copy Markdown

This PR implements #69

It achieves that the nginx can serve local assets as before, but it will send HTTP 302 redirects to the standard live_endpoint url for files that are NOT found locally.

@antonym

antonym commented Aug 9, 2024

Copy link
Copy Markdown
Member

Seemed to work, but I notice it seems to have issues with the assets that contain a part2 file. It only downloads the first part if not found locally but fails on the second part.

@sjuerges

sjuerges commented Aug 9, 2024

Copy link
Copy Markdown
Author

Weird, wasnt an issue for me, specifically tested for it:
(local):

172.18.0.5 - - [03/Aug/2024:22:02:31 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f/vmlinuz HTTP/1.1" 200 12881792 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                            
172.18.0.5 - - [03/Aug/2024:22:02:32 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f/initrd HTTP/1.1" 200 104039372 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                            
172.18.0.5 - - [03/Aug/2024:22:02:34 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs HTTP/1.1" 206 1 "-" "curl/8.4.0"                                                                                        
172.18.0.5 - - [03/Aug/2024:22:02:52 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs HTTP/1.1" 200 2097151999 "-" "curl/8.4.0"                                                                               
172.18.0.5 - - [03/Aug/2024:22:02:53 +0000] "HEAD /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs.part2 HTTP/1.1" 200 0 "-" "curl/8.4.0"                                                                                 
172.18.0.5 - - [03/Aug/2024:22:02:55 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs.part2 HTTP/1.1" 200 258531329 "-" "curl/8.4.0"  

(remote):

172.18.0.5 - - [03/Aug/2024:21:55:07 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/vmlinuz HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                                            
172.18.0.5 - - [03/Aug/2024:21:55:09 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/initrd HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                                             
172.18.0.5 - - [03/Aug/2024:21:55:19 +0000] "HEAD /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs HTTP/1.1" 302 0 "-" "curl/7.67.0"                                                                                           
172.18.0.5 - - [03/Aug/2024:21:55:19 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs HTTP/1.1" 302 145 "-" "curl/7.67.0"                                                                                          
172.18.0.5 - - [03/Aug/2024:21:57:05 +0000] "HEAD /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs.part2 HTTP/1.1" 302 0 "-" "curl/7.67.0"                                                                                     
172.18.0.5 - - [03/Aug/2024:21:57:05 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs.part2 HTTP/1.1" 302 145 "-" "curl/7.67.0" 

@antonym Which image did you test with, maybe I can replicate/debug?

@nrvale0

nrvale0 commented Aug 19, 2024

Copy link
Copy Markdown

Related but tangential. Hope that's ok...

If I wanted to test this using netboot.xyz running in Docker Compose would it be sufficient to just pull the branch and then use the docker-compose.yml.example? I suspect I might also have to modify the image name also? Or just do a docker build of the default Dockerfile + tags but while this branch is selected in order to pull the modified default file?

EDIT: Yes, changing to the PR branch and rebuilding the image with the same tag/name worked.

@jakelmg

jakelmg commented Dec 26, 2024

Copy link
Copy Markdown

I tried this out and it worked fine. Any word on pulling this? @antonym

@antonym

antonym commented Dec 30, 2024

Copy link
Copy Markdown
Member

@sjuerges

Tested with the Ubuntu 22.04 Image (Gnome), with endpoint set to local machine and cache downloaded:

10.0.100.148 - - [30/Dec/2024:18:35:05 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/vmlinuz HTTP/1.1" 200 14928264 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:35:08 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/initrd HTTP/1.1" 200 129283057 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:35:11 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 200 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:13 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 200 2097151999 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:13 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part2 HTTP/1.1" 200 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:14 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part2 HTTP/1.1" 200 848650241 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:14 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part3 HTTP/1.1" 302 0 "-" "curl/8.4.0"

Removed cache, and attempted, dumps to busy box after the first image pull, doesn't seem to handle the redirect for part2 or part3 checks correctly:

10.0.100.148 - - [30/Dec/2024:18:41:19 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/vmlinuz HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:41:21 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/initrd HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:41:28 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 302 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:41:28 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 302 145 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:42:17 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part2 HTTP/1.1" 302 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:42:17 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part3 HTTP/1.1" 302 0 "-" "curl/8.4.0"

After it pulls the first squashfs part, it dumps to busy box.

Also noticed that the generation of the config file was squashing the variables with envsubst which I'm going to fix as well.

@antonym

antonym commented Dec 30, 2024

Copy link
Copy Markdown
Member

Fix for the vars in this commit getting squashed during new deploy: #78

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants