[RFC] arm64 SDK

[vbatts]

Current situation
the flatcar sdk is for an amd64 host. All arm64 image and binaries are cross-compiled.

Impact
now that powerful arm64 machines are readily available, it's currently not an option to use them for builds

Ideal future situation
A flatcar arm64 image can be built from an arm64 host.

TODO

• Provide SDK for ARM64
• Integrate ARM64 SDK into release pipeline and release SDK tarballs (and possibly SDK containers)

[pothos]

A first step would be to align the USE flags and related things of the package setup to be equal on or independent of both architectures.
The SDK bootstrap needs a seed tar ball with a compiler – I guess the arm64 developer container can be used for that.

[jepio]

So I managed to get it working with the arm64 developer container as a base, however I would not recommend trying that a second time: the developer container has the wrong profile and CHOST, making the process trickier than necessary to begin with. After that it's all about keywords and fixing x86 specific assumptions.

The built SDK can be fetched from https://jepio.blob.core.windows.net/flatcar-arm64/stage4-arm64-2920.0.0+2021-07-13-1007.tar.bz2 https://jepio.azureedge.net/flatcar-arm64/2942.0.0/flatcar-sdk-arm64-2942.0.0+2021-07-27-0724.tar.bz2. The binary packages, intermediate stages and my hacked up developer container can be found in the same bucket (https://jepio.blob.core.windows.net/flatcar-arm64?restype=container&comp=list).

I believe most of the changes necessary to get the bootstrap working (starting from a reasonable seed) can be merged, I'll submit the PRs in the next weeks.

[vbatts]

@jepio that's awesome! that was the path I began, but abandoned when I was trying from my pinebook. Looking forward to it being an option

[jepio]

Here's a bit more instructions for how to use this @dongsupark:

git clone https://github.com/kinvolk/mantle
pushd mantle
./build cork
# move to somewhere on your PATH
popd

# pull my key
gpg --keyserver hkps://keys.openpgp.org --recv-keys 3717D1B5C719A9BD

mkdir -p flatcar-sdk/.cache/sdks/
pushd flatcar-sdk
latest=$(curl https://jepio.azureedge.net/flatcar-arm64/latest.txt | awk '{ print $1 }')
wget -O .cache/sdks/flatcar-sdk-arm64-2942.0.0.tar.bz2 "https://jepio.azureedge.net/flatcar-arm64/${latest}"
wget -O .cache/sdks/flatcar-sdk-arm64-2942.0.0.tar.bz2.sig "https://jepio.azureedge.net/flatcar-arm64/${latest}.sig"
gpg --armor --export jpiotrowski@microsoft.com >key.gpg
cork create --sdk-version 2942.0.0 --verify-key key.gpg
cork enter
git -C ~/trunk/src/scripts/ checkout jepio/arm64-sdk-support
git -C ~/trunk/src/third_party/coreos-overlay/ checkout jepio/arm64-sdk-support
./update_chroot
./boostrap_sdk --seed_tarball /mnt/host/source/.cache/sdks/flatcar-sdk-arm64-2942.0.0.tar.bz2

[pothos]

Currently the SDK sets up QEMU process emulation for arm64 and now you would need to set up the reverse direction because during compilation of some packages the build system has to run some amd64 binaries.

The key action is setting up QEMU_LD_PREFIX and the rest can actually be removed because the normal Debian/Fedora packing of qemu-user loads the qemu binary into RAM on boot and we don't need to set it up from the SDK with dynamic loading which does not cover all cases.

[jepio]

I agree.

It doesn't look like it's going to be as "easy" as crosscompiling arm64 from amd64 though, because some programs that are called from the SDK for image building (syslinux, x86-specific parts of grub) can't be built for arm64 natively.

My first objective is to get arm64 -> arm64 working, and get us into a state where we can support the SDK infrastructure on our servers. Without that, it's a pain to use.

[dongsupark]

In general it looks good.
Following @jepio's instruction, I was able to create an arm64 SDK on an arm64 host. (actually an arm64 VM on Mac M1)
From that tarball, I created another flatcar-sdk environment.
Inside that, I was able to successfully build an arm64 qemu image, with fantastic speed.

However some hacks or tweaks are needed.
I am just listing all, probably some of them are just my testing failure.

• mantle/cork needed code change to skip verifying gpg signature. Not sure why.
• mantle/cork needed code change to skip downloading SDK tarballs, because arm64 tarballs are not on flatcar-linux.net yet.
• build_image failed with ERROR : Required WHITE_LIST items ld-linux-x86-64.so.2 not found!!!. That's because WHITELIST has only x86_64 linker. I am not sure how we should support multi-arch there.
• After creating an qemu image, running flatcar_production_qemu_uefi.sh fails, because the script simply assumes the host is x86_64. e.g. accel=kvm.

[pothos]

For the last point: the script already checks for an arm64 host but it assumes that KVM is available, maybe you can enable nested virtualization for the host VM?

[jepio]

Nested virt is not available on M1 right now.

@dongsupark are you using qemu for virtualization? You could try to adapt the script to get it working when ran from the host. Probably some of the helper tools are not available, and instead of accel=kvm you need accel=hvf.

[dongsupark]

are you using qemu for virtualization? You could try to adapt the script to get it working when ran from the host. Probably some of the helper tools are not available, and instead of accel=kvm you need accel=hvf.

I am using UTM, basically a wrapper around qemu.
UTM is already passing accel=hvf as expected.
Still inside the guest Linux VM, the Flatcar script fails.
Anyway don't worry, that's not super critical, I could find out other options for testing qemu images. ;-)

[dongsupark]

As for the multi-arch issue in generate_au_zip, I ended up writing a PR like that: flatcar/scripts#141