Improve support for model-spec format; add support for raw layers and zstd compression

[amisevsk]

Description

Improve support for model-spec formatted artifacts in KitOps. Our goal is to support any valid ModelPack transparently within KitOps; as a result this PR adds support for raw layer types (i.e. non-tar) and zstd compression as specified by the model-spec mediatypes.

• Kitfile generation for ModelPacks that do not contain a Kitfile is improved:
  • LayerInfo (digest and diffId) are populated in the generated Kitfile. DiffIds are only included if the model config is available (e.g. in library usage, a config may not be available/desirable)
  • When we generate a Kitfile for a ModelPack, we now read the model config to get DiffIDs.
  • Kitfiles for ModelPacks that contain multiple model.weight layers are now properly generated -- additional model.weight layers are included in modelparts within the Kitfile
• ModelPacks with multiple weight layers are now handled correctly in unpack operations. Even though Kitfiles only support one "model" and N "modelparts"
• Unpacking in general has been refactored and simplified -- the old process is moved into a legacy.go file (as it is required for ModelKits created pre KitOps v0.5.0 when we added LayerInfo) and the new process uses LayerInfo and is more extensively tested.
• kit pack now supports zstd compression and raw layers:
  • Raw layers are supported by a new flag: --layer-format with options tar (default) and raw (no tar wrapper)
  • Compression is supported by an additional zstd option on the existing --compression flag
• kit unpack also now supports zstd compression and raw layers, transparently.
• In addition to significant unit testing additions, functional tests have been updated to verify behaviour across all valid configurations of compression, layer format, and model format (ModelKit vs ModelPack)
  • For raw layer-format artifacts, tests also not ensure that layer diffID (or digest in the no compression case) is the actual file's sha256sum

I've tried to keep the Git commit history useful for this one, as there are a lot of changes here. Each commit should be self-contained and include a meaningful description of the changes within

Linked issues

N/A, forgot to create one

AI-Assisted Code

• This PR contains AI-generated code that I have reviewed and tested (mostly test cases for this one)
• I take full responsibility for all code in this PR, regardless of how it was created

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[gorkem]

@amisevsk Other than the CI failure this looks OK

[amisevsk]

Odd, the sources are well-formatted on my end

[amisevsk]

This is fun... MacOS and Linux disagree on go fmt:

diff --git a/testing/pack-unpack_test.go b/testing/pack-unpack_test.go
index d11b873..0d85bee 100644
--- a/testing/pack-unpack_test.go
+++ b/testing/pack-unpack_test.go
@@ -205,9 +205,9 @@ func TestPackReproducibility(t *testing.T) {
                {"--layer-format", "raw"}, // raw format
                {"--compression", "gzip"}, // tar + gzip
                {"--layer-format", "raw", "--compression", "gzip"}, // raw + gzip
-               {"--use-model-pack"},                               // modelpack default
-               {"--use-model-pack", "--layer-format", "raw"},      // modelpack + raw format
-               {"--use-model-pack", "--compression", "gzip"},      // modelpack + tar + gzip
+               {"--use-model-pack"},                          // modelpack default
+               {"--use-model-pack", "--layer-format", "raw"}, // modelpack + raw format
+               {"--use-model-pack", "--compression", "gzip"}, // modelpack + tar + gzip
        }

        testKitfile := `