[bp/1.37] Don't use sysroot when building using non-hermetic toolchain (#43681)

Commit Message:

Currently, even when using non-hermetic toolchain we in a few places use
Envoy sysroot. Depending on the environment you use when building it can
cause linking issues.

The problem is that sysroot contains a libc library, while libc++ comes
with the toolchain. If libc++ coming with the toolchain was built
against a different version of libc than the one we have in Envoy
sysroot, libc++ may contain references to the symbols that do not exist
in the libc that we have in Envoy sysroot - that results in linking
failures.

In practice it affects systems that use libc version much newere than
what we have in Envoy sysroot. The newer version of libc exposes more
functions and libc++, when linked againt it, takes advantage of those
newer symbols. As a result, when you combine libc++ linked against a
newer version of libc and older version of libc you get linking issues
due to a bunch of undefined symbols that libc++ uses.

Envoy sysroot currently supports libc version 2.31 (the default) and
2.28 (which seem to only exist for Istio, because Istio intentionally
wants to use as old version of libc as practical, because it needs to
distribute Envoy binaries that will run outside of container
environments and using a very old version of libc makes it easier).

glibc 2.31 is 6 years old at this point, so many distribution (both
popular like Ubuntu and more rare like Azure Linux) moved on to newer
versions of glibc.

There are a few ways we can handle this situation:

1. We can add more sysroot versions with newer versions of libc - given
that we don't test non-hermetic toolchain builts and different versions
of sysroot on CI, I think it's better to not place the burden of
supporting different sysroots on the community.
2. We can add libc++ to the sysroot - I didn't test it, but I'd imagine
it's possible to make it work, but because hermetic toolchain already
comes with libc++ linked against an old enough version of libc to not
cause problems, that will introduce some duplication (both sysroot and
toolchain will have a version of libc++ libraries)
3. We can say that non-hermetic builds for Envoy just aren't supported
at all - that's undesirable, because LLVM toolchains on GitHub are only
available for a few operating systems (Ubuntu, RedHat) and if you're not
using any of those, you are out of luck.
4. When not using hermetic toolchain, just don't provide sysroot and let
the host toolchain figure out how and where to find the libraries in the
host.

The option 4 makes most sense to me, so this PR implements that, but we
can discuss other options as well.

Additional Description:

It would be nice to cherry pick this to 1.37 release branch as well,
that is if we aggree that this approach makes sense.

Risk Level: low
Testing: I did test builds both with hermetic toolchain and with host
toolchain for both FIPS and non-FIPS builds to verify that the changes
work. And CI run regular tests with hermetic toolchain.
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

Author: krinkinmu

bazel/external/fips_build.bzl

@@ -5,17 +5,23 @@ BUILD_COMMAND = """
 set -eo pipefail
 
 # c++
-SYSROOT="$$(realpath $$(dirname "$(location %s)"))"
+SYSROOT="%s"
+if [ -n "$${SYSROOT}" ]; then
+    SYSROOT="$$(realpath $$(dirname "$${SYSROOT}"))"
+    export SYSROOT_FLAG="--sysroot=$${SYSROOT}"
+else
+    export SYSROOT_FLAG=""
+fi
 export CC="$$(realpath $(CC))"
 # bazel doesnt expose CXX so we have to construct it (or use foreign_cc)
 if [[ "%s" == "libc++" ]]; then
-    export CXXFLAGS="-stdlib=libc++ --sysroot=$${SYSROOT}"
-    export LDFLAGS="-fuse-ld=lld -stdlib=libc++ -lc++ -lc++abi -lm -pthread --sysroot=$${SYSROOT}"
+    export CXXFLAGS="-stdlib=libc++ $${SYSROOT_FLAG}"
+    export LDFLAGS="-fuse-ld=lld -stdlib=libc++ -lc++ -lc++abi -lm -pthread $${SYSROOT_FLAG}"
 else
-    export CXXFLAGS="--sysroot=$${SYSROOT}"
-    export LDFLAGS="-fuse-ld=lld -lstdc++ -lm -pthread --sysroot=$${SYSROOT}"
+    export CXXFLAGS="$${SYSROOT_FLAG}"
+    export LDFLAGS="-fuse-ld=lld -lstdc++ -lm -pthread $${SYSROOT_FLAG}"
 fi
-export CGO_CFLAGS="--sysroot=$${SYSROOT}"
+export CGO_CFLAGS="$${SYSROOT_FLAG}"
 export CGO_CXXFLAGS="$${CXXFLAGS}"
 export CGO_LDFLAGS="$${LDFLAGS}"
 
@@ -63,14 +69,20 @@ PYTHON_BIN=$$(realpath $(PYTHON3))
 export CC="$$(realpath $(CC))"
 export CXX="$$(realpath $(CC))"
 export AR="$$(realpath $(AR))"
-SYSROOT="$$(realpath $$(dirname "$(location %s)"))"
+SYSROOT="%s"
+if [ -n "$${SYSROOT}" ]; then
+    SYSROOT="$$(realpath $$(dirname "$${SYSROOT}"))"
+    export SYSROOT_FLAG="--sysroot=$${SYSROOT}"
+else
+    export SYSROOT_FLAG=""
+fi
 # bazel doesnt expose CXX so we have to construct it (or use foreign_cc)
 if [[ "%s" == "libc++" ]]; then
-    export CXXFLAGS="-stdlib=libc++ --sysroot=$${SYSROOT}"
-    export LDFLAGS="-fuse-ld=lld -stdlib=libc++ -lc++ -lc++abi -lm -pthread --sysroot=$${SYSROOT}"
+    export CXXFLAGS="-stdlib=libc++ $${SYSROOT_FLAG}"
+    export LDFLAGS="-fuse-ld=lld -stdlib=libc++ -lc++ -lc++abi -lm -pthread $${SYSROOT_FLAG}"
 else
-    export CXXFLAGS="--sysroot=$${SYSROOT}"
-    export LDFLAGS="-fuse-ld=lld -lstdc++ -lm -pthread --sysroot=$${SYSROOT}"
+    export CXXFLAGS="$${SYSROOT_FLAG}"
+    export LDFLAGS="-fuse-ld=lld -lstdc++ -lm -pthread $${SYSROOT_FLAG}"
 fi
 cd $$SRC_DIR
 OUTPUT=$$(mktemp)
@@ -82,21 +94,34 @@ fi
 cp ninja $$OUT_FILE
 """
 
-def _create_build_config(prefix, lib, arch, arch_alias):
+def _config_name(prefix, arch, lib, hermetic_sysroot):
+    return "%s_%s_%s%s" % (prefix, arch, lib, "_hermetic_sysroot" if hermetic_sysroot else "")
+
+def _create_build_config(prefix, lib, arch, arch_alias, hermetic_sysroot):
     """Create the config_setting_group combination."""
     conditions = ["@platforms//cpu:%s" % arch]
+
+    # We currently support only libc++ and libstdc++, so these variants are mutually exclusive
     if lib == "libc++":
         conditions.append("@envoy//bazel:libc++_enabled")
+    else:
+        conditions.append("@envoy//bazel:libstdc++_enabled")
+
+    if hermetic_sysroot:
+        conditions.append("@envoy_repo//:use_hermetic_sysroot")
+    else:
+        conditions.append("@envoy_repo//:use_local_sysroot")
+
     selects.config_setting_group(
-        name = "%s_%s_%s" % (prefix, arch, lib),
+        name = _config_name(prefix, arch, lib, hermetic_sysroot),
         match_all = conditions,
     )
 
-def _create_boringssl_fips_build_command(lib, arch, arch_alias):
+def _create_boringssl_fips_build_command(lib, arch, arch_alias, hermetic_sysroot):
     """Create the command."""
-    _create_build_config("boringssl", lib, arch, arch_alias)
+    _create_build_config("boringssl", lib, arch, arch_alias, hermetic_sysroot)
     return BUILD_COMMAND % (
-        "@sysroot_linux_%s//:WORKSPACE" % arch_alias,
+        ("$(location @sysroot_linux_%s//:WORKSPACE)" % arch_alias) if hermetic_sysroot else "",
         lib,
         "@fips_cmake_linux_%s" % arch,
         "@fips_go_linux_%s" % arch_alias,
@@ -105,31 +130,35 @@ def _create_boringssl_fips_build_command(lib, arch, arch_alias):
 def boringssl_fips_build_command(arches, libs):
     """Create conditional commands from the cartesian product of possible arches/stdlib."""
     return {
-        ":boringssl_%s_%s" % (arch, lib): _create_boringssl_fips_build_command(
+        ":%s" % _config_name("boringssl", arch, lib, hermetic_sysroot): _create_boringssl_fips_build_command(
             lib,
             arch,
             arch_alias,
+            hermetic_sysroot,
         )
         for arch, arch_alias in arches.items()
         for lib in libs
+        for hermetic_sysroot in [False, True]
     }
 
-def _create_ninja_build_command(lib, arch, arch_alias):
+def _create_ninja_build_command(lib, arch, arch_alias, hermetic_sysroot):
     """Create the command."""
-    _create_build_config("ninja", lib, arch, arch_alias)
+    _create_build_config("ninja", lib, arch, arch_alias, hermetic_sysroot)
     return NINJA_BUILD_COMMAND % (
-        "@sysroot_linux_%s//:WORKSPACE" % arch_alias,
+        ("$(location @sysroot_linux_%s//:WORKSPACE)" % arch_alias) if hermetic_sysroot else "",
         lib,
     )
 
 def ninja_build_command(arches, libs):
     """Create the ninja command conditioned to correct stdlib."""
     return {
-        ":ninja_%s_%s" % (arch, lib): _create_ninja_build_command(
+        ":%s" % _config_name("ninja", arch, lib, hermetic_sysroot): _create_ninja_build_command(
             lib,
             arch,
             arch_alias,
+            hermetic_sysroot,
         )
         for arch, arch_alias in arches.items()
         for lib in libs
+        for hermetic_sysroot in [False, True]
     }

bazel/repo.bzl

@@ -93,7 +93,19 @@ def _envoy_repo_impl(repository_ctx):
     llvm_path = repository_ctx.os.environ.get("BAZEL_LLVM_PATH", "")
     local_llvm = "True" if llvm_path else "False"
 
-    repository_ctx.file("compiler.bzl", "LLVM_PATH = '%s'" % (llvm_path))
+    # By default, even when local toolchain is used, we still use the hermetic
+    # sysroot, when it's undesirable, you can set this environment variable to True
+    # to fallback to the host libc.
+    #
+    # NOTE: The cleanest way to provide this environment variable would be via Bazel's
+    # repo_env flag. It's particularly important when using remote build execution (aka
+    # RBE), as host environment variables are not directly passed to remote workers.
+    local_sysroot = repository_ctx.os.environ.get("BAZEL_USE_HOST_SYSROOT", "False")
+
+    repository_ctx.file("compiler.bzl", """
+LLVM_PATH = '%s'
+USE_LOCAL_SYSROOT = %s
+""" % (llvm_path, local_sysroot))
     repository_ctx.file("version.bzl", "VERSION = '%s'\nAPI_VERSION = '%s'" % (version, api_version))
     repository_ctx.file("path.bzl", "PATH = '%s'" % repo_version_path.dirname)
     repository_ctx.file("envoy_repo.py", "PATH = '%s'\nVERSION = '%s'\nAPI_VERSION = '%s'" % (repo_version_path.dirname, version, api_version))
@@ -121,6 +133,34 @@ config_setting(
     visibility = ["//visibility:public"],
 )
 
+bool_flag(
+    name = "use_local_sysroot_flag",
+    build_setting_default = %s,
+    visibility = ["//visibility:public"],
+)
+
+config_setting(
+    name = "use_local_sysroot",
+    flag_values = {
+        ":use_local_sysroot_flag": "True",
+    },
+    constraint_values = [
+        "@platforms//os:linux",
+    ],
+    visibility = ["//visibility:public"],
+)
+
+config_setting(
+    name = "use_hermetic_sysroot",
+    flag_values = {
+        ":use_local_sysroot_flag": "False",
+    },
+    constraint_values = [
+        "@platforms//os:linux",
+    ],
+    visibility = ["//visibility:public"],
+)
+
 py_library(
     name = "envoy_repo",
     srcs = ["envoy_repo.py"],
@@ -253,7 +293,7 @@ py_console_script_binary(
     data = [":envoy_repo.py"],
 )
 
-''' % local_llvm)
+''' % (local_llvm, local_sysroot))
 
 _envoy_repo = repository_rule(
     implementation = _envoy_repo_impl,

bazel/toolchains.bzl

@@ -1,4 +1,4 @@
-load("@envoy_repo//:compiler.bzl", "LLVM_PATH")
+load("@envoy_repo//:compiler.bzl", "LLVM_PATH", "USE_LOCAL_SYSROOT")
 load("@envoy_toolshed//repository:utils.bzl", "arch_alias")
 load("@toolchains_llvm//toolchain:rules.bzl", "llvm_toolchain")
 
@@ -15,7 +15,7 @@ def envoy_toolchains():
         name = "llvm_toolchain",
         llvm_version = "18.1.8",
         cxx_standard = {"": "c++20"},
-        sysroot = {
+        sysroot = {} if USE_LOCAL_SYSROOT else {
             "linux-x86_64": "@sysroot_linux_amd64//:sysroot",
             "linux-aarch64": "@sysroot_linux_arm64//:sysroot",
         },