[regalloc-demo] Issues Building Chromium and Extracting Corpus

Hi, I’ve encountered issues building Chromium while experimenting with the suggested (non-)thinLTO configurations. Here are the main challenges:

• Linker Errors: When attempting to build Chromium using various (non-)thinLTO setups, I ran into numerous linker errors that halted progress. A few examples:

ld.lld: error: (...): loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering
ld.lld: error: section .llvmbc at 0x0 of size (...) exceeds available address space
ld.lld: error: File name too long

• Unicode Decode Error: Using a different configuration, I successfully built the project but encountered this error when extracting corpus:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position (...): invalid continuation byte`

After changing the extract_ir.py script to force encoding to be latin-1, the first command line runs but in the end no files get converted.

Given that collecting the Default Trace relies on a complete build (my assumption for some missing libraries errors), resolving this is essential for continuing the demo.

Questions:

• Has there been a recently tested MLGO setup with Chromium that bypasses these issues?
• Would greatly appreciate any advice on resolving the linker errors and the UnicodeDecodeError.

Thanks in advance for your help!

[mtrofin]

Can you share your steps for one example where this happens (git hashes, gclient sync and all) - we did recently do some corpus collection for the latest model (cca August), don't remember any issues.

[boomanaiden154]

For the non-ThinLTO configuration, linker errors are to be expected. They don't actually impact corpus creation though as we only need the object files rather than the final executable.

I would expect clang to keep everything in utf-8, but maybe not? What do you have your system default set to?

Chromium hash 4fd00a6132e76
MLGO hash 6efc0a8
.gclient:

solutions = [
  {
    "name": "src",
    "url": "https://chromium.googlesource.com/chromium/src.git",
    "managed": False,
    "custom_deps": {},
    "custom_vars": {
       "checkout_pgo_profiles" : True
    },
  },
]
target_os = ["linux", "android"]

example args.gn:

is_official_build=true
use_ml_inliner=false
lld_emit_indexes_and_imports=true
is_debug=false
symbol_level=0
enable_nacl=false

=> ld.lld: error: File name too long

+

target_cpu="arm64"
target_os="android"

=> ld.lld: error: (...): loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering

host: Ubuntu 22.04.4 LTS, x86_64

@boomanaiden154 you're right. The issue I encountered when generating the Default Trace in the non-ThinLTO case was related to the path I'd specify for the PGO profile, which wouldn't be matched (flags that were expected to be replaced were not found). Commenting out that check in corpus.py shows that the path specified is still relative, although I've added the absolute path to -fprofile-instrument-use-path. generate_default_trace.py can't find ../../chrome/android/profiles/afdo.prof either, so should its absolute path should also be added to common.gin?

[boomanaiden154]

Yes. You need to setup the problem_config.flags_to_replace.replace_flags={} variable to replace the value of -fprofile-instrument-use-path with the absolute path to the profile. There should be a reasonably close example in compiler_opt/rl/regalloc/gin_configs/common.gin that you just need to uncomment and modify slightly.

Maybe I wasn't clear enough but that's what I did and the path is not parsed properly by corpus.py (matched_replace_flags is empty). Also, seems like the absolute path of ../../chrome/android/profiles/afdo.prof also needs to be specified. Which value would this path replace?

[mtrofin]

@ioana-ghiban-arm why do you need to provide both a iFDO and AFDO profile? (i.e. how does ../../chrome/android/profiles/afdo.prof come into play?)

[ioghiban]

FYI ,the ghost behind the issue is me, I've merged my accounts and kept another username. I'm giving the demos another try these days. Will update with my findings.