Compile time cache for `torch.compile` in flashdreams

[fangjunzhou-nv]

torch.compile is taking a very long time when flashdreams boots a large model. This impact benchmarks and CI significantly and it's wasting our CI capacity.

We need to investigate this and improve the model compile time.

I believe the Triton autotune is cached already but torch.compile is not.

Possible solution: https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html