`#include` resolution inconsistency: DXC's include handler vs. `ProcessShaderIncludesImpl`

[hzqst]

TL;DR

DiligentCore resolves shader #include "X" directives two different ways:

• At compile time, DXC/FXC maintain include-stack context. Once an include file has been loaded
  as e.g. Lighting/LightingTypes.hlsli, a nested #include "LightingConfig.h" inside that
  file is resolved relative to the loaded include path, so the compiler can request
  Lighting/LightingConfig.h.
• At scan time, DiligentCore's own ProcessShaderIncludesImpl — a hand-rolled text scanner
  used to compute the BY_CONTENT render-state-cache hash (and for hot-reload dependency
  tracking and archiver include enumeration) — passes the literal include string to the
  stream factory with no include-stack / parent-directory resolution, so the same nested
  include is looked up as bare LightingConfig.h.

For shaders with nested directory-relative includes (RTXPT's PathTracer/Lighting/… tree), DXC/FXC
compiles cleanly while the scanner throws Failed to load shader source file 'LightingConfig.h'.
This is a latent DiligentCore bug. RTXPT works around it by hashing the
cache BY_NAME (which never invokes the scanner). The proper fix is to make
ProcessShaderIncludesImpl resolve nested includes against their parent include file's path,
matching DXC/FXC include-stack behavior.

---

Symptom

With the render-state cache in RENDER_STATE_CACHE_FILE_HASH_MODE_BY_CONTENT (the default):

ERROR: Failed to create input stream for source file LightingConfig.h
ERROR: Failed to load shader source file 'LightingConfig.h'
[std::runtime_error / std::pair exceptions]
ERROR: Failed to process includes in file 'PathTracer/Lighting/LightsBaker.hlsl': Unknown error

The shader compiles fine (DXC/FXC resolve the nested include) — only the content-hash step,
which re-scans the source to enumerate includes, fails. In the current RTXPT chain,
LightsBaker.hlsl includes Lighting/LightingTypes.hlsli; then LightingTypes.hlsli includes
the sibling LightingConfig.h. DXC/FXC resolve the second include as Lighting/LightingConfig.h,
but the scanner tries bare LightingConfig.h against the flat search dirs
shaders;shaders\PathTracer, neither of which contains the file at that bare name.

---

The two resolution paths

DXC/FXC keep include-stack context for nested includes

Diligent builds a source string and passes it to DXC/FXC with an empty source name; the emitted
#line 1 "PathTracer/Lighting/LightsBaker.hlsl" marker is useful for diagnostics, but it is not
a reliable mechanism for resolving a top-level sibling include relative to ShaderCI.FilePath.

The RTXPT case that compiles is nested: LightsBaker.hlsl includes
Lighting/LightingTypes.hlsli, which is reachable from the shaders\PathTracer search dir. Once
the compiler has loaded that include file, a sibling include inside it,
#include "LightingConfig.h", is resolved relative to the loaded include path, so DXC/FXC ask
their include handlers for a path equivalent to Lighting/LightingConfig.h.

DxcIncludeHandlerImpl::LoadSource
(DXCompiler.cpp:185) and
D3DIncludeImpl::Open
(ShaderD3DBase.cpp:65) do not compute that parent directory themselves; they open the path
requested by the compiler. The DXC handler additionally strips a leading .\ if DXC provides one:

// DXCompiler.cpp:210-214
if (fileName.size() > 2 && fileName[0] == '.' && (fileName[1] == '\\' || fileName[1] == '/'))
    fileName.erase(0, 2);
m_pStreamFactory->CreateInputStream(fileName.c_str(), &pSourceStream);

shaders\PathTracer (search dir) + Lighting/LightingConfig.h → found. ✔

The scanner passes the literal include text verbatim

ProcessShaderIncludesImpl
(ShaderToolsCommon.cpp:428) is a text scanner. It first opens Lighting/LightingTypes.hlsli;
inside that file, FindIncludes extracts the raw string between the quotes
(LightingConfig.h) and, when it recurses, sets the child's path to that bare name,
discarding the parent include path Lighting/:

// ShaderToolsCommon.cpp:439-448
[&](const std::string& FilePath, size_t Start, size_t End)
{
    if (!Includes.insert(FilePath).second)
        return;

    ShaderCreateInfo IncludeCI{ShaderCI};
    IncludeCI.FilePath     = FilePath.c_str();   // "LightingConfig.h" — parent include path thrown away
    IncludeCI.Source       = nullptr;
    IncludeCI.SourceLength = 0;
    ProcessShaderIncludesImpl(IncludeCI, Includes, IncludeHandler);
}

ReadShaderSourceFile then calls CreateInputStream("LightingConfig.h"), which only sees the
factory's flat search dirs (shaders;shaders\PathTracer), none of which contain Lighting/, and
throws (ShaderToolsCommon.cpp:243-245). ✘

---

The crux

The parent include path is available — during the failing nested recursion,
ShaderCI.FilePath holds Lighting/LightingTypes.hlsli — but the scanner never combines it with
the nested include name. So:

	Include resolution rule
DXC/FXC	include-stack-aware: a nested include is resolved relative to the loaded include file path, then search paths
Diligent scanner	literal include string only; no include-stack / parent-path context

They agree only when every nested include is reachable from a search dir by the exact spelling
used in the source (e.g. Tutorial26's flat assets). RTXPT uses nested directory-relative includes,
so DXC/FXC compile cleanly while the scanner — invoked only to compute the BY_CONTENT cache hash —
fails.

It is a genuine latent bug, not just a quirk: any project with directory-relative shader
includes hits it under content hashing, hot-reload dependency tracking
(ShaderReloadFactory/RenderStateCache::Reload), or the archiver's include enumeration.

---

Scope / impact

Affected whenever ProcessShaderIncludesImpl runs over shaders with directory-relative includes:

• RENDER_STATE_CACHE_FILE_HASH_MODE_BY_CONTENT — content hashing (the failure above).
• Hot reload — ProcessShaderIncludes builds the include→dependency set for Reload().
• Archiver — include enumeration when packaging shader source.

Not affected: actual DXC/FXC compilation (resolves correctly), and BY_NAME hashing.

---

Proposed fix

Make ProcessShaderIncludesImpl resolve each nested include against the path of the currently
scanned include file before recursing — matching DXC/FXC include-stack behavior. DiligentCore
already provides the needed helpers in BasicFileSystem.hpp (already included by
ShaderToolsCommon.cpp): FileSystem::GetPathComponents, FileSystem::IsPathAbsolute,
FileSystem::SimplifyPath, FileSystem::SlashSymbol.

Minimal version (only safe for parent-relative include trees)

In the FindIncludes recursion lambda (ShaderToolsCommon.cpp:439-448), a simple parent-directory
join fixes trees where includes are consistently parent-relative. It is not safe for mixed
search-dir-relative includes such as RTXPT's first Lighting/LightingTypes.hlsli include; use the
robust version below for the engine fix.

[&](const std::string& IncludeName, size_t /*Start*/, size_t /*End*/)
{
    // Resolve the include relative to the path of the currently scanned file.
    // Without this, a nested include like "LightingConfig.h" inside
    // Lighting/LightingTypes.hlsli loses the "Lighting/" parent path and
    // cannot be found via the flat search dirs.
    std::string ResolvedPath = IncludeName;
    if (ShaderCI.FilePath != nullptr && !FileSystem::IsPathAbsolute(IncludeName.c_str()))
    {
        std::string ParentDir;
        FileSystem::GetPathComponents(ShaderCI.FilePath, &ParentDir, nullptr);
        if (!ParentDir.empty())
        {
            ResolvedPath = FileSystem::SimplifyPath(
                (ParentDir + FileSystem::SlashSymbol + IncludeName).c_str(),
                FileSystem::SlashSymbol);
        }
    }

    if (!Includes.insert(ResolvedPath).second)
        return;

    ShaderCreateInfo IncludeCI{ShaderCI};
    IncludeCI.FilePath     = ResolvedPath.c_str();   // resolved, not bare -> next level's parent dir is correct
    IncludeCI.Source       = nullptr;
    IncludeCI.SourceLength = 0;
    ProcessShaderIncludesImpl(IncludeCI, Includes, IncludeHandler);
}

Because the recursion now stores the resolved path in IncludeCI.FilePath, deeper
parent-relative includes compose correctly: A/B/file.hlsl → "C/inc.h" → A/B/C/inc.h →
"deep.h" → A/B/C/deep.h.

Robust version (matches DXC/FXC include-stack lookup)

DXC/FXC's effective behavior is current-include-relative first, then include search paths. A
search-dir-relative include (e.g. RTXPT's first Lighting/LightingTypes.hlsli) should still
resolve under its original spelling, not under a blindly prepended parent directory. To match
this, probe the stream factory and fall back to the original include spelling when the
current-include-relative path can't be opened:

static std::string ResolveIncludePath(const ShaderCreateInfo& ShaderCI, const std::string& IncludeName)
{
    if (ShaderCI.FilePath == nullptr || FileSystem::IsPathAbsolute(IncludeName.c_str()))
        return IncludeName;

    std::string ParentDir;
    FileSystem::GetPathComponents(ShaderCI.FilePath, &ParentDir, nullptr);
    if (ParentDir.empty())
        return IncludeName;

    const std::string Relative = FileSystem::SimplifyPath(
        (ParentDir + FileSystem::SlashSymbol + IncludeName).c_str(), FileSystem::SlashSymbol);

    // Phase 1: current-include-relative. Phase 2 (fallback): search-dir/original spelling.
    if (ShaderCI.pShaderSourceStreamFactory != nullptr)
    {
        RefCntAutoPtr<IFileStream> pStream;
        ShaderCI.pShaderSourceStreamFactory->CreateInputStream(Relative.c_str(), &pStream);
        if (pStream)
            return Relative;
        return IncludeName; // let the existing error path report the original name if this also fails
    }
    return Relative;
}

Notes / risks

• Hash stability: the values inserted into Includes change from bare names to resolved
  paths, so BY_CONTENT hashes shift once. This is a one-time cache invalidation, not a
  correctness problem.
• Backward compatibility: flat-asset projects (e.g. Tutorial26) still resolve, because the
  robust version falls back to the original include spelling. The minimal version can break mixed
  trees where a nested file is first included by a search-dir-relative path.
• Angle vs quote includes: the scanner does not distinguish <...> from "...". The robust
  (try-relative-then-bare) approach is a superset of correct behavior for both and is the safer
  choice.

---

Key references

What	Location
Scanner that discards the parent dir (root cause)	DiligentCore/Graphics/ShaderTools/src/ShaderToolsCommon.cpp:428-454
Scanner throws on unresolved include	DiligentCore/Graphics/ShaderTools/src/ShaderToolsCommon.cpp:224-245
DXC handler — opens the include path requested by DXC and strips leading .\	DiligentCore/Graphics/ShaderTools/src/DXCompiler.cpp:185-237
FXC handler — opens the include path requested by D3DCompile	DiligentCore/Graphics/GraphicsEngineD3DBase/src/ShaderD3DBase.cpp:57-96
BY_NAME hashing (never calls the scanner)	DiligentCore/Graphics/GraphicsTools/src/RenderStateCacheImpl.cpp (HashShaderCIByFileName)
Path helpers for the fix	DiligentCore/Platforms/Basic/interface/BasicFileSystem.hpp:171-197
RTXPT workaround (BY_NAME)	DiligentSamples/Samples/RTXPT/src/RTXPTSample.cpp (cache create block)

[hzqst]

Btw, DiligentCore\Graphics\ShaderTools\src\GLSLangUtils.cpp :

    // For the "local"-only aspect of a "" include. Should not search in the
    // "system" paths, because on returning a failure, the parser will
    // call includeSystem() to look in the "system" locations.
    virtual IncludeResult* includeLocal(const char* headerName,
                                        const char* includerName,
                                        size_t      inclusionDepth)
    {
        return nullptr;
    }

need to be implemented to match the DXC/FXC include behaviors.

[TheMostDiligent]

Yes, ProcessShaderIncludesImpl is not very robust at the moment

[TheMostDiligent]

There are concerns:

1
The resolver normalizes nested include paths to BasicFileSystem::SlashSymbol. On Windows this turns Nested/Config.hlsl into Nested\Config.hlsl. That works for disk files, but MemoryShaderSourceFactory uses exact string keys and looks them up unchanged. A memory-backed shader package using portable / names would still fail. The new test uses the default disk factory and even expects platform slashes, so it misses this.

2

Before this commit, the scanner ignored include-stack context for everything:

#include "Lighting/Types.hlsl"  // scanner asks factory for Lighting/Types.hlsl

Then inside Lighting/Types.hlsl:

#include <Config.hlsl>          // scanner asks factory for Config.hlsl

After this commit, the scanner parent-resolves everything, because FindIncludes passes only the include string, not whether it came from quotes or angle brackets:

#include <Config.hlsl>          // scanner may ask factory for Lighting/Config.hlsl

Specific example:

shaders/
  Main.hlsl
  Config.hlsl
  Lighting/
    Types.hlsl
    Config.hlsl

Main.hlsl:

#include "Lighting/Types.hlsl"

Lighting/Types.hlsl:

#include <Config.hlsl>

shaders/Config.hlsl:

#define CONFIG_SOURCE 1
shaders/Lighting/Config.hlsl:
#define CONFIG_SOURCE 2

With the default stream factory rooted at shaders:
Before the commit:
Scanner sees #include <Config.hlsl>.
It passes literal Config.hlsl.
Factory loads shaders/Config.hlsl.
After the commit:

Scanner sees the same directive, but does not know it was <...>.
It sees parent file path Lighting/Types.hlsl.
It first tries Lighting/Config.hlsl.
Since shaders/Lighting/Config.hlsl exists, it uses that.

That is the new risk: a system/angle include can silently become parent-relative in the scanner. If the compiler treats <Config.hlsl> as search-path/system include, the scanner hashes/unrolls/enumerates Lighting/Config.hlsl while the compiler uses Config.hlsl.
The clean fix is to make FindIncludes preserve delimiter kind, then only apply parent-relative probing for quoted includes:

const bool IsLocalInclude = *pOpenQuoteOrAngleBracket == '"';
IncludeHandler(IncludeName, IsLocalInclude, Start, End);

Then:

ResolvedPath = IsLocalInclude ?
    ResolveIncludePathForPreprocess(ShaderCI, IncludeName) :
    IncludeName;

A regression test should include both Config.hlsl files and assert that #include <Config.hlsl> inside Nested/Types.hlsl still resolves/reports the root/search-path Config.hlsl, while #include "Config.hlsl" resolves to Nested/Config.hlsl.

[hzqst]

There are concerns:

1 The resolver normalizes nested include paths to BasicFileSystem::SlashSymbol. On Windows this turns Nested/Config.hlsl into Nested\Config.hlsl. That works for disk files, but MemoryShaderSourceFactory uses exact string keys and looks them up unchanged. A memory-backed shader package using portable / names would still fail. The new test uses the default disk factory and even expects platform slashes, so it misses this.

2
A regression test should include both Config.hlsl files and assert that #include <Config.hlsl> inside Nested/Types.hlsl still resolves/reports the root/search-path Config.hlsl, while #include "Config.hlsl" resolves to Nested/Config.hlsl.

I've fixed those concerns in hzqst@3cb15d9 and hzqst@8815d02, but there is an odd failure in Linux build / Linux -> Clang-TSAN-x64, RelWithDebInfo, is that intended?

[TheMostDiligent]

There were some TSAN issues in dev branch, which are fixed now

[hzqst]

Since TSAN issues have been fixed, can we get them merged now?

[TheMostDiligent]

Yes, I take another look.
This requires thorough review because this affects the cache, which may have some unintended consequences.