[kassvl]

Closes #5192.

Unreal's Crash Report Client does not write a minidump on Linux, so these events arrive without a stack trace. The crash context (CrashContext.runtime-xml) still has the crashing call stack. With the new project feature projects:relay-unreal-crash-context-stacktrace, Relay builds an exception from it when the report has no minidump or Apple crash report and the event has no exception yet.

• Frames come from the legacy CallStack: module, plus function and file:line when the build had symbols. Symbols with an offset like __libc_start_main(+0x8a) are the nearest export and are left out as function names, so they don't merge unrelated crashes.
• Instruction addresses come from PCallStack only when both stacks agree on every frame's module, and on the offset where the legacy frame has one. Engines older than 4.26 write 32-bit image bases, so they get no addresses. Frames are capped at 256.
• As discussed in the issue: platform: native and mechanism.synthetic: true. level keeps the existing assert/ensure mapping from merge_unreal_context, and since ref(unreal): Migrate unreal processing to its own processor #6578 every Unreal event without a level falls back to fatal, so this PR adds no level handling of its own. That fallback also applies to stalls and VerseRuntimeError, which are reported while the game keeps running (handled: true). Happy to map those to warning if you prefer.
• Rebased onto ref(unreal): Migrate unreal processing to its own processor #6578: the flag is read in the new UnrealErrorIntegrationProcessor, so the stack trace is added by whichever Relay expands the report.

Effect when the flag is on: affected events become error events, so existing Linux crash issues regroup once. Because the exception is synthetic, Sentry ignores its type for grouping and prefers the crashing function as title. For builds without symbols, like the sample in the issue, every frame is UnknownFunction, so grouping stays on the error message and the title becomes Ensure: <message>. Builds with symbols get the crashing function as title and group by stack. Inbound filters and PII rules on the error message now also see the exception value. Nothing is symbolicated: the XML has no debug IDs, so these native frames have no debug images to resolve against.

To turn it on, the flag still needs to be registered on the Sentry side.

Tests: unit tests for parsing, combining both stacks, crash types, the level (including a level sent by the client), and the disabled flag. test_unreal_stacktrace_without_minidump_with_processing in test_unreal.py runs with and without the feature against a processing Relay with Kafka, Redis and objectstore; with the feature it fails on master and passes here. I ran the Unreal unit and integration tests locally, plus clippy and fmt. Prepared with AI assistance.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

[Dav1dde]

@tustanivsky we now support retaining SDK exceptions in error events accompanying a minidump crash. Do you think for unreal crashes we should parse the stack trace even if there is a minidump or only fall back to parsing if there is no minidump?

[Dav1dde]

I've yet to go through the code more deeply, just some quick things I found comparing the parser with the unreal source.

There is also one case we still need to figure out, unsymbolicated stack traces can only be symbolicated if we have a source to fill out the event's debug images. Either we need the SDK to fill this in or somehow extract it from the crash (if that's possible).

[Dav1dde]

This seems to not be unconditionally true, from the Unreal code:

			// Program counters in the backtrace point to the location from where the execution will be resumed (in all frames except the one where we crashed),
			// which results in callstack pointing to the next lines in code. In order to determine the source line where the actual call happened, we need to go
			// back to the line that had the "call" instruction. Since x86(-64) instructions vary in length, we cannot do it reliably without disassembling,
			// just go back one byte - even if it's not the actual address of the call site.
			int OffsetToCallsite = CurrentCallDepth > 0 ? 1 : 0;

[kassvl]

Thanks, fixed in d5b4a25. In the sample from #5192 the first frame is adjusted as well (0xb3a7401 in CallStack vs b3a7402 in PCallStack), probably because the crash handler frames are skipped (NumMinidumpFramesToIgnore is 7), so the first frame written is not at depth 0. Since the crash context does not say which case it is, the first frame now matches with or without the adjustment, and all later frames still need it.

[Dav1dde]

I don't think we can unconditionally assume the module name does not contain a whitespace.

[kassvl]

Right, fixed in d5b4a25. A frame is now split at the + between image base and offset: the image base is the last word before it and everything before that is the module, so module names can contain whitespace (or a +, like libstdc++.so).

[Dav1dde]

I think it's possible that there are frames without module.

From the source code:

	//
	// Callstack lines should be written in this standard format
	//
	//	0xaddress module!func [file]
	// 
	// E.g. 0x045C8D01 OrionClient.self!UEngine::PerformError() [D:\Epic\Orion\Engine\Source\Runtime\Engine\Private\UnrealEngine.cpp:6481]
	//
	// Module may be omitted, everything else should be present, or substituted with a string that conforms to the expected type
	//
	// E.g 0x00000000 UnknownFunction []
	//
	// 

[kassvl]

Fixed in d5b4a25. A line without ! is now read as a function when it has arguments or a [file] part, like UnknownFunction [], and still as a module otherwise. When the stacks line up, such a frame takes its module from PCallStack. Since it cannot be checked by its module, it only counts as matching if its offset agrees, or if it has no function or file that could end up on the wrong address.

[tustanivsky]

Do you think for unreal crashes we should parse the stack trace even if there is a minidump or only fall back to parsing if there is no minidump?

I'd keep it as a fallback only. The callstack in the crash context describes the same crash as the minidump so it would basically be a duplicate.

[kassvl]

Thanks for the review! I pushed fixes for the three comments (d5b4a25) and merged master.

On the fallback question: that's how it works now, the stack trace is only added when there is no minidump or Apple crash report.

On debug images: in the sample from the issue the crash context has no build IDs (Modules is empty), so the only image information is the module name and image base from PCallStack. I don't see a way to get debug images that symbolication can use out of the crash itself, so I think that part needs the SDK to send them. Until then the frames have the module and the address, plus function and file when the build had symbols.

[Dav1dde]

I was talking on Slack with @tustanivsky what we can do, and he added a PR which would include the images information in the __sentry metadata (getsentry/sentry-unreal#1673).

I am not sure yet if we want to gate the callstack extraction also behind the presence of the debug images if we don't have an already symbolicated stack trace, as otherwise it may quite heavily mess with grouping.

[kassvl]

Thanks, and thanks @tustanivsky for the SDK PR!

I tried the #5192 sample with Sentry master and Symbolicator. Without symbols, grouping doesn't change: same hash, same issue, only the title becomes Ensure: <message>. But since the event now has frames, it goes to Symbolicator for nothing (every frame comes back as unknown_image), and new issues become candidates for Seer similarity with only Ensure: <message> to compare. With function or file names in the frames it's different: the hash moves from the message to the frames, so those events regroup once.

So a gate makes sense to me: only add the stack trace if a frame has a function or file name, or the event has debug_meta.images. Otherwise the event stays exactly as it is today, and the one-time regroup only happens where the frames are actually useful.

One thing for later: in the sample the first frame's address is 1 lower in CallStack than in PCallStack, so Unreal treats it as a return address. In public Linux crash logs I see the same for asserts and fatal errors, while SIGSEGV/SIGABRT keep the first frame as is. So once images arrive, events like this one will need instruction_addr_adjustment: all. I'd leave that for a follow-up.

I'll also fix the description: "regroup once" only applies when the frames have names, and the frames do go to Symbolicator. Should I add the gate with a test here?