[Bug]: Track follow-up for DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING behavior in eval runs #207

Bug Description

Why this was required

On main, benchmarks/appworld/appworld_eval.py initialized Langfuse whenever the langfuse package/client was available:

if _LANGFUSE_AVAILABLE and _get_langfuse_client is not None:
    _langfuse = _get_langfuse_client()
    ...

That meant an eval run could try to initialize or read Langfuse state even when the run was not intended to use tracing. In the PR-regression workflow addressed in PR #199 , this caused failures when Langfuse credentials were not configured, for example missing Langfuse host/public/secret key.

PR evals should not require Langfuse. They should still produce evaluation results and reproducibility bundles without tracing enabled.

How PR #199 addresses this

PR #199 adds an explicit tracing gate in benchmarks/appworld/appworld_eval.py:

def _is_langfuse_tracing_enabled() -> bool:
    env_value = os.getenv("DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING")
    if env_value is not None:
        return env_value.strip().lower() in {"1", "true", "yes", "on"}

    return bool(getattr(getattr(settings, "advanced_features", None), "langfuse_tracing", False))

Langfuse initialization is then guarded by that flag:

if _is_langfuse_tracing_enabled() and _LANGFUSE_AVAILABLE and _get_langfuse_client is not None:
    ...

The PR also avoids fetching Langfuse data unless a trace ID was actually created:

if langfuse_trace_id:
    langfuse_handler = LangfuseTraceHandler(langfuse_trace_id)
    langfuse_data = await langfuse_handler.get_langfuse_data()

Finally, the PR-regression wrapper defaults tracing off for /run-pr-eval:

export DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING="${DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING:-false}"

This makes PR eval runs work without Langfuse configuration while preserving the ability to opt into tracing by setting:

DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING=true

Follow-up

We should audit the remaining eval paths and make sure they consistently honor DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING as the source of truth for whether Langfuse tracing should be initialized or skipped.

Potential paths to review:

• benchmarks/appworld/appworld_eval.py
• benchmarks/appworld/appworld_eval_react.py
• benchmarks/appworld/appworld_eval_codeact.py
• SDK/helper flows that call setup_langfuse()
• bundle trace download paths using --fetch-langfuse

Acceptance criteria

• With DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING=false, eval runs do not initialize Langfuse clients/callback handlers.
• Eval runs complete without LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, or LANGFUSE_SECRET_KEY.
• Langfuse trace download is skipped cleanly when tracing is disabled or credentials are absent.
• With DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING=true and valid credentials, existing Langfuse behavior continues to work.

Steps to Reproduce

1. Run an AppWorld evaluation path without Langfuse credentials configured:

   • LANGFUSE_HOST unset
   • LANGFUSE_PUBLIC_KEY unset
   • LANGFUSE_SECRET_KEY unset

2. Ensure Langfuse tracing is not explicitly disabled through the eval tracing config.

3. Trigger an eval flow that initializes Langfuse, for example an AppWorld PR-regression eval run.

4. Observe that the eval path may attempt to initialize or fetch Langfuse data even though the run is not intended to use tracing.

Expected Behavior

When DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING=false, eval runs should not initialize Langfuse clients, callbacks, or trace-fetching helpers.

The eval should complete successfully without requiring:

• LANGFUSE_HOST
• LANGFUSE_PUBLIC_KEY
• LANGFUSE_SECRET_KEY

Bundle creation should also skip Langfuse trace download cleanly when tracing is disabled or credentials are absent.

When DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING=true and valid Langfuse credentials are configured, existing Langfuse tracing behavior should continue to work.

Actual Behavior

Before the PR-regression changes, some eval paths could attempt to initialize Langfuse or fetch Langfuse data whenever the Langfuse package/client was available, even if the eval run was not intended to use tracing.

In PR-regression runs without Langfuse credentials configured, this caused failures related to missing Langfuse host/public/secret key configuration.

Environment

• OS: Ubuntu 22.04 / self-hosted Linux runner
• Python version: 3.12
• CUGA version: PR branch from cuga-project/cuga-eval PR feat: PR Regression Testing Workflow  #199
• Browser: N/A
• Node.js version: N/A

Error Logs / Screenshots

No response

Configuration

No response

Additional Context

• Repository: cuga-project/cuga-eval
• Related PR: feat: PR Regression Testing Workflow  #199
• Workflow/use case: PR-regression evaluation triggered by /run-pr-eval
• Benchmark path: AppWorld eval flows
• Relevant config variable: DYNACONF_ADVANCED_FEATURES__LANGFUSE_TRACING
• Langfuse credentials may be unset:
  • LANGFUSE_HOST
  • LANGFUSE_PUBLIC_KEY
  • LANGFUSE_SECRET_KEY

Checklist

• I have searched existing issues to ensure this is not a duplicate
• I have provided all the required information above
• I am using the latest version of CUGA