feat: add --debug flag and DATAPILOT_DEBUG env var for verbose logging

[suryaiyer95]

Problem

A customer hit Error in uploading the manifest. and had no way to learn anything more — they
asked whether there was "a verbose/debug flag, or logs that I could use to get more character on
the error."

There wasn't. APIClient already records the HTTP status and the API's error body, but it does so
via logger.debug, and dbt/cli/cli.py pinned the root logger at INFO at import time. Every
DEBUG record was thrown away, and no flag or environment variable could change that.

The backend was answering perfectly clearly the whole time. Reproduced against
api.myaltimate.com:

GET /dbt/v1/signed_url?dbt_core_integration_id=2&dbt_core_integration_environment_type=DEV...
-> 400 {"detail": "dbt_core_integration with id:2 and env:DEV not found"}

The CLI collapsed that into eight words.

Change

Verbose output is now opt-in two ways, both resolving to the same click parameter:

datapilot dbt onboard --debug ...

# or, for CI/CD pipelines where the command line is generated:
export DATAPILOT_DEBUG=1
datapilot dbt onboard ...

Before / after on the same failing command:

Error in uploading the manifest.

DEBUG:APIClient:Sending GET request for tenant elastic at url: https://api.myaltimate.com/dbt/v3/validate-credentials
DEBUG:APIClient:Received GET response with status: 200
DEBUG:APIClient:Sending GET request for tenant elastic at url: https://api.myaltimate.com/dbt/v1/validate-permissions
DEBUG:APIClient:Received GET response with status: 200
DEBUG:APIClient:Sending GET request for tenant elastic at url: https://api.myaltimate.com/dbt/v1/signed_url
DEBUG:APIClient:Request params: {'dbt_core_integration_id': '2', 'dbt_core_integration_environment_type': 'DEV', 'file_type': 'manifest'}
DEBUG:APIClient:HTTP Error: {'detail': 'dbt_core_integration with id:2 and env:DEV not found'} - Status code: 400
DEBUG:APIClient:Error getting signed URL.
Error in uploading the manifest.

Implementation notes

• New datapilot/utils/logging_utils.py — configure_logging(), is_debug_enabled(), redact_url().
• Replaced the import-time logging.basicConfig(level=logging.INFO) in dbt/cli/cli.py and
  mcp.py with configure_logging(), so DATAPILOT_DEBUG is honoured even on paths that never
  reach a command callback.
• Debug is sticky — an import-time call at INFO cannot undo a --debug flag.
• configure_logging() installs a handler only when the root logger has none, and otherwise just
  sets the level. An earlier draft used basicConfig(force=True); a test caught it wiping pytest's
  caplog handler, which would equally have clobbered handlers in any app embedding datapilot.

Credential redaction

Debug output exists to be pasted into support tickets, so it must be safe to share. Enabling it
previously printed the presigned S3 URL in full — AWSAccessKeyId=...&Signature=... — four times
per run, from our own put() log and from urllib3, which logs each request line verbatim.

• redact_url() strips the query string in both places, keeping the object path
  (.../manifest.json?<redacted>).
• urllib3 is held at INFO whenever we go to DEBUG.
• To keep the diagnostic value that would otherwise be lost, APIClient.get() now logs its request
  params — the integration id, environment and file type, with no secrets. This is the line that
  actually pinpoints the failure above.

Verified on a successful upload against production: AWSAccessKeyId 0 occurrences, Signature=
0, API token 0.

Testing

• 137 passed (was 107 on main). 30 new tests across tests/utils/test_logging_utils.py,
  tests/cli/test_debug_option.py and tests/clients/altimate/test_client_redaction.py, covering
  env-var parsing, flag/env precedence, stickiness, CLI-level log-level assertions, and
  credential-redaction regressions.
• ruff and black clean on all changed files.
• Manually verified all four paths against api.myaltimate.com with the built package installed
  into a clean venv: default (unchanged output), DATAPILOT_DEBUG=1, --debug, and
  DATAPILOT_DEBUG=0 (stays quiet).

Not in this PR

No CHANGELOG.rst entry or version bump, matching the last three releases where those land in a
separate chore: bump version commit.

Two related issues found while reproducing this, both worth separate PRs:

1. Manifests are rejected outright when dbt-adapters >= 1.24.0 is installed.
   dbt-adapters 1.24.0 changed its bundled function materialization to
   supported_languages=['sql', 'python', 'javascript'], but the vendored SupportedLanguage
   enum only allows python/sql, so load_manifest fails before any upload happens.
   This is not gated on the dbt-core version: dbt-core 1.9.10 parses fine with
   dbt-adapters 1.23.0 and fails with 1.24.5. Verified boundary by inspecting every
   dbt-adapters wheel from 1.16.0 to 1.24.5.
   Worth noting the enum is not a deliberate restriction — it is generated from dbt's own
   published schema, and https://schemas.getdbt.com/dbt/manifest/v12.json still declares
   ['python', 'sql']. dbt now emits javascript in a manifest that still identifies itself
   as v12, so upstream has drifted from its own published schema.
2. The disabled strip-and-retry fallback added in fix: resilient manifest parsing for disabled field and unknown versions #95 can never succeed.
   _strip_unused_fields removes disabled, but ManifestV12.disabled is declared
   Field(...) — required. Feeding a known-good manifest through _strip_unused_fields
   fails with disabled Field required, so the fallback turns every parse failure into a
   misleading two-error message that hides the real cause.

🤖 Generated with Claude Code

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (11 files)

• README.md
• src/datapilot/cli/decorators.py
• src/datapilot/clients/altimate/client.py
• src/datapilot/clients/altimate/utils.py
• src/datapilot/core/mcp_utils/mcp.py
• src/datapilot/core/platforms/dbt/cli/cli.py
• src/datapilot/utils/logging_utils.py
• tests/cli/__init__.py
• tests/cli/test_debug_option.py
• tests/clients/altimate/test_client_redaction.py
• tests/utils/test_logging_utils.py

Notes: Credential redaction is complete at both signed-URL log sites (client.py:75 for put() and utils.py:93 for onboard_file), and the actual HTTP requests correctly use the unredacted URL. The sticky debug logic (_debug_enabled), the configure_logging handler-installation guard, and the urllib3 INFO clamp are sound. The new debug_option decorator mirrors the existing auth_options pattern and correctly pops debug before delegating. No redundancy or simplification opportunities worth flagging.

---

_{Reviewed by glm-5.2 · Input: 38.5K · Output: 11K · Cached: 177.2K}