feat(ci): monitors marker + three-way Better Stack heartbeat split

.github/CLAUDE.md

@@ -712,12 +712,13 @@ External monitoring and alerting for scheduled jobs to detect failures outside G
 
 * `_scheduled-audit.yml` - Audit job monitoring
 * `_scheduled-test.yml` - Test job monitoring (staging & production)
+* `_scheduled-test-hourly.yml` - Three-way split hourly monitoring
 
 **Functionality**:
 
 1. Job runs (audit or test)
 2. Captures exit code (0 = success, non-zero = failure)
-3. Constructs JSON payload with metadata
+3. Constructs JSON payload with metadata via `_betterstack_heartbeat.py`
 4. Sends POST request to BetterStack heartbeat URL with exit code appended
 5. BetterStack tracks heartbeat and alerts on failures or missed beats
 
@@ -745,16 +746,35 @@ External monitoring and alerting for scheduled jobs to detect failures outside G
 
 **URL Format**: `{HEARTBEAT_URL}/{EXIT_CODE}`
 
+**Hourly Three-Way Split** (`_scheduled-test-hourly.yml`):
+
+The hourly workflow runs tests in three independent slices and sends a separate heartbeat per slice so failures are routed to the correct BetterStack monitor:
+
+| Slice | pytest marker filter | Heartbeat secret |
+|-------|---------------------|-----------------|
+| SDK | `not platform_api and not platform_applications` | `BETTERSTACK_HEARTBEAT_URL_{STAGING|PRODUCTION}` |
+| Platform API | `platform_api` | `BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_{STAGING|PRODUCTION}` |
+| Platform Applications | `platform_applications` | `BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_{STAGING|PRODUCTION}` |
+
+**Marker scheme for scheduled tests**:
+
+* `platform_api` — test monitors the Platform API layer (auth, app listing, run listing)
+* `platform_applications` — test monitors a platform application (he-tme, test-app)
+* Neither marker — test is an SDK-layer health check (token management, service wiring)
+
+Tag tests in `tests/` with the boolean markers above. The `monitors` parametrised marker has been removed; use the boolean markers for pytest `-m` expressions.
+
 **Required Secrets**:
 
 * `BETTERSTACK_AUDIT_HEARTBEAT_URL` - For audit jobs
-* `BETTERSTACK_HEARTBEAT_URL_STAGING` - For staging test jobs
-* `BETTERSTACK_HEARTBEAT_URL_PRODUCTION` - For production test jobs
+* `BETTERSTACK_HEARTBEAT_URL_{STAGING|PRODUCTION}` - SDK / general test monitoring
+* `BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_{STAGING|PRODUCTION}` - Platform API monitoring (optional)
+* `BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_{STAGING|PRODUCTION}` - Platform Applications monitoring (optional)
 
 **Behavior**:
 
 * If heartbeat URL is configured: Sends heartbeat regardless of job success/failure
-* If heartbeat URL is NOT configured: Logs warning and continues
+* If heartbeat URL is NOT configured: Logs info and continues (heartbeat steps never fail the job)
 * Exit code passed to URL allows BetterStack to distinguish success (0) from failures
 
 ## Environment Configuration
@@ -805,7 +825,9 @@ External monitoring and alerting for scheduled jobs to detect failures outside G
 * `AIGNOSTICS_REFRESH_TOKEN_{STAGING|PRODUCTION}`
 * `GCP_CREDENTIALS_{STAGING|PRODUCTION}` - Base64 encoded JSON
 * `BETTERSTACK_AUDIT_HEARTBEAT_URL` - Audit monitoring
-* `BETTERSTACK_HEARTBEAT_URL_{STAGING|PRODUCTION}` - Test monitoring
+* `BETTERSTACK_HEARTBEAT_URL_{STAGING|PRODUCTION}` - SDK test monitoring
+* `BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_{STAGING|PRODUCTION}` - Platform API monitoring (optional)
+* `BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_{STAGING|PRODUCTION}` - Platform Applications monitoring (optional)
 * `CODECOV_TOKEN` - Coverage reporting to Codecov
 * `SONAR_TOKEN` - Code quality reporting to SonarCloud
 * `UV_PUBLISH_TOKEN` - PyPI publishing token

.github/workflows/_betterstack_heartbeat.py

@@ -0,0 +1,60 @@
+from __future__ import annotations
+
+import json
+import os
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+from datetime import UTC, datetime
+
+
+def send_betterstack_heartbeat(url: str, exit_code: str, label: str) -> None:
+    if not url:
+        print(f"INFO: No BetterStack {label} heartbeat URL configured, skipped.")
+        return
+
+    if urllib.parse.urlparse(url).scheme not in {"http", "https"}:
+        print(f"WARNING: Refusing non-HTTP BetterStack URL for {label}, skipped.", file=sys.stderr)
+        return
+
+    payload = {
+        "github": {
+            "workflow": os.environ.get("GITHUB_WORKFLOW", ""),
+            "run_url": (
+                f"{os.environ.get('GITHUB_SERVER_URL', '')}"
+                f"/{os.environ.get('GITHUB_REPOSITORY', '')}"
+                f"/actions/runs/{os.environ.get('GITHUB_RUN_ID', '')}"
+            ),
+            "run_id": os.environ.get("GITHUB_RUN_ID", ""),
+            "job": os.environ.get("GITHUB_JOB", ""),
+            "sha": os.environ.get("GITHUB_SHA", ""),
+            "actor": os.environ.get("GITHUB_ACTOR", ""),
+            "repository": os.environ.get("GITHUB_REPOSITORY", ""),
+            "ref": os.environ.get("GITHUB_REF", ""),
+            "event_name": os.environ.get("GITHUB_EVENT_NAME", ""),
+        },
+        "job": {"status": os.environ.get("JOB_STATUS", "")},
+        "timestamp": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
+    }
+
+    data = json.dumps(payload).encode()
+    req = urllib.request.Request(  # noqa: S310
+        f"{url}/{exit_code}",
+        data=data,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    try:
+        urllib.request.urlopen(req, timeout=10)  # noqa: S310
+        print(f"INFO: Sent {label} heartbeat to BetterStack (exit={exit_code})")
+    except urllib.error.URLError as exc:
+        print(f"WARNING: Failed to send {label} heartbeat to BetterStack: {exc}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    send_betterstack_heartbeat(
+        url=os.environ.get("BETTERSTACK_HEARTBEAT_URL", ""),
+        exit_code=os.environ.get("BETTERSTACK_EXIT_CODE", "1"),
+        label=os.environ.get("BETTERSTACK_LABEL", "unknown"),
+    )

.github/workflows/_scheduled-test-hourly.yml

@@ -22,6 +22,10 @@
         required: true
       BETTERSTACK_HEARTBEAT_URL_STAGING:
         required: true
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING:
+        required: false
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_STAGING:
+        required: false
       AIGNOSTICS_CLIENT_ID_DEVICE_PRODUCTION:
         required: true
       AIGNOSTICS_REFRESH_TOKEN_PRODUCTION:
@@ -30,6 +34,10 @@
         required: true
       BETTERSTACK_HEARTBEAT_URL_PRODUCTION:
         required: true
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION:
+        required: false
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_PRODUCTION:
+        required: false
       SENTRY_DSN:
         required: true
 
@@ -90,18 +98,59 @@
           echo "$GCP_CREDENTIALS" | base64 -d > credentials.json
           echo "GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/credentials.json" >> $GITHUB_ENV
 
-      - name: Test / scheduled
+      - name: Test / scheduled / sdk
+        id: test_sdk
+        env:
+          SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
+        shell: bash
+        run: |
+          set +e
+          XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+            -m "(scheduled or scheduled_only) and not platform_api and not platform_applications and not stress_only"
+          echo "exit_code=$?" >> $GITHUB_OUTPUT
+
+      - name: Test / scheduled / platform-api
+        id: test_platform_api
         env:
-          BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}"
           SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
         shell: bash
         run: |
           set +e
-          make test_scheduled
-          EXIT_CODE=$?
-          # Show test execution in GitHub Job summary
+          XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+            -m "(scheduled or scheduled_only) and platform_api and not stress_only"
+          echo "exit_code=$?" >> $GITHUB_OUTPUT
+
+      - name: Test / scheduled / platform-applications
+        id: test_platform_applications
+        env:
+          SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
+        shell: bash
+        run: |
+          set +e
+          XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+            -m "(scheduled or scheduled_only) and platform_applications and not stress_only"
+          echo "exit_code=$?" >> $GITHUB_OUTPUT
+
+      - name: Collect combined exit code and publish summary
+        id: collect
+        if: always()
+        shell: bash
+        run: |
+          SDK_EXIT=${{ steps.test_sdk.outputs.exit_code || '1' }}
+          PLATFORM_API_EXIT=${{ steps.test_platform_api.outputs.exit_code || '1' }}
+          PLATFORM_APPLICATIONS_EXIT=${{ steps.test_platform_applications.outputs.exit_code || '1' }}
+          if [ "$SDK_EXIT" != "0" ] || [ "$PLATFORM_API_EXIT" != "0" ] || [ "$PLATFORM_APPLICATIONS_EXIT" != "0" ]; then
+            COMBINED_EXIT=1
+          else
+            COMBINED_EXIT=0
+          fi
+          echo "sdk_exit=${SDK_EXIT}" >> $GITHUB_OUTPUT
+          echo "platform_api_exit=${PLATFORM_API_EXIT}" >> $GITHUB_OUTPUT
+          echo "platform_applications_exit=${PLATFORM_APPLICATIONS_EXIT}" >> $GITHUB_OUTPUT
+          echo "combined_exit=${COMBINED_EXIT}" >> $GITHUB_OUTPUT
+
           found_files=0
-          for file in reports/pytest_*.md; do
+          for file in reports/pytest_*.md reports/pytest.md; do
             if [ -f "$file" ]; then
               cat "$file" >> $GITHUB_STEP_SUMMARY
               echo "" >> $GITHUB_STEP_SUMMARY
@@ -112,7 +161,6 @@
             echo "# All scheduled tests passed" >> $GITHUB_STEP_SUMMARY
             echo "" >> $GITHUB_STEP_SUMMARY
           fi
-          # Show test coverage in GitHub Job summary
           if [ -f "reports/coverage.md" ]; then
             cat "reports/coverage.md" >> $GITHUB_STEP_SUMMARY
             echo "" >> $GITHUB_STEP_SUMMARY
@@ -121,63 +169,61 @@
             echo "" >> $GITHUB_STEP_SUMMARY
           fi
 
-          # Send heartbeat to Sentry, defining the schedule on the fly
-          SENTRY_EXIT_CODE=$(sentry-cli monitors run -e CI --schedule "0 * * * *" --check-in-margin 30 --max-runtime 1 scheduled-testing-${{ inputs.platform_environment }}-hourly --timezone "Europe/Berlin" -- sh -c "exit $EXIT_CODE")
-
-          # Provide heartbeat to BetterStack for monitoring/alerting if heartbeat url is configured as secret
-          if [ -n "$BETTERSTACK_HEARTBEAT_URL" ]; then
-            BETTERSTACK_METADATA_PAYLOAD=$(jq -n \
-              --arg github_workflow "${{ github.workflow }}" \
-              --arg github_run_url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
-              --arg github_run_id "${{ github.run_id }}" \
-              --arg github_job "${{ github.job }}" \
-              --arg github_sha "${{ github.sha }}" \
-              --arg github_actor "${{ github.actor }}" \
-              --arg github_repository "${{ github.repository }}" \
-              --arg github_ref "${{ github.ref }}" \
-              --arg job_status "${{ job.status }}" \
-              --arg github_event_name "${{ github.event_name }}" \
-              --arg timestamp "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
-              '{
-                github: {
-                  workflow: $github_workflow,
-                  run_url: $github_run_url,
-                  run_id: $github_run_id,
-                  job: $github_job,
-                  sha: $github_sha,
-                  actor: $github_actor,
-                  repository: $github_repository,
-                  ref: $github_ref,
-                  event_name: $github_event_name
-                },
-                job: {
-                  status: $job_status,
-                },
-                timestamp: $timestamp,
-              }'
-            )
-            curl \
-              --fail-with-body \
-              --silent \
-              --request POST \
-              --header "Content-Type: application/json" \
-              --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \
-              "${BETTERSTACK_HEARTBEAT_URL}/${EXIT_CODE}"
-            echo "INFO: Sent heartbeat to betterstack with exit code '${EXIT_CODE}'"
-          else
-            echo "WARNING: No BetterStack heartbeat URL configured, skipped heartbeat notification."
-          fi
-          exit $EXIT_CODE
+      - name: Heartbeat / Sentry
+        if: always()
+        shell: bash
+        env:
+          SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
+          COMBINED_EXIT: ${{ steps.collect.outputs.combined_exit }}
+        run: |
+          sentry-cli monitors run -e CI --schedule "0 * * * *" --check-in-margin 30 --max-runtime 1 \
+            scheduled-testing-${{ inputs.platform_environment }}-hourly \
+            --timezone "Europe/Berlin" -- sh -c "exit ${COMBINED_EXIT}" || true
+
+      - name: Heartbeat / BetterStack / SDK
+        if: always()
+        env:
+          BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}"
+          BETTERSTACK_EXIT_CODE: ${{ steps.collect.outputs.sdk_exit }}
+          BETTERSTACK_LABEL: SDK
+          JOB_STATUS: ${{ job.status }}
+        shell: bash
+        run: python3 .github/workflows/_betterstack_heartbeat.py
+
+      - name: Heartbeat / BetterStack / Platform API
+        if: always()
+        env:
+          BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_PRODUCTION }}"
+          BETTERSTACK_EXIT_CODE: ${{ steps.collect.outputs.platform_api_exit }}
+          BETTERSTACK_LABEL: "Platform API"
+          JOB_STATUS: ${{ job.status }}
+        shell: bash
+        run: python3 .github/workflows/_betterstack_heartbeat.py
+
+      - name: Heartbeat / BetterStack / Platform Applications
+        if: always()
+        env:
+          BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION }}"
+          BETTERSTACK_EXIT_CODE: ${{ steps.collect.outputs.platform_applications_exit }}
+          BETTERSTACK_LABEL: "Platform Applications"
+          JOB_STATUS: ${{ job.status }}
+        shell: bash
+        run: python3 .github/workflows/_betterstack_heartbeat.py
 
       - name: Upload test results
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         if: ${{ always() && (env.GITHUB_WORKFLOW_RUNTIME != 'ACT') }}
         with:
           name: test-results-scheduled
           path: |
-            reports/junit.xml
+            reports/junit_*.xml
             reports/coverage.xml
             reports/coverage.md
             reports/coverage_html
             aignostics.log
           retention-days: 7
+
+      - name: Fail job if any tests failed
+        if: always()
+        shell: bash
+        run: exit ${{ steps.collect.outputs.combined_exit }}

.github/workflows/scheduled-testing-production-hourly.yml

@@ -24,8 +24,12 @@ jobs:
       AIGNOSTICS_REFRESH_TOKEN_STAGING: ${{ secrets.AIGNOSTICS_REFRESH_TOKEN_STAGING }}
       GCP_CREDENTIALS_STAGING: ${{ secrets.GCP_CREDENTIALS_STAGING }}
       BETTERSTACK_HEARTBEAT_URL_STAGING: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_STAGING }}
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING }}
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_STAGING: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_STAGING }}
       AIGNOSTICS_CLIENT_ID_DEVICE_PRODUCTION: ${{ secrets.AIGNOSTICS_CLIENT_ID_DEVICE_PRODUCTION }}
       AIGNOSTICS_REFRESH_TOKEN_PRODUCTION: ${{ secrets.AIGNOSTICS_REFRESH_TOKEN_PRODUCTION }}
       GCP_CREDENTIALS_PRODUCTION: ${{ secrets.GCP_CREDENTIALS_PRODUCTION }}
       BETTERSTACK_HEARTBEAT_URL_PRODUCTION: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}
-      SENTRY_DSN: ${{ secrets.SENTRY_DSN }} # For metrics and heartbeat
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION }}
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_PRODUCTION: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_PRODUCTION }}
+      SENTRY_DSN: ${{ secrets.SENTRY_DSN }}

.github/workflows/scheduled-testing-staging-hourly.yml

@@ -24,8 +24,12 @@ jobs:
       AIGNOSTICS_REFRESH_TOKEN_STAGING: ${{ secrets.AIGNOSTICS_REFRESH_TOKEN_STAGING }}
       GCP_CREDENTIALS_STAGING: ${{ secrets.GCP_CREDENTIALS_STAGING }}
       BETTERSTACK_HEARTBEAT_URL_STAGING: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_STAGING }}
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING }}
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_STAGING: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_STAGING }}
       AIGNOSTICS_CLIENT_ID_DEVICE_PRODUCTION: ${{ secrets.AIGNOSTICS_CLIENT_ID_DEVICE_PRODUCTION }}
       AIGNOSTICS_REFRESH_TOKEN_PRODUCTION: ${{ secrets.AIGNOSTICS_REFRESH_TOKEN_PRODUCTION }}
       GCP_CREDENTIALS_PRODUCTION: ${{ secrets.GCP_CREDENTIALS_PRODUCTION }}
       BETTERSTACK_HEARTBEAT_URL_PRODUCTION: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}
-      SENTRY_DSN: ${{ secrets.SENTRY_DSN }} # For metrics and heartbeat
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION }}
+      BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_PRODUCTION: ${{ secrets.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_PRODUCTION }}
+      SENTRY_DSN: ${{ secrets.SENTRY_DSN }}

pyproject.toml

@@ -383,6 +383,8 @@ markers = [
     "unit: Solitary unit tests - test a layer of a module in isolation with all dependencies mocked, except interaction with shared utils and the systems module. Unit tests must be able to pass offline, i.e. not calls to external services. The timeout should not be bigger than the default 10s, and must be <5 min.",
     "integration: Sociable integration tests - test interactions across architectural layers (e.g. CLI/GUI→Service, Service→Utils) or between modules (e.g. Application→Platform), using real SDK collaborators, real file I/O, real subprocesses, and real Docker containers. Integration test must be able to pass offline, i.e. mock external services (Aignostics Platform API, Auth0, S3/GCS buckets, IDC). The timeout should not be bigger than the default 10s, and must be <5 min.",
     "e2e: End-to-end tests - test complete workflows with real external network services (Aignostics Platform API, cloud storage, IDC, etc). If the test timeout is >= 5 min and < 60 min, additionally mark as `long_running`, if >= 60min mark as 'very_long_running'.",
+    "platform_api: Tag a scheduled test that monitors the Platform API layer (auth, application listing, run listing). Routes Better Stack heartbeats to the Platform API monitor.",
+    "platform_applications: Tag a scheduled test that monitors a platform application (he-tme, test-app). Routes Better Stack heartbeats to the Platform Applications monitor.",
 ]
 md_report = true
 md_report_output = "reports/pytest.md"

tests/aignostics/application/cli_test.py

@@ -164,6 +164,7 @@ def test_cli_application_list_non_verbose(runner: CliRunner, record_property) ->
 
 @pytest.mark.e2e
 @pytest.mark.scheduled
+@pytest.mark.platform_api
 @pytest.mark.timeout(timeout=60)
 def test_cli_application_list_verbose(runner: CliRunner, record_property) -> None:
     """Check application list command runs successfully."""
@@ -801,6 +802,7 @@ def test_cli_run_submit_and_describe_and_cancel_and_download_and_delete(  # noqa
 
 @pytest.mark.e2e
 @pytest.mark.scheduled
+@pytest.mark.platform_api
 @pytest.mark.timeout(timeout=60)
 def test_cli_run_list_limit_10(runner: CliRunner, record_property) -> None:
     """Check run list command runs successfully."""