feat(ci): monitors marker + three-way Better Stack heartbeat split#641
Draft
ari-nz wants to merge 1 commit into
Draft
feat(ci): monitors marker + three-way Better Stack heartbeat split#641ari-nz wants to merge 1 commit into
ari-nz wants to merge 1 commit into
Conversation
ari-nz
requested review from
a team and
helmut-hoffer-von-ankershoffen
as code owners
There was a problem hiding this comment.
Pull request overview
This PR introduces a monitors(...) pytest marker intended to tag scheduled E2E tests by the Better Stack monitor they should feed, and updates the hourly scheduled CI workflow to split test execution and emit separate heartbeats (SDK vs. application-specific).
Changes:
- Added
@pytest.mark.monitors("he-tme" | "test-app")to selected platform E2E scheduled tests. - Registered the new
monitorsmarker inpyproject.tomlpytest configuration. - Split the hourly scheduled workflow into multiple pytest invocations and added a dedicated Better Stack heartbeat URL secret for HE-TME (propagated via the staging/production wrapper workflows).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/aignostics/platform/e2e_test.py |
Tags scheduled E2E tests with monitors(...) to support routing results/heartbeats by monitored system. |
pyproject.toml |
Registers the new pytest marker so --strict-markers runs remain valid. |
.github/workflows/_scheduled-test-hourly.yml |
Splits scheduled tests into separate runs and sends separate Better Stack heartbeats (SDK + HE-TME), plus combined status to Sentry. |
.github/workflows/scheduled-testing-staging-hourly.yml |
Passes through the new HE-TME Better Stack heartbeat secret to the reusable workflow. |
.github/workflows/scheduled-testing-production-hourly.yml |
Passes through the new HE-TME Better Stack heartbeat secret to the reusable workflow. |
Comment on lines
+123
to
+128
| # set +e so a test failure does not abort the step — we capture the exit code | ||
| # manually and send it to Better Stack regardless of outcome. | ||
| set +e | ||
| make test_scheduled | ||
| EXIT_CODE=$? | ||
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors and not stress_only" \ | ||
| --junit-xml=reports/junit_he_tme.xml |
Comment on lines
+117
to
+129
| - name: Test / scheduled / he-tme | ||
| id: test_he_tme | ||
| env: | ||
| BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}" | ||
| SENTRY_DSN: ${{ secrets.SENTRY_DSN }} | ||
| shell: bash | ||
| run: | | ||
| # set +e so a test failure does not abort the step — we capture the exit code | ||
| # manually and send it to Better Stack regardless of outcome. | ||
| set +e | ||
| make test_scheduled | ||
| EXIT_CODE=$? | ||
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors and not stress_only" \ | ||
| --junit-xml=reports/junit_he_tme.xml | ||
| echo "exit_code=$?" >> $GITHUB_OUTPUT |
| "unit: Solitary unit tests - test a layer of a module in isolation with all dependencies mocked, except interaction with shared utils and the systems module. Unit tests must be able to pass offline, i.e. not calls to external services. The timeout should not be bigger than the default 10s, and must be <5 min.", | ||
| "integration: Sociable integration tests - test interactions across architectural layers (e.g. CLI/GUI→Service, Service→Utils) or between modules (e.g. Application→Platform), using real SDK collaborators, real file I/O, real subprocesses, and real Docker containers. Integration test must be able to pass offline, i.e. mock external services (Aignostics Platform API, Auth0, S3/GCS buckets, IDC). The timeout should not be bigger than the default 10s, and must be <5 min.", | ||
| "e2e: End-to-end tests - test complete workflows with real external network services (Aignostics Platform API, cloud storage, IDC, etc). If the test timeout is >= 5 min and < 60 min, additionally mark as `long_running`, if >= 60min mark as 'very_long_running'.", | ||
| "monitors: Tag a scheduled test with the application it monitors, e.g. @pytest.mark.monitors('he-tme'). Tests without this marker are considered SDK-layer health checks. Used to route Better Stack heartbeats to the correct monitor.", |
Comment on lines
+288
to
+289
| reports/junit_sdk.xml | ||
| reports/junit_he_tme.xml |
ari-nz
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. |
ari-nz
force-pushed
the
feat/monitors-marker-split
branch
from
129c9aa to
0683245
Compare
Comment on lines
+139
to
+150
| - name: Test / scheduled / he-tme | ||
| id: test_he_tme | ||
| env: | ||
| SENTRY_DSN: ${{ secrets.SENTRY_DSN }} | ||
| shell: bash | ||
| run: | | ||
| # set +e so a test failure does not abort the step — we capture the exit code | ||
| # manually and send it to Better Stack regardless of outcome. | ||
| set +e | ||
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors and not monitors_platform_api and not stress_only" \ | ||
| --junit-xml=reports/junit_he_tme.xml |
| --request POST \ | ||
| --header "Content-Type: application/json" \ | ||
| --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \ | ||
| "${BETTERSTACK_HEARTBEAT_URL}/${SDK_EXIT}" |
Comment on lines
+297
to
+304
| curl \ | ||
| --fail-with-body \ | ||
| --silent \ | ||
| --request POST \ | ||
| --header "Content-Type: application/json" \ | ||
| --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \ | ||
| "${BETTERSTACK_HEARTBEAT_URL_PLATFORM_API}/${PLATFORM_API_EXIT}" | ||
| echo "INFO: Sent Platform API heartbeat to BetterStack with exit code '${PLATFORM_API_EXIT}'" |
Comment on lines
+347
to
+354
| curl \ | ||
| --fail-with-body \ | ||
| --silent \ | ||
| --request POST \ | ||
| --header "Content-Type: application/json" \ | ||
| --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \ | ||
| "${BETTERSTACK_HEARTBEAT_URL_HE_TME}/${HE_TME_EXIT}" | ||
| echo "INFO: Sent HE-TME heartbeat to BetterStack with exit code '${HE_TME_EXIT}'" |
| retention-days: 7 | ||
|
|
||
| - name: Fail job if any tests failed | ||
| shell: bash |
Comment on lines
+120
to
+123
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and not monitors and not monitors_platform_api and not stress_only" \ | ||
| --junit-xml=reports/junit_sdk.xml | ||
| echo "exit_code=$?" >> $GITHUB_OUTPUT |
Comment on lines
+134
to
+137
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors_platform_api and not stress_only" \ | ||
| --junit-xml=reports/junit_platform_api.xml | ||
| echo "exit_code=$?" >> $GITHUB_OUTPUT |
Comment on lines
+148
to
+151
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors and not monitors_platform_api and not stress_only" \ | ||
| --junit-xml=reports/junit_he_tme.xml | ||
| echo "exit_code=$?" >> $GITHUB_OUTPUT |
Comment on lines
+362
to
+364
| reports/junit_sdk.xml | ||
| reports/junit_platform_api.xml | ||
| reports/junit_he_tme.xml |
ari-nz
force-pushed
the
feat/monitors-marker-split
branch
from
eaa4b76 to
15c7e5d
Compare
ari-nz
force-pushed
the
feat/monitors-marker-split
branch
from
15c7e5d to
0fdfed4
Compare
ari-nz
force-pushed
the
feat/monitors-marker-split
branch
from
0fdfed4 to
39f8b21
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
monitors(\"name\")marker with two boolean markers that pytest `-m` expressions can filter on directly:platform_api— test monitors Platform API layer (auth, listing, connectivity)platform_applications— test monitors a platform application (he-tme, test-app)Action required
Rename GitHub secret (repository settings → Secrets):
BETTERSTACK_HEARTBEAT_URL_HE_TME_STAGING→BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGINGBETTERSTACK_HEARTBEAT_URL_HE_TME_PRODUCTION→BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTIONUntil the secret is renamed the Platform Applications heartbeat will silently skip (step logs a warning).
Test plan