Update towards template repo

[Jun Aishima (JunAishima)]

Update this repo to match what is already in https://github.com/NSLS2/nsls2-workflows-template
Will obsolete the following PRs, so will close them once this PR is approved:

• chore: upgrade pixi.lock files to v7 format #12
• Dotenv dry run fromuri #7

[Copilot]

Copilot review overview

🟡 Changes recommended

Three unresolved moderate issues affect dry-run behavior, credential lifetime, and failure notification coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3

Open (3)

• TILED_API_KEY remains exposed in the worker environment · New
• dry_run still reads and reports all Tiled streams · New
• Early Tiled failures bypass failure notifications · New

What changed in this PR

Updates TST workflows to align with the template repository and use dotenv-based Tiled authentication.

Changes:

• Replaces legacy Tiled/Prefect Secret access with URI-based clients.
• Adds Slack notifications and workflow parameters.
• Updates deployment configuration, mounts, and dependencies.

Three unresolved moderate issues remain: dry_run still reads data, load_dotenv leaves credentials in the environment, and early Tiled failures bypass notifications.

File	Summary
utils.py	Removes the obsolete Tiled client helper.
test_extra_client.py	Uses the shared Tiled run lookup.
prefect.yaml	Updates deployment version and volume mounts.
pixi.toml	Adds python-dotenv.
end_of_run_workflow.py	Adds notifications and workflow orchestration.
data_validation.py	Loads credentials and validates Tiled streams.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved credential handling, dry-run behavior, and preflight failure reporting issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 3

Open (4)

• api_key exposed through deployed flow parameters · New
• Early Tiled failures bypass failure notifications
• dry_run still reads and reports all Tiled streams
• TILED_API_KEY remains exposed in the worker environment

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved credential-handling, notification-error, preflight-reporting, and deployment-mount issues remain.

Review effort: Lite
Findings: 1 · 2

Open (3)

• api_key exposed through deployed flow parameters
• Early Tiled failures bypass failure notifications
• TILED_API_KEY remains exposed in the worker environment

Resolved since last review (1)

• dry_run still reads and reports all Tiled streams

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical mount security and moderate failure-notification issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 1 · 1

Open (4)

• Mount only the secret file read-only · New
• api_key exposed through deployed flow parameters
• Early Tiled failures bypass failure notifications
• Remove unused import · New

Resolved since last review (1)

• TILED_API_KEY remains exposed in the worker environment

[Copilot]

Copilot review overview

🔵 Needs a closer look

Move the Tiled metadata lookup into protected error handling so failures are reported consistently.

Review effort: Lite
Findings: 2 · 1

Open (3)

• Mount only the secret file read-only
• api_key exposed through deployed flow parameters
• Early Tiled failures bypass failure notifications

Resolved since last review (1)

• Remove unused import

[Copilot]

Copilot review overview

🔵 Needs a closer look

Tiled lookup failures can bypass the guarded Slack failure notifications.

Review effort: Lite
Findings: 1

Open (1)

• Early Tiled failures bypass failure notifications

Resolved since last review (2)

• Mount only the secret file read-only
• api_key exposed through deployed flow parameters

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved moderate issues in end_of_run_workflow.py must be addressed before approval.

Review effort: Lite
Findings: 1

Open (1)

• Early Tiled failures bypass failure notifications

Previously missed (1)

In code that hasn't changed since last review

Prevent success notification failures from marking runs failed

end_of_run_workflow.py:32

Because this success notification is inside the try, a Slack webhook failure after func has completed successfully enters the exception handler, sends failure notifications, and re-raises. That can mark a successful validation run as failed solely because monitoring is unavailable; isolate notification errors from the wrapped workflow result.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The API key is exposed through Prefect flow/task parameters and should be removed from those interfaces.

Review effort: Lite
Findings: 1

Open (1)

• Early Tiled failures bypass failure notifications

[Copilot]

Copilot review overview

🟡 Changes recommended

Tiled API keys remain exposed through Prefect task and flow parameters.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Prevent api_key exposure through Prefect task inputs · New

Resolved since last review (1)

• Early Tiled failures bypass failure notifications

[Jun Aishima (JunAishima)]

fixes #13

[Abby Giles (AbbyGi)]

Looks good. One comment removal

[Copilot]

Copilot review overview

🔵 Needs a closer look

Authentication, workflow orchestration, and deployment changes warrant final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Prevent api_key exposure through Prefect task inputs

[Jun Aishima (JunAishima)]

just tested this with a UID that was successful 5 months ago, so merging.