Skip to content

Add reprioritize API: single job + bulk by filter #307

@hardbyte

Description

@hardbyte

Context

Awa has no first-class way to change the priority of an existing job. Grep confirms: no set_priority / reprioritize / change_priority in awa-model::admin, awa-worker::client, awa-ui::handlers, awa-cli, or awa-python. InsertOpts.priority sets it at enqueue time; age_waiting_priorities decrements it automatically on canonical storage (queue storage uses claim-time aging instead); the UI surfaces priority + original_priority as read-only fields.

Today the only paths are:

Path Problem
UPDATE awa.jobs_hot SET priority = … Canonical only; fails for queue-storage rows in ready_entries/done_entries; bypasses metadata bookkeeping (_awa_original_priority).
Cancel + re-enqueue Loses attempt count + history; collides with unique_key.
Wait for aging Monotonic decrement only (toward higher importance); canonical only; time-bounded.
UPDATE {schema}.ready_entries SET priority = X Also needs to bump lane_seq into the new priority's claim-head lane (since (queue, priority, enqueue_shard) partitions claim sequences); not safe without a transactional helper.

This is an operational gap for the 0.6 stable release. Incidents where one queue's pending work suddenly needs to jump the line (e.g. SLO breach, customer escalation, partner-facing job blocking a release) currently have no clean answer.

Proposed work

Two admin surfaces matching the existing bulk-retry / bulk-cancel shape:

  • Single: POST /api/jobs/:id/priority { priority: i16 } → updates one job
  • Bulk: POST /api/jobs/bulk-priority { kind?, queue?, ids?, priority } → updates the filter selection

Mirror in awa_model::admin, awa-worker::Client, awa-cli (awa job priority <id> --to N, awa job bulk-priority --queue X --to N), and awa-python.

Queue-storage implementation note

The write path is non-trivial. For an available row in queue storage, changing priority means moving the row between (queue, priority, enqueue_shard) claim-head lanes — the row needs a fresh lane_seq from the destination lane's queue_enqueue_heads.next_seq, and the source lane's claim_seq must advance past the old lane_seq so the abandoned slot doesn't block the head. Has to be one transaction; the change must be invisible to the claimer between the source-lane advance and the destination-lane insert.

For scheduled (deferred) rows in deferred_jobs, this is a simple UPDATE.

For running / waiting_external / terminal rows: define what reprioritization means. Options:

  • (a) Reject — only available and scheduled are reprioritizable
  • (b) For running, update the column so the next attempt picks up the new priority but the current attempt finishes at the old one
  • (c) For running, also cancel-and-re-enqueue if the operator wants the change to take effect immediately

(a) is the simplest, most predictable, and matches operator intuition ("a job that's already running can't change lanes mid-flight"). Recommend (a) unless someone has a concrete use case for (b)/(c).

Acceptance criteria

  • Single and bulk admin endpoints land with the same auth/permission shape as bulk-retry/bulk-cancel.
  • Queue-storage moves are transactional; concurrent claimer can never see a "missing" row or claim a row at the wrong priority.
  • Canonical storage path also updated (so the feature works across the upgrade window).
  • Metadata bookkeeping: _awa_original_priority is written exactly once (on the first reprioritize OR on the first aging pass, whichever comes first), so subsequent reprioritize calls don't overwrite the originally-enqueued value.
  • Rust + Python client APIs.
  • CLI: awa job priority <id> --to N + awa job bulk-priority --kind X --queue Y --to N (with the same --all guard the existing bulk DLQ commands use).
  • UI: priority field becomes editable on the job detail page; bulk-priority button on the jobs list when one or more rows are selected.
  • Tests: unit + integration covering the queue-storage transactional move, the canonical path, the lane-seq advance invariant, and a chaos-suite test for concurrent claimer + reprioritize on the same row.
  • Docs: brief note in docs/configuration.md (priority semantics) and an admin-API doc entry.

Out of scope

  • Reprioritize for terminal rows. Not meaningful.
  • Per-priority queue tracking (already part of the queue-storage shape).
  • Job-kind-level "default priority" overrides — separate feature.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew functionalityoperationalOperational tooling and configuration

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions