Context
Awa has no first-class way to change the priority of an existing job. Grep confirms: no set_priority / reprioritize / change_priority in awa-model::admin, awa-worker::client, awa-ui::handlers, awa-cli, or awa-python. InsertOpts.priority sets it at enqueue time; age_waiting_priorities decrements it automatically on canonical storage (queue storage uses claim-time aging instead); the UI surfaces priority + original_priority as read-only fields.
Today the only paths are:
| Path |
Problem |
UPDATE awa.jobs_hot SET priority = … |
Canonical only; fails for queue-storage rows in ready_entries/done_entries; bypasses metadata bookkeeping (_awa_original_priority). |
| Cancel + re-enqueue |
Loses attempt count + history; collides with unique_key. |
| Wait for aging |
Monotonic decrement only (toward higher importance); canonical only; time-bounded. |
UPDATE {schema}.ready_entries SET priority = X |
Also needs to bump lane_seq into the new priority's claim-head lane (since (queue, priority, enqueue_shard) partitions claim sequences); not safe without a transactional helper. |
This is an operational gap for the 0.6 stable release. Incidents where one queue's pending work suddenly needs to jump the line (e.g. SLO breach, customer escalation, partner-facing job blocking a release) currently have no clean answer.
Proposed work
Two admin surfaces matching the existing bulk-retry / bulk-cancel shape:
- Single:
POST /api/jobs/:id/priority { priority: i16 } → updates one job
- Bulk:
POST /api/jobs/bulk-priority { kind?, queue?, ids?, priority } → updates the filter selection
Mirror in awa_model::admin, awa-worker::Client, awa-cli (awa job priority <id> --to N, awa job bulk-priority --queue X --to N), and awa-python.
Queue-storage implementation note
The write path is non-trivial. For an available row in queue storage, changing priority means moving the row between (queue, priority, enqueue_shard) claim-head lanes — the row needs a fresh lane_seq from the destination lane's queue_enqueue_heads.next_seq, and the source lane's claim_seq must advance past the old lane_seq so the abandoned slot doesn't block the head. Has to be one transaction; the change must be invisible to the claimer between the source-lane advance and the destination-lane insert.
For scheduled (deferred) rows in deferred_jobs, this is a simple UPDATE.
For running / waiting_external / terminal rows: define what reprioritization means. Options:
- (a) Reject — only
available and scheduled are reprioritizable
- (b) For
running, update the column so the next attempt picks up the new priority but the current attempt finishes at the old one
- (c) For
running, also cancel-and-re-enqueue if the operator wants the change to take effect immediately
(a) is the simplest, most predictable, and matches operator intuition ("a job that's already running can't change lanes mid-flight"). Recommend (a) unless someone has a concrete use case for (b)/(c).
Acceptance criteria
- Single and bulk admin endpoints land with the same auth/permission shape as
bulk-retry/bulk-cancel.
- Queue-storage moves are transactional; concurrent claimer can never see a "missing" row or claim a row at the wrong priority.
- Canonical storage path also updated (so the feature works across the upgrade window).
- Metadata bookkeeping:
_awa_original_priority is written exactly once (on the first reprioritize OR on the first aging pass, whichever comes first), so subsequent reprioritize calls don't overwrite the originally-enqueued value.
- Rust + Python client APIs.
- CLI:
awa job priority <id> --to N + awa job bulk-priority --kind X --queue Y --to N (with the same --all guard the existing bulk DLQ commands use).
- UI: priority field becomes editable on the job detail page; bulk-priority button on the jobs list when one or more rows are selected.
- Tests: unit + integration covering the queue-storage transactional move, the canonical path, the lane-seq advance invariant, and a chaos-suite test for concurrent claimer + reprioritize on the same row.
- Docs: brief note in
docs/configuration.md (priority semantics) and an admin-API doc entry.
Out of scope
- Reprioritize for terminal rows. Not meaningful.
- Per-priority queue tracking (already part of the queue-storage shape).
- Job-kind-level "default priority" overrides — separate feature.
Related
Context
Awa has no first-class way to change the priority of an existing job. Grep confirms: no
set_priority/reprioritize/change_priorityinawa-model::admin,awa-worker::client,awa-ui::handlers,awa-cli, orawa-python.InsertOpts.prioritysets it at enqueue time;age_waiting_prioritiesdecrements it automatically on canonical storage (queue storage uses claim-time aging instead); the UI surfacespriority+original_priorityas read-only fields.Today the only paths are:
UPDATE awa.jobs_hot SET priority = …ready_entries/done_entries; bypasses metadata bookkeeping (_awa_original_priority).unique_key.UPDATE {schema}.ready_entries SET priority = Xlane_seqinto the new priority's claim-head lane (since(queue, priority, enqueue_shard)partitions claim sequences); not safe without a transactional helper.This is an operational gap for the 0.6 stable release. Incidents where one queue's pending work suddenly needs to jump the line (e.g. SLO breach, customer escalation, partner-facing job blocking a release) currently have no clean answer.
Proposed work
Two admin surfaces matching the existing
bulk-retry/bulk-cancelshape:POST /api/jobs/:id/priority{ priority: i16 }→ updates one jobPOST /api/jobs/bulk-priority{ kind?, queue?, ids?, priority }→ updates the filter selectionMirror in
awa_model::admin,awa-worker::Client,awa-cli(awa job priority <id> --to N,awa job bulk-priority --queue X --to N), andawa-python.Queue-storage implementation note
The write path is non-trivial. For an
availablerow in queue storage, changing priority means moving the row between(queue, priority, enqueue_shard)claim-head lanes — the row needs a freshlane_seqfrom the destination lane'squeue_enqueue_heads.next_seq, and the source lane'sclaim_seqmust advance past the oldlane_seqso the abandoned slot doesn't block the head. Has to be one transaction; the change must be invisible to the claimer between the source-lane advance and the destination-lane insert.For
scheduled(deferred) rows indeferred_jobs, this is a simpleUPDATE.For
running/waiting_external/ terminal rows: define what reprioritization means. Options:availableandscheduledare reprioritizablerunning, update the column so the next attempt picks up the new priority but the current attempt finishes at the old onerunning, also cancel-and-re-enqueue if the operator wants the change to take effect immediately(a) is the simplest, most predictable, and matches operator intuition ("a job that's already running can't change lanes mid-flight"). Recommend (a) unless someone has a concrete use case for (b)/(c).
Acceptance criteria
bulk-retry/bulk-cancel._awa_original_priorityis written exactly once (on the first reprioritize OR on the first aging pass, whichever comes first), so subsequent reprioritize calls don't overwrite the originally-enqueued value.awa job priority <id> --to N+awa job bulk-priority --kind X --queue Y --to N(with the same--allguard the existing bulk DLQ commands use).docs/configuration.md(priority semantics) and an admin-API doc entry.Out of scope
Related
age_waiting_prioritiesinawa-worker::maintenancefor the existing automatic aging path.