Skip to content

Commit e0b5bc7

Browse files
jeffjensenjeffjensen
committed
feat(audit): Add UnusedIndexAudit reporting indexes unused by the captured workload
Every index taxes every write and consumes cache/storage; one that no real query uses is pure cost. The inverse of the other plan audits - instead of proving one statement lacks a serving index, it proves one index serves no statement in the whole captured workload - so it deliberately does not extend CapturedSqlPlanAuditTemplate (built for per-statement findings, not a workload-wide union). Plans each candidate via the natural generic plan (no planner penalties) and collects every Index Name the plan mentions at any depth; an index is justified (never reported) when it backs a primary key or unique constraint, is partial, appears in that usage, or covers a foreign key via the new IndexDefinition.leadingColumnsCover() (extracted from ForeignKeyIndexAudit.covers(), which now delegates to it). Advisory and workload-dependent by nature: a generic plan has no real statistics, so PlanAuditsPostgreSqlIT needed real rows inserted into plan_indexed (not just empty, ANALYZEd tables) before its natural plan would actually pick the index over a Seq Scan - an empty table's natural plan prefers Seq Scan regardless of available indexes, which would have made every index look unused.
1 parent 358a339 commit e0b5bc7

11 files changed

Lines changed: 593 additions & 12 deletions

File tree

‎src/main/java/io/github/databaseaudits/audit/catalog/ForeignKeyIndexAudit.java‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
package io.github.databaseaudits.audit.catalog;
22

33
import java.util.ArrayList;
4-
import java.util.HashSet;
54
import java.util.LinkedHashMap;
65
import java.util.List;
76
import java.util.Map;
@@ -89,9 +88,7 @@ public List<Finding> audit(final String schema,
8988
* (null column) never matches.
9089
*/
9190
boolean covers(final IndexDefinition index, final List<String> fkColumns) {
92-
return !index.partial() && index.columns().size() >= fkColumns.size()
93-
&& new HashSet<>(index.columns().subList(0, fkColumns.size()))
94-
.containsAll(fkColumns);
91+
return index.leadingColumnsCover(fkColumns);
9592
}
9693

9794
private List<ForeignKey> readForeignKeys(final String schema) {

‎src/main/java/io/github/databaseaudits/audit/finding/Finding.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ public sealed interface Finding
2222
RepeatedStatementFinding, SchemaTableMissingFinding,
2323
SchemaColumnMissingFinding, SchemaColumnTypeMismatchFinding,
2424
UnconditionalMutationFinding, UniqueIndexNullableColumnFinding,
25-
UnmappedColumnFinding, UnmappedTableFinding, PlanIndexFinding {
25+
UnmappedColumnFinding, UnmappedTableFinding, UnusedIndexFinding,
26+
PlanIndexFinding {
2627
/**
2728
* Returns the human-readable description of this violation — the exact line
2829
* the audit reports.

‎src/main/java/io/github/databaseaudits/audit/finding/UnusedIndexFinding.java‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
package io.github.databaseaudits.audit.finding;
2+
3+
/**
4+
* An index no captured statement's plan uses — pure write-amplification cost
5+
* with no observed read benefit.
6+
*
7+
* @param table The table the index belongs to.
8+
* @param index The unused index.
9+
*/
10+
public record UnusedIndexFinding(String table, String index)
11+
implements Finding {
12+
@Override
13+
public String description() {
14+
return "%s.%s is used by no captured statement's plan".formatted(table,
15+
index);
16+
}
17+
}

‎src/main/java/io/github/databaseaudits/audit/runtime/plan/PlanJson.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ final class PlanJson {
1515
// EXPLAIN JSON field names.
1616
static final String NODE_TYPE = "Node Type";
1717
static final String RELATION_NAME = "Relation Name";
18+
static final String INDEX_NAME = "Index Name";
1819
static final String PLANS = "Plans";
1920
static final String PARENT_RELATIONSHIP = "Parent Relationship";
2021
static final String FILTER = "Filter";

‎src/main/java/io/github/databaseaudits/audit/runtime/plan/UnusedIndexAudit.java‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
package io.github.databaseaudits.audit.runtime.plan;
2+
3+
import java.util.Comparator;
4+
import java.util.HashSet;
5+
import java.util.List;
6+
import java.util.Locale;
7+
import java.util.Set;
8+
9+
import com.fasterxml.jackson.databind.JsonNode;
10+
11+
import org.jspecify.annotations.Nullable;
12+
13+
import io.github.databaseaudits.audit.finding.Finding;
14+
import io.github.databaseaudits.audit.finding.UnusedIndexFinding;
15+
import io.github.databaseaudits.capture.SqlCapturingStatementInspector;
16+
import io.github.databaseaudits.catalog.ForeignKeyCatalog;
17+
import io.github.databaseaudits.catalog.ForeignKeyDefinition;
18+
import io.github.databaseaudits.catalog.IndexCatalog;
19+
import io.github.databaseaudits.catalog.IndexDefinition;
20+
import io.github.databaseaudits.plan.QueryPlanExplainer;
21+
import lombok.AllArgsConstructor;
22+
import lombok.extern.slf4j.Slf4j;
23+
24+
/**
25+
* Advisory: every index should be used by at least one captured statement's
26+
* plan.
27+
*
28+
* <p>
29+
* Every index taxes every write and consumes cache/storage; one that no real
30+
* query uses is pure cost. This is the <em>inverse</em> of the other plan
31+
* audits — instead of proving one statement lacks a serving index, it proves
32+
* one <em>index</em> serves no statement in the whole captured workload — so
33+
* it does not extend {@link CapturedSqlPlanAuditTemplate} (whose fixed
34+
* algorithm emits one finding per offending statement; this audit needs the
35+
* union of index usage across every statement, then a diff against the
36+
* catalog). Candidates are planned via the natural generic plan (no planner
37+
* penalties): {@link QueryPlanExplainer#planWith(String, String...)} with no
38+
* session settings, walking every {@code Index Name} the plan mentions at any
39+
* depth. An index from {@link IndexCatalog} is <em>justified</em> — never
40+
* reported — when it backs a primary key or a unique constraint (a partial
41+
* index is also never reported: a generic plan without bind values usually
42+
* cannot prove a partial index unusable, so this is a conservative skip), when
43+
* its name appears in the collected usage, or when it covers a foreign key
44+
* (an index {@link io.github.databaseaudits.audit.catalog.ForeignKeyIndexAudit
45+
* ForeignKeyIndexAudit} demands must never be reported unused here).
46+
*
47+
* <p>
48+
* <strong>This audit is advisory and workload-dependent</strong>: the capture
49+
* must hold a representative workload, or a genuinely used index looks
50+
* unused. A generic plan also has no real table statistics, so it can miss an
51+
* index the planner would pick under production data's distribution. Always
52+
* confirm against production {@code pg_stat_user_indexes} before dropping an
53+
* index this audit reports. Requires PostgreSQL 16+ and
54+
* {@code preferQueryMode=simple} on the JDBC URL, exactly like the other plan
55+
* audits; fails fast via {@link QueryPlanExplainer#requirePlanAuditSupport(String)}
56+
* on any other platform, and throws rather than reporting nothing (or, worse,
57+
* reporting every non-justified index as unused with no evidence at all) on an
58+
* empty capture, a capture with no {@code SELECT}/{@code WITH}/{@code UPDATE}/
59+
* {@code DELETE} candidates at all (e.g. an INSERT-only workload), or a
60+
* wholly-unexplainable run.
61+
*
62+
* <p>
63+
* Fix: drop the index after confirming against production usage statistics,
64+
* or exclude it (e.g. an index kept for a rare admin query outside the
65+
* captured workload).
66+
*/
67+
@AllArgsConstructor
68+
@Slf4j
69+
public class UnusedIndexAudit {
70+
private static final String FAIL_NO_EXPLAINS_MSG = """
71+
%d candidate statement shape(s) were captured but none could be EXPLAINed,\
72+
so this audit verified nothing\
73+
— this plan-based audit is PostgreSQL 16+ only.
74+
On PostgreSQL, the most likely cause is a missing \
75+
preferQueryMode=simple on the test datasource JDBC URL.
76+
See: https://database-audits.github.io/spring-boot-integration/usage.html#postgresql-jdbc-requirement""";
77+
78+
private static final String FAIL_NO_CANDIDATES_MSG = """
79+
%d statement(s) were captured but none were SELECT/WITH/UPDATE/DELETE,\
80+
so this audit verified nothing about index usage\
81+
— every catalog index would otherwise look unused with no evidence at all.
82+
Capture a representative read/write workload (not just INSERTs) before running this audit.""";
83+
84+
private final QueryPlanExplainer queryPlanExplainer;
85+
private final SqlCapturingStatementInspector sqlCapturer;
86+
private final IndexCatalog indexCatalog;
87+
private final ForeignKeyCatalog foreignKeyCatalog;
88+
89+
/**
90+
* Returns one {@link Finding} for every index used by no captured
91+
* statement's plan, except the excluded ones; an empty list when every
92+
* index is justified.
93+
*
94+
* @param schema
95+
* The schema to scan.
96+
* @param excludedIndexes
97+
* The index names to skip.
98+
* @return One {@link Finding} per unused index, sorted by table then index
99+
* — its {@link Finding#description() description} is the reported
100+
* line; an empty list when every index is justified.
101+
* @throws UnsupportedOperationException
102+
* On any non-PostgreSQL
103+
* platform.
104+
* @throws IllegalStateException
105+
* If nothing was captured, if
106+
* nothing captured was a
107+
* SELECT/WITH/UPDATE/DELETE
108+
* candidate, or if candidates
109+
* were captured but none could
110+
* be EXPLAINed.
111+
*/
112+
public List<Finding> audit(final String schema,
113+
final Set<String> excludedIndexes) {
114+
queryPlanExplainer.requirePlanAuditSupport("UnusedIndexAudit");
115+
116+
final Set<String> capturedSql = sqlCapturer.capturedSql();
117+
if (capturedSql.isEmpty()) {
118+
throw new IllegalStateException(
119+
SqlCapturingStatementInspector.EMPTY_CAPTURE_MESSAGE);
120+
}
121+
122+
final Set<String> usedIndexNames = new HashSet<>();
123+
final Set<String> checkedShapes = new HashSet<>();
124+
int explainedCount = 0;
125+
for (final String rawSql : capturedSql) {
126+
final String trimmedSql = rawSql.strip();
127+
final String normalizedSql = sqlCapturer.normalize(trimmedSql);
128+
final String upperCasedSql =
129+
normalizedSql.toUpperCase(Locale.ROOT);
130+
if (!isCandidate(upperCasedSql)
131+
|| !checkedShapes.add(upperCasedSql)) {
132+
continue;
133+
}
134+
explainedCount += explain(trimmedSql, usedIndexNames);
135+
}
136+
137+
if (checkedShapes.isEmpty()) {
138+
throw new IllegalStateException(
139+
FAIL_NO_CANDIDATES_MSG.formatted(capturedSql.size()));
140+
}
141+
if (explainedCount == 0) {
142+
throw new IllegalStateException(
143+
FAIL_NO_EXPLAINS_MSG.formatted(checkedShapes.size()));
144+
}
145+
146+
final List<ForeignKeyDefinition> foreignKeys =
147+
foreignKeyCatalog.readAll(schema);
148+
149+
return indexCatalog.readAll(schema).stream()
150+
.filter(index -> !isJustified(index, usedIndexNames,
151+
foreignKeys))
152+
.filter(index -> !excludedIndexes.contains(index.indexName()))
153+
.sorted(Comparator.comparing(IndexDefinition::tableName)
154+
.thenComparing(IndexDefinition::indexName))
155+
.<Finding>map(index -> new UnusedIndexFinding(
156+
index.tableName(), index.indexName()))
157+
.toList();
158+
}
159+
160+
private boolean isCandidate(final String upperCasedSql) {
161+
return upperCasedSql.startsWith("SELECT")
162+
|| upperCasedSql.startsWith("WITH")
163+
|| upperCasedSql.startsWith("UPDATE")
164+
|| upperCasedSql.startsWith("DELETE");
165+
}
166+
167+
private int explain(final String sql, final Set<String> usedIndexNames) {
168+
try {
169+
final JsonNode plan = queryPlanExplainer.planWith(sql);
170+
collectIndexNames(plan, usedIndexNames);
171+
return 1;
172+
} catch (final Exception e) {
173+
log.debug(
174+
"Skipping un-explainable statement [{}]: un-checkable (parameter "
175+
+ "type inference, jsonb `?`, unparsable). The subsequent "
176+
+ "all-skipped guard still catches a wholly vacuous run.",
177+
sql, e);
178+
return 0;
179+
}
180+
}
181+
182+
private void collectIndexNames(final @Nullable JsonNode node,
183+
final Set<String> usedIndexNames) {
184+
if (node == null) {
185+
return;
186+
}
187+
final String indexName =
188+
queryPlanExplainer.textOf(node, PlanJson.INDEX_NAME);
189+
if (indexName != null) {
190+
usedIndexNames.add(indexName);
191+
}
192+
final JsonNode planNodes = node.get(PlanJson.PLANS);
193+
if (planNodes != null) {
194+
for (final JsonNode planNode : planNodes) {
195+
collectIndexNames(planNode, usedIndexNames);
196+
}
197+
}
198+
}
199+
200+
private boolean isJustified(final IndexDefinition index,
201+
final Set<String> usedIndexNames,
202+
final List<ForeignKeyDefinition> foreignKeys) {
203+
return index.primary() || index.unique() || index.partial()
204+
|| usedIndexNames.contains(index.indexName())
205+
|| coversAnyForeignKey(index, foreignKeys);
206+
}
207+
208+
private boolean coversAnyForeignKey(final IndexDefinition index,
209+
final List<ForeignKeyDefinition> foreignKeys) {
210+
return foreignKeys.stream()
211+
.filter(fk -> fk.tableName().equals(index.tableName()))
212+
.anyMatch(fk -> index.leadingColumnsCover(fk.columns()));
213+
}
214+
}

‎src/main/java/io/github/databaseaudits/catalog/IndexDefinition.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.databaseaudits.catalog;
22

3+
import java.util.HashSet;
34
import java.util.List;
45
import java.util.Objects;
56

@@ -38,4 +39,19 @@ public record IndexDefinition(String tableName, String indexName,
3839
public boolean hasExpressionColumn() {
3940
return columns.stream().anyMatch(Objects::isNull);
4041
}
42+
43+
/**
44+
* Whether this index's leading key columns cover the given columns in any
45+
* order — the rule for an index supporting a foreign key. A partial index
46+
* never covers; an expression part (null column) never matches.
47+
*
48+
* @param columns
49+
* The columns to cover.
50+
* @return {@code true} if the leading key columns cover {@code columns}.
51+
*/
52+
public boolean leadingColumnsCover(final List<String> columns) {
53+
return !partial && this.columns.size() >= columns.size()
54+
&& new HashSet<>(this.columns.subList(0, columns.size()))
55+
.containsAll(columns);
56+
}
4157
}

‎src/site/asciidoc/architecture.adoc‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ a dedicated query. It is the shared building block for `DuplicateForeignKeyAudit
6161

6262
== Runtime family
6363

64-
Six audits that inspect the *real SQL Hibernate executed* during the test run, via
65-
`SqlCapturingStatementInspector`. Three are <<plan-based>> (PostgreSQL 16+ only, via `EXPLAIN`); three are
64+
Seven audits that inspect the *real SQL Hibernate executed* during the test run, via
65+
`SqlCapturingStatementInspector`. Four are <<plan-based>> (PostgreSQL 16+ only, via `EXPLAIN`); three are
6666
<<capture-scan>> (every platform, token-scanning the captured SQL text).
6767

6868
=== SQL capture
@@ -82,10 +82,18 @@ generic-plan EXPLAIN only works over PostgreSQL's simple query protocol.
8282

8383
=== Template method
8484

85-
**`CapturedSqlPlanAuditTemplate`** is the base class for the three EXPLAIN-based audits. The fixed algorithm
86-
de-duplicates captured SQL by normalized statement shape, plans each candidate with penalties applied, and
87-
collects offending nodes. Both vacuous-run guards — empty capture and wholly-unexplainable run — live here.
88-
Subclasses supply which statements to EXPLAIN, which GUCs to penalize, and how to recognize an offending node.
85+
**`CapturedSqlPlanAuditTemplate`** is the base class for three of the four EXPLAIN-based audits
86+
(`WhereClauseIndexAudit`, `OrderByIndexAudit`, `JoinIndexAudit`). The fixed algorithm de-duplicates captured SQL
87+
by normalized statement shape, plans each candidate with penalties applied, and collects offending nodes. Both
88+
vacuous-run guards — empty capture and wholly-unexplainable run — live here. Subclasses supply which statements
89+
to EXPLAIN, which GUCs to penalize, and how to recognize an offending node.
90+
91+
**`UnusedIndexAudit`** is the fourth plan-based audit, and deliberately does *not* extend the template: the
92+
template emits one finding per _offending statement_, but this audit needs the union of index usage across
93+
_every_ statement, then a diff against the catalog — the inverse shape. It plans each candidate via the natural
94+
generic plan (no penalties), collects every `Index Name` the plan mentions, and reports each catalog index
95+
(from `IndexCatalog`) that is neither justified by a primary key, a unique constraint, being partial, appearing
96+
in that usage, nor covering a foreign key (via `ForeignKeyCatalog`).
8997

9098
[[capture-scan]]
9199
=== Capture-scan audits (every platform)

‎src/site/asciidoc/audits.adoc‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
= Audits
22

3-
Eighteen audits organized into three families. Catalog and JPA audits run on every supported database platform;
3+
Nineteen audits organized into three families. Catalog and JPA audits run on every supported database platform;
44
runtime audits split into two kinds — plan-based (PostgreSQL 16+ only) and capture-scan (every platform).
55

66
Every `audit()` call returns a `List<Finding>` — empty when clean. Each `Finding` is a record
@@ -284,6 +284,36 @@ Nested Loop with inner Seq Scan on 'orders' joining (order_items.order_id = orde
284284

285285
*Fix:* Add an index on the joined column(s), or exclude the relation or SQL fragment.
286286

287+
==== UnusedIndexAudit
288+
289+
Advisory: every index should be used by at least one captured statement's plan. Every index taxes every write
290+
and consumes cache/storage; one that no real query uses is pure cost. The inverse of the other plan audits — it
291+
proves one *index* serves no statement in the whole captured workload, planning each candidate via the natural
292+
generic plan (no planner penalties) and walking every `Index Name` the plan mentions. An index is never
293+
reported when it backs a primary key or a unique constraint, when it is partial (a generic plan without bind
294+
values usually cannot prove a partial index unusable), when its name appears in the collected usage, or when it
295+
covers a foreign key (an index `ForeignKeyIndexAudit` demands is never reported unused here).
296+
297+
CAUTION: This audit is advisory and workload-dependent. The capture must hold a representative workload, or a
298+
genuinely used index looks unused. A generic plan also has no real table statistics, so it can miss an index
299+
the planner would pick under production data's distribution — always confirm against production
300+
`pg_stat_user_indexes` before dropping an index this audit reports.
301+
302+
[source,java]
303+
----
304+
List<Finding> findings = new UnusedIndexAudit(explainer, inspector, indexes, foreignKeys)
305+
.audit("my_schema", Set.of("idx_kept_for_admin_report"));
306+
----
307+
308+
*Finding format:*
309+
310+
----
311+
orders.idx_orders_legacy_status is used by no captured statement's plan
312+
----
313+
314+
*Fix:* Drop the index after confirming against production usage statistics, or exclude it (e.g. an index kept
315+
for a rare admin query outside the captured workload).
316+
287317
[[runtime-capture]]
288318
=== Capture-scan runtime audits (all platforms)
289319

0 commit comments

Comments
 (0)