ice: raw (protocol-agnostic) FDIR rules added by a VF return flow_id=0, making every delete fail with RULE_NONEXIST

[nsubramanian-gif]

Summary

A VF adds protocol-agnostic ("raw") FDIR rules via VIRTCHNL_OP_ADD_FDIR_FILTER.
The rules are programmed and demonstrably steer traffic to the intended
queue. However the flow_id returned to the VF is 0 for every rule, so
every subsequent VIRTCHNL_OP_DEL_FDIR_FILTER is rejected by the PF:

ice 0000:81:00.1: VF 4: FDIR invalid flow_id:0x0

Because the VF never receives a usable handle, rules that work can never be
deleted. They accumulate until the VF's VSI is destroyed (sriov_numvfs=0).
VF FLR, PF devlink driver_reinit and PF PCIe FLR do not release them.

This makes raw FDIR unusable for any application that restarts: each restart
permanently consumes filter resources on that VF.

Reproduced with stock dpdk-testpmd — no application code required.

Environment

item	value
ice driver	2.3.10 (out-of-tree). Also verified present in v2.6.7 / main by source inspection
adapter	E810, PCI ID [8086:1592], subsystem [8086:0002]
firmware / NVM	fw 7.9.1 api 1.7.11 nvm 4.90 0x80020ef2 1.3863.0
DDP	ICE COMMS Package 1.3.57.0 (fw.app.bundle_id 0xc0000002), loaded successfully at probe
fw.netlist	4.4.2000-3.28.0 (build 0x2723426e)
host kernel	6.8.12-15-pve (SMP PREEMPT_DYNAMIC)
guest kernel	7.0.0-29-generic
VF driver	DPDK 25.11.0 iavf PMD via vfio-pci

Reproducer

VF bound to vfio-pci:

dpdk-testpmd -l 6-7 -n 4 -a 0000:05:00.0 --file-prefix=t -m 1024 \
    -- -i --rxq=4 --txq=4 --forward-mode=rxonly

Create raw rules. The spec is a complete 79-byte Ethernet/IPv4/UDP frame; the
mask selects only the EtherType, IP protocol, UDP destination port, and two
bytes of UDP payload (a 1-bit flag and a 4-bit selector at a fixed offset).
Each rule targets a different queue by matching a different selector value:

# queue 0
flow create 0 ingress pattern raw pattern spec aabbccddeeff11223344556608004500004112340000401142640a0909010a090902303901bb002d000040000000000000000000000000000000000000000000000000000000000000000000000000 pattern mask 000000000000000000000000ffff000000000000000000ff000000000000000000000000ffff00000000803c0000000000000000000000000000000000000000000000000000000000000000000000 / end actions mark id 233472 / queue index 0 / end

# queue 1 (payload byte 43 = 0x04 instead of 0x00)
flow create 0 ingress pattern raw pattern spec aabbccddeeff11223344556608004500004112340000401142640a0909010a090902303901bb002d000040040000000000000000000000000000000000000000000000000000000000000000000000 pattern mask 000000000000000000000000ffff000000000000000000ff000000000000000000000000ffff00000000803c0000000000000000000000000000000000000000000000000000000000000000000000 / end actions mark id 233473 / queue index 1 / end

Repeat for 16 rules, then delete them one at a time:

flow destroy 0 rule 0
flow destroy 0 rule 1
...

Observed

All creates report success. Every delete fails:

iavf_fdir_del(): Failed to delete rule request due to this rule doesn't exist
iavf_flow_destroy(): Failed to destroy flow
port_flow_complain(): Caught PMD error type 2 (flow rule (handle)):
                      Failed to delete filter rule.: Operation not permitted

With PF-side dynamic debug enabled:

echo 'file ice_virtchnl_fdir.c +p' > /sys/kernel/debug/dynamic_debug/control

the PF logs, once per delete attempt:

ice 0000:81:00.1: VF 4: FDIR invalid flow_id:0x0

Expected

Each delete succeeds — or, failing that, the preceding add reports failure.

Side observation in the same run

The first flow create of a fresh session fails, and subsequent identical
creates succeed:

port_flow_complain(): Caught PMD error type 2 (flow rule (handle)):
                      Failed to create parser engine.: Invalid argument

In the attached log 15 of 16 rules were created, the single failure being the
first. This is reproducible and looks like first-use initialisation of the
parser profile; it is reported here for completeness rather than as the main
issue, but may share a root cause with the flow_id handling.

Attachments

• repro-raw-fdir-delete.sh — standalone reproducer (testpmd only)
• testpmd-raw-fdir-delete-failure.log — full testpmd session showing the
  creates succeeding and all deletes failing
• pf-dmesg-excerpt.txt — PF-side driver/firmware/DDP provenance and the
  FDIR invalid flow_id:0x0 lines

Evidence the rules really are programmed in hardware

With a full rule set (16 raw rules, 4 queues) installed on a freshly created VF,
the device steers every packet to the intended queue:

	rules	RX packets	delivered to wrong queue
no rules (baseline)	0	6,106,479	4,971,215 — 81.4%
16 raw rules	16	3,451,435	0 — none

All 16 rules installed first try, zero retries. Every received packet was
delivered to the queue its matched rule selected; the cross-queue counter did
not increment once. Sustained throughput 501 Mbps at 0.0000% loss
(0 of 7,614,267 packets).

A second run with a partial rule set (8 rules covering 2 of the 4 packet shapes)
steered 47.5% — consistent with the uncovered shapes having no rule to match,
and confirming the rules act exactly where they are installed and nowhere else.

These rules work perfectly. They still cannot be deleted. That is the whole
of this report: the filters are correctly programmed and demonstrably steering
millions of packets, while the flow_id returned to the VF is 0, so there is no
handle with which to remove them.

Analysis

1. Blocker — flow_id returned as 0 for rules that were programmed

ice_vc_del_fdir_fltr() rejects at its first step:

conf = ice_vc_fdir_lookup_entry(vf, fltr->flow_id);
if (!conf) {
	stat->status = VIRTCHNL_FDIR_FAILURE_RULE_NONEXIST;
	dev_dbg(dev, "VF %d: FDIR invalid flow_id:0x%X", vf->vf_id, fltr->flow_id);

On its success path ice_vc_add_fdir_raw() does return 0 without sending a
response; the reply is emitted asynchronously by ice_vc_add_fdir_fltr_post(),
which is the only site that sets resp->flow_id = conf->flow_id. A raw add that
programs hardware but whose reply carries flow_id = 0 implies that completion
either did not run for the raw path, or ran against a stale ctx->conf.

2. VF reset orphans raw rules in hardware

ice_vf_fdir_exit() runs on every VF reset. Its rule-clearing step is
software-only:

static void ice_vc_fdir_flush_entry(struct ice_vf *vf)
{
	list_for_each_entry_safe(desc, temp, &vf->fdir.fdir_rule_list, fltr_node) {
		list_del(&desc->fltr_node);
		kfree(conf);            /* no hardware write anywhere */
	}
}

It never calls ice_vc_fdir_add_del_raw(vf, conf, false). The classic FDIR path
does reach hardware (ice_vc_fdir_rem_prof_all() → ice_rem_prof_id_flow() /
ice_flow_rem_entry()); the raw path has no equivalent, and
vf->fdir_prof_info[ptg] is touched by no cleanup path in the tree. Hardware
entries survive while the conf holding their handle is freed.

This function is byte-identical in v2.6.7 and main.

3. 0 is a legal flow_id

ice_vc_fdir_insert_entry() uses
idr_alloc(&vf->fdir.fdir_rule_idr, conf, 0, ICE_FDIR_MAX_FLTRS, GFP_KERNEL),
so id 0 is allocatable and the PF cannot distinguish "id 0" from "unset".
Unchanged in v2.6.7.

Status in the latest release

src/ice_virtchnl_fdir.c at tag v2.6.7 is byte-identical to main. Diffed
against 2.3.10 the whole file changes by 102 lines, and no changed line
touches flow_id, fdir_prof_info, or idr_alloc. So items 1–3 above are
all still present in the latest release.

One related fix has landed since 2.3.10 and is acknowledged: the
ice_flow_set_hw_prof() failure path in ice_vc_add_fdir_raw() now sets
stat->status = VIRTCHNL_FDIR_FAILURE_RULE_NORESOURCE rather than falling
through with a kzalloc'd stat (which previously reported success with
flow_id = 0). That is a real improvement, but it does not address the case
reported here, where the rule is successfully programmed.

Requested fixes, in priority order

1. Return the real flow_id in the raw add response. This is the blocker.
2. ice_vc_fdir_flush_entry() should delete each entry from hardware before
   freeing its conf, so a VF reset does not orphan working rules.
3. ice_vf_fdir_exit() should release vf->fdir_prof_info[].
4. Reserve 0 as an invalid flow_id.

Ruled out

• Not a DPDK bug — the iavf PMD returns to the PF exactly the flow_id
  the PF supplied. Reproduced identically under a second, unrelated DPDK
  application.
• Not resource exhaustion — reproduced immediately after
  sriov_numvfs=0 teardown and recreate, on a PF with no other FDIR users.
• Not the action set — fails identically with queue alone and with
  mark + queue.
• Not DDP — devlink dev info reports ICE COMMS Package 1.3.57.0, which
  provides raw/protocol-agnostic pattern support, loaded successfully at probe.

pf-dmesg-excerpt.txt
repro-raw-fdir-delete.sh
testpmd-raw-fdir-delete-failure.log

[nsubramanian-gif]

Root cause found: FD profile IDs exhausted, and SR-IOV teardown does not reclaim them

Follow-up with whole-module dynamic debug (module ice +p). My original report
blamed flow_id handling — that is a symptom three layers down. Here is the
actual failure, captured on a freshly SR-IOV-reset PF with the VF's first and
only driver incarnation installing 16 raw FDIR rules:

ice 0000:81:00.1: AQ Command: opcode 0x0208, datalen 0x0006          x110
ice 0000:81:00.1: AQ Response: ... retval 0x0000                      x95
ice 0000:81:00.1: Control Send Queue command 0x0208 error 0x10        x15
ice 0000:81:00.1: Failed to set LAN Tx queue context, error: -22
ice 0000:81:00.1: Could not open control VSI for VF 4
ice 0000:81:00.1: Init FDIR for VF 4 failed, error: -22 (-EINVAL)     x1
ice 0000:81:00.1: VF 4: FDIR invalid flow_id:0x0                      x16

The failing alloc_res request buffer decodes unambiguously:

58 20 01 00 00 00
└─┬─┘ └─┬─┘
0x2058  num_elems = 1
  0x58 = ICE_AQC_RES_TYPE_FD_PROF_BLDR_PROFID
0x2000 = ICE_AQC_RES_TYPE_FLAG_SCAN_BOTTOM
0x10   = ICE_AQ_RC_ENOSPC

Firmware is refusing to allocate FD profile-builder profile IDs.

Chain

alloc_res(FD_PROF_BLDR_PROFID) -> ENOSPC
  -> Failed to set LAN Tx queue context -22
  -> Could not open control VSI for VF
  -> Init FDIR for VF failed -22
  -> ice_vc_add_fdir_raw() returns the kzalloc'd stat:
       status = 0 = VIRTCHNL_FDIR_SUCCESS, flow_id = 0
  -> VF is told 16 rules added successfully
  -> every delete sends flow_id 0 -> lookup_entry NULL -> RULE_NONEXIST

Because VIRTCHNL_FDIR_SUCCESS == 0 and stat comes from kzalloc(), a bail-out
that sets no status is indistinguishable from success on the wire.

It is all-or-nothing, and nearly silent

Every flow_id in the capture is 0x0 — 16 of them, one per delete — with no
add-success line anywhere. There is no partial success. And 16 adds produced 15
ENOSPC failures but only one Init FDIR failed: the other fifteen bail before
reaching any dev_err, so the VF has no way to know.

SR-IOV teardown does not reclaim FD profile IDs

95 of 110 allocations succeeded on this freshly reset PF, so it is not broadly out
of resources — only FD profile IDs are gone. sriov_numvfs=0 + recreate rebuilds
VFs and VSIs but does not return FD profile IDs to firmware. VF FLR, PF
devlink driver_reinit and PF PCIe FLR do not either. On this host the pool has
been draining across days of VF add/exit cycles with no reboot.

Requested fixes, revised

1. Free FD profile IDs when a VF's raw FDIR rules go away — on rule delete and
   on ice_vf_fdir_exit(). This is the leak; everything else follows from it.
2. Do not report success for an add that was never programmed. The bail-out
   after a failed ice_vf_start_ctrl_vsi() / ice_flow_set_hw_prof() must set
   stat->status to a failure before ice_vc_send_msg_to_vf().
3. Log every failed add, not just the first — fifteen were silent.
4. Reserve 0 as an invalid flow_id (idr_alloc(..., 0, ...) makes it a legal
   id today, so "unset" and "id 0" are indistinguishable).

Items 2–4 make the failure visible; item 1 stops it happening. The
testpmd reproducer in the original report still applies — it will show this
behaviour once the FD profile ID pool on the PF is exhausted.