PF_RING Zero Copy Intel igb Driver Crash (Rocky Linux 9.5)

[fandigunawan]

Hi, I am trying to install PF_RING Zero Copy on our Intel igb but when the computer boot for few seconds it is suddenly crash and reboot.
Here is the full information:

• OS: Rocky Linux 9.5
• Kernel version: 5.14.0-503.40.1.el9_5.x86_64
• Machine: Dell PowerEdge R660
• PF_RING version: pfring-9.0.0-9896
• Software that use PF_RING: Suricata

Driver versions:

• e1000e-zc-3.8.7.9896-dkms.noarch.rpm
• i40e-zc-2.24.6.9896-dkms.noarch.rpm
• iavf-zc-4.9.5.9896-dkms.noarch.rpm
• ice-zc-1.12.7.9896-dkms.noarch.rpm
• igb-zc-5.14.16.9896-dkms.noarch.rpm
• ixgbe-zc-5.19.6.9896-dkms.noarch.rpm
• ixgbevf-zc-4.18.9.9896-dkms.noarch.rpm
• ntopng-data-6.4.250515-25785.noarch.rpm
• pfring-dkms-9.0.0.9896-dkms.noarch.rpm
• pfring-drivers-zc-dkms-9.0.0-9896.noarch.rpm

Main error trace:

[   46.168771] igb 0000:16:00.0 ens1f0: PCIe link lost
[   46.263302] pcieport 0000:15:01.0: AER: Root Port link has been reset (0)
[   46.263559] igb 0000:16:00.0: enabling device (0000 -> 0002)
[   46.263838] igb 0000:16:00.1: enabling device (0000 -> 0002)
[   46.264097] igb 0000:16:00.2: enabling device (0000 -> 0002)
[   46.264355] igb 0000:16:00.3: enabling device (0000 -> 0002)
[   46.349430] [PF_RING] Registering ZC device ens1f0@0 [rx-ring=00000000f46bd8dc][tx-ring=0000000083715ebb]
[   46.349623] [PF_RING] Registering ZC device ens1f0@1 [rx-ring=00000000de739896][tx-ring=00000000816c4074]
[   46.349764] [PF_RING] Registering ZC device ens1f0@2 [rx-ring=000000001a77f489][tx-ring=00000000249120d8]
[   46.349901] [PF_RING] Registering ZC device ens1f0@3 [rx-ring=000000002dfa07ed][tx-ring=000000005a2c76be]
[   46.350035] [PF_RING] Registering ZC device ens1f0@4 [rx-ring=00000000b703d9af][tx-ring=000000005dfac6c1]
[   46.350164] [PF_RING] Registering ZC device ens1f0@5 [rx-ring=00000000e34aadca][tx-ring=00000000061e1fda]
[   46.350289] [PF_RING] Registering ZC device ens1f0@6 [rx-ring=000000008fb077f2][tx-ring=00000000436445ed]
[   46.350418] [PF_RING] Registering ZC device ens1f0@7 [rx-ring=000000003d77cdb0][tx-ring=00000000d44ca7ba]
[   46.386184] list_del corruption. prev->next should be ff42815208eb38b8, but was ff3830f0c8f64458
[   46.386387] ------------[ cut here ]------------
[   46.386576] kernel BUG at lib/list_debug.c:51!
[   46.386745] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   46.386748] CPU: 29 PID: 2602 Comm: W#01-zc:e..f0@0 Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.40.1.el9_5.x86_64 #1
[   46.386749] Hardware name: Dell Inc. PowerEdge R660/XXXXXX, BIOS 2.4.4 09/27/2024
[   46.386750] RIP: 0010:__list_del_entry_valid.cold+0x31/0x47
[   46.386755] Code: c0 c7 a0 e8 35 77 fe ff 0f 0b 48 c7 c7 b0 c0 c7 a0 e8 27 77 fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 70 c0 c7 a0 e8 13 77 fe ff <0f> 0b 48 89 fe 4c 89 c2 48 c7 c7 38 c0 c7 a0 e8 ff 76 fe ff 0f 0b
[   46.386757] RSP: 0018:ff42815208eb36e8 EFLAGS: 00010046
[   46.386758] RAX: 0000000000000054 RBX: ff42815208eb38a0 RCX: 0000000000000000
[   46.386759] RDX: 0000000000000000 RSI: ff38310fffba08c0 RDI: ff38310fffba08c0
[   46.386760] RBP: ff3830f0c8f64450 R08: 0000000000000000 R09: ff42815208eb35a8
[   46.386761] R10: ff42815208eb35a0 R11: ffffffffa17e93e8 R12: 0000000000000286
[   46.386761] R13: ff42815208eb3890 R14: 0000000000000001 R15: ff42815208eb3760
[   46.386762] FS:  00007f05677ff640(0000) GS:ff38310fffb80000(0000) knlGS:0000000000000000
[   46.386763] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   46.386764] CR2: 00007f058c5a8008 CR3: 000000018227c004 CR4: 0000000000771ef0
[   46.386765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   46.386765] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   46.386766] PKRU: 55555554
[   46.386766] Call Trace:
[   46.386767]  <TASK>
[   46.386768]  ? show_trace_log_lvl+0x1c4/0x2df
[   46.386773]  ? show_trace_log_lvl+0x1c4/0x2df
[   46.386775]  ? remove_wait_queue+0x20/0x60
[   46.386778]  ? __die_body.cold+0x8/0xd
[   46.386780]  ? die+0x2b/0x50
[   46.386782]  ? do_trap+0xce/0x120
[   46.386784]  ? __list_del_entry_valid.cold+0x31/0x47
[   46.386786]  ? do_error_trap+0x65/0x80
[   46.386788]  ? __list_del_entry_valid.cold+0x31/0x47
[   46.386789]  ? exc_invalid_op+0x4e/0x70
[   46.386793]  ? __list_del_entry_valid.cold+0x31/0x47
[   46.386794]  ? asm_exc_invalid_op+0x16/0x20
[   46.386799]  ? __list_del_entry_valid.cold+0x31/0x47
[   46.386800]  ? __list_del_entry_valid.cold+0x31/0x47
[   46.386802]  remove_wait_queue+0x20/0x60
[   46.386804]  poll_freewait+0x3d/0xa0
[   46.386808]  do_sys_poll+0x176/0x230
[   46.386810]  ? remove_wait_queue+0x20/0x60
[   46.386812]  ? poll_freewait+0x45/0xa0
[   46.386813]  ? do_sys_poll+0x176/0x230
[   46.386815]  ? __pfx_pollwake+0x10/0x10
[   46.386817]  ? pick_next_task_idle+0x26/0x40
[   46.386819]  ? pick_next_task+0x9f9/0xaf0
[   46.386821]  ? dequeue_task_fair+0xaa/0x370
[   46.386824]  ? __switch_to_asm+0x3a/0x80
[   46.386826]  ? finish_task_switch.isra.0+0x8c/0x2a0
[   46.386828]  ? __pfx_pollwake+0x10/0x10
[   46.386830]  ? schedule+0x2e/0xd0
[   46.386833]  ? schedule_hrtimeout_range_clock+0x9d/0x120
[   46.386836]  ? wait_packet_function_ptr+0x64/0xc0 [igb_zc]
[   46.386852]  ? ring_poll+0x61/0x280 [pf_ring]
[   46.386864]  ? sock_poll+0x4c/0xe0
[   46.386867]  ? do_poll.constprop.0+0x298/0x380
[   46.386868]  ? __seccomp_filter+0x45/0x480
[   46.386871]  ? ktime_get_ts64+0x49/0xf0
[   46.386873]  __x64_sys_poll+0xa6/0x140
[   46.386876]  do_syscall_64+0x5c/0xf0
[   46.386877]  ? __audit_filter_op+0xa5/0xf0
[   46.386879]  ? fpregs_restore_userregs+0x47/0xd0
[   46.386881]  ? exit_to_user_mode_prepare+0xef/0x100
[   46.386883]  ? syscall_exit_to_user_mode+0x19/0x40
[   46.386884]  ? do_syscall_64+0x6b/0xf0
[   46.386886]  ? __pfx_pollwake+0x10/0x10
[   46.386887]  ? do_syscall_64+0x6b/0xf0
[   46.386888]  ? schedule+0x2e/0xd0
[   46.386890]  ? rseq_get_rseq_cs+0x1d/0x240
[   46.386892]  ? rseq_ip_fixup+0x6e/0x1a0
[   46.386893]  ? rseq_get_rseq_cs+0x1d/0x240
[   46.386894]  ? rseq_ip_fixup+0x6e/0x1a0
[   46.386895]  ? rseq_get_rseq_cs+0x1d/0x240
[   46.386896]  ? rseq_ip_fixup+0x6e/0x1a0
[   46.386897]  ? fpregs_restore_userregs+0x47/0xd0
[   46.386899]  ? exit_to_user_mode_prepare+0xef/0x100
[   46.386900]  ? syscall_exit_to_user_mode+0x19/0x40
[   46.386900]  ? do_syscall_64+0x6b/0xf0
[   46.386901]  ? do_syscall_64+0x6b/0xf0
[   46.386902]  ? do_syscall_64+0x6b/0xf0
[   46.386903]  ? do_syscall_64+0x6b/0xf0
[   46.386904]  ? do_syscall_64+0x6b/0xf0
[   46.386905]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[   46.386906] RIP: 0033:0x7f05e1d015df
[   46.386932] Code: 54 24 1c 48 89 74 24 10 48 89 7c 24 08 e8 69 4b f8 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 48 8b 7c 24 08 b8 07 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 89 44 24 08 e8 bd 4b f8 ff 8b 44
[   46.386933] RSP: 002b:00007f05677fceb0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[   46.386934] RAX: ffffffffffffffda RBX: 00007f056028cbf0 RCX: 00007f05e1d015df
[   46.386935] RDX: 000000000000003d RSI: 0000000000000001 RDI: 00007f05677fcee8
[   46.386935] RBP: 00007f056028cbf0 R08: 0000000000000000 R09: 0000000000000000
[   46.386936] R10: 0000000000000000 R11: 0000000000000293 R12: 000000000000003d
[   46.386936] R13: 0000000000000000 R14: 00007f05677fd018 R15: 00007f05677fcf30
[   46.386938]  </TASK>
[   46.386938] Modules linked in: binfmt_misc ice_zc(OE) gnss igb_zc(OE) vxlan ip6_udp_tunnel udp_tunnel uio pf_ring(OE) nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables libcrc32c nfnetlink ipmi_ssif vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i40e kvm ib_uverbs rapl dell_wmi ledtrig_audio acpi_ipmi sparse_keymap rfkill iTCO_wdt video intel_cstate ipmi_si dell_smbios pmt_telemetry mgag200 iTCO_vendor_support dcdbas ib_core pmt_class intel_sdsi dell_wmi_descriptor wmi_bmof intel_uncore mei_me ipmi_devintf pcspkr drm_shmem_helper i2c_i801 isst_if_mbox_pci isst_if_mmio drm_kms_helper mei intel_vsec isst_if_common i2c_ismt i2c_smbus ipmi_msghandler acpi_power_meter joydev fuse drm ext4 mbcache jbd2 sd_mod t10_pi sg iaa_crypto ahci
[   46.386976]  crct10dif_pclmul libahci crc32_pclmul crc32c_intel megaraid_sas idxd i2c_algo_bit tg3 libata ghash_clmulni_intel idxd_bus wmi pinctrl_emmitsburg dm_mirror dm_region_hash dm_log dm_mod [last unloaded: gnss]

Complete log:

pf_ring_zc_igb_crash_RL9.5.log

[cardigliano]

Are you able to boot the system by blacklisting the driver?
What is the exact adapter model? (as reported by lspci)

[fandigunawan]

This is the result from lspci

lspci | grep -i eth
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
16:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
16:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
16:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
16:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
2a:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
2a:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
2a:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
2a:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
98:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
98:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)

Yes, if we disable the driver then it is okay. I will try with PF_RING standard mode (without ZC). I will let you know.

[fandigunawan]

There are two types of Intel Ethernet here:

• igb: Intel I350
• ice: Intel E810

Intel I350 crash using PF_RING Zero Copy

** pfcount -i zc:ens1f0
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC 9.0.0.250428
# license for ens1f0 (MAC 20:3A:43:0A:80:F8) [Intel 1 Gbit igb 82580-based]
# ERROR: Missing license file
# ERROR: Please get one at http://shop.ntop.org/
#########################################################################
# PF_RING ZC running in demo mode (packet capture and transmission
# limited to 5 minutes)
#########################################################################
Using PF_RING v.9.0.0.250428 kernel module v.9.0.0
Dumping statistics on /proc/net/pf_ring/stats/3335-ens1f0.48
error reading link speed on ens1f0
Capturing from zc:ens1f0 [mac: 20:3A:43:0A:80:F8][if_index: 34][speed: 1000Mb/s]
# Device RX channels: 8
# Polling threads:    1**

Intel E810 does not crash using PF_RING Zero Copy

pfcount -i zc:ens2f1np1
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC 9.0.0.250428
# license for ens2f1np1 (MAC B4:83:51:1E:25:F1) [Intel 100 Gbit ice family]
# ERROR: Missing license file
# ERROR: Please get one at http://shop.ntop.org/
#########################################################################
# PF_RING ZC running in demo mode (packet capture and transmission
# limited to 5 minutes)
#########################################################################
Using PF_RING v.9.0.0.250428 kernel module v.9.0.0
Dumping statistics on /proc/net/pf_ring/stats/3330-ens2f1np1.47
Capturing from zc:ens2f1np1 [mac: B4:83:51:1E:25:F1][if_index: 43][speed: 10000Mb/s]
# Device RX channels: 16
# Polling threads:    1
=========================
Absolute Stats: [0 pkts total][0 pkts dropped][0.0% dropped]
[0 pkts rcvd][0 bytes rcvd]
=========================

=========================
Absolute Stats: [0 pkts total][0 pkts dropped][0.0% dropped]
[0 pkts rcvd][0 bytes rcvd][0.00 pkt/sec][0.00 Mbit/sec]
=========================
Actual Stats: [0 pkts rcvd][1'000.03 ms][0.00 pps][0.00 Gbps]
=========================


=========================
Absolute Stats: [0 pkts total][0 pkts dropped][0.0% dropped]
[0 pkts rcvd][0 bytes rcvd][0.00 pkt/sec][0.00 Mbit/sec]
=========================
Actual Stats: [0 pkts rcvd][1'000.03 ms][0.00 pps][0.00 Gbps]
=========================

[cardigliano]

We are checking the driver, please use the interface without zc: in the meantime. Thank you.

[fandigunawan]

We are checking the driver, please use the interface without zc: in the meantime. Thank you.

We plan to use Zero Copy in the production, please let us know. FYI, we have bought licenses in the past and will use PF_RING ZC in the near future.

[cardigliano]

Does it crash as soon as you start the application (e.g. pfcount)? Or after any specific action (e.g. when you close it)?

[fandigunawan]

Does it crash as soon as you start the application (e.g. pfcount)? Or after any specific action (e.g. when you close it)?

It crashed a few seconds after application running (pfcount or suricata)

[cardigliano]

Does it start after receiving some packets, or just during initialization? Sorry for all those questions, unable to reproduce so far, thus any additional info are useful digging..

[fandigunawan]

Does it start after receiving some packets, or just during initialization? Sorry for all those questions, unable to reproduce so far, thus any additional info are useful digging..

Actually, the cable is unplugged (no cable connected to the ethernet) therefore there is no data received. In E810, it is okay when the cable is unplugged.

[cardigliano]

From the log it seems an hardware error is triggering when opening the device, and this is unloading the driver while the interface is in use by the application. I am digging to figure out why this error occurs.
In the meantime, could you try plugging a cable, to make sure this is not related to the absence of carrier?

[cardigliano]

Also, please try configuring 8 RSS queues rather than 16

[fandigunawan]

Also, please try configuring 8 RSS queues rather than 16

Yes, it is already 8

# pf_ringcfg --list-interfaces
Name: eno8303              Driver: tg3        RSS:     1    [Linux Driver]
Name: eno8403              Driver: tg3        RSS:     1    [Linux Driver]
Name: eno12399             Driver: igb        RSS:     8    [Running ZC]
Name: eno12409             Driver: igb        RSS:     8    [Running ZC]
Name: eno12419             Driver: igb        RSS:     8    [Running ZC]
Name: eno12429             Driver: igb        RSS:     8    [Running ZC]
Name: ens3f0               Driver: igb        RSS:     8    [Running ZC]
Name: ens3f1               Driver: igb        RSS:     8    [Running ZC]
Name: ens3f2               Driver: igb        RSS:     8    [Running ZC]
Name: ens3f3               Driver: igb        RSS:     8    [Running ZC]
Name: ens1f0np0            Driver: ice_zc     RSS:     16   [Running ZC]
Name: ens1f1np1            Driver: ice_zc     RSS:     16   [Running ZC]
Name: ens2f0np0            Driver: ice_zc     RSS:     16   [Running ZC]
Name: ens2f1np1            Driver: ice_zc     RSS:     16   [Running ZC]