PF_RING ZC crashes when using ixgbevf

[dbdante1]

For context, I am running this on a c4.8xlarge machine on AWS that supports sriov and has ixgbevf driver. The OS is Rocky linux 8. I am using the packaged stable build for this OS. We are considering to purchase pf_ring with zc but need to make sure it can work in our envt.

Below is the error that keeps popping up in dmesg -w that starts showing up after running pfsend -i zc:eth1 once.

 jiffies              <100718640>
 [ 7741.188390] [PF_RING] Removing ZC device eth2@0 [rx-ring=000000007d79a4f8][tx-ring=000000009d69b52e]
 [ 7741.197391] [PF_RING] Registering ZC device eth2@0 [rx-ring=000000007d79a4f8][tx-ring=000000009d69b52e]
 [ 7741.201529] ixgbevf 0000:00:05.0 eth2: NIC Link is Up 10 Gbps
[ 7743.274095] Detected Tx Unit Hang
                  Tx Queue             <0>
                  TDH, TDT             <0>, <2>
                  next_to_use          <2>
                  next_to_clean        <0>
                tx_buffer_info[next_to_clean]
                  next_to_watch         <00000000a7b0848f>
                  eop_desc->wb.status  <168080>
                  time_stamp           <1007182b2>
                  jiffies              <100719280>
 [ 7745.258037] Detected Tx Unit Hang
                  Tx Queue             <0>
                  TDH, TDT             <0>, <2>
                  next_to_use          <2>
                  next_to_clean        <0>
                tx_buffer_info[next_to_clean]
                  next_to_watch         <000000000ab5a238>
                  eop_desc->wb.status  <168080>
                  time_stamp           <100718a6c>
                  jiffies              <100719a40>
 [ 7745.283639] [PF_RING] Removing ZC device eth1@0 [rx-ring=00000000afd0d75b][tx-ring=00000000c54d3acd]
[ 7745.291441] [PF_RING] Registering ZC device eth1@0 [rx-ring=00000000afd0d75b][tx-ring=00000000c54d3acd]
[ 7745.295458] ixgbevf 0000:00:04.0 eth1: NIC Link is Up 10 Gbps
[ 7747.331834] [PF_RING] Removing ZC device eth2@0 [rx-ring=000000007d79a4f8][tx-ring=000000009d69b52e]
[ 7747.339545] [PF_RING] Registering ZC device eth2@0 [rx-ring=000000007d79a4f8][tx-ring=000000009d69b52e]
[ 7747.343525] ixgbevf 0000:00:05.0 eth2: NIC Link is Up 10 Gbps
[ 7749.354125] Detected Tx Unit Hang
                  Tx Queue             <0>
                  TDH, TDT             <0>, <2>
                  next_to_use          <2>
                  next_to_clean        <0>
                tx_buffer_info[next_to_clean]
                  next_to_watch         <00000000a7b0848f>
                  eop_desc->wb.status  <168080>
                  time_stamp           <100719a6a>
                  jiffies              <10071aa40>
 [ 7751.402001] Detected Tx Unit Hang
                  Tx Queue             <0>
                  TDH, TDT             <0>, <2>
                  next_to_use          <2>
                  next_to_clean        <0>
                tx_buffer_info[next_to_clean]
                  next_to_watch         <000000000ab5a238>
                  eop_desc->wb.status  <168080>
                  time_stamp           <10071a26a>
                  jiffies              <10071b240>

As for output from pfsend
`

pfsend -i zc:eth1
Sending packets on zc:eth1
Using PF_RING v.9.2.0 on zc:eth1
Dumping statistics on /proc/net/pf_ring/stats/78723-eth1.1
TX rate: [current 4'010.23 pps/0.00 Gbps][average 4'010.23 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 2'026.27 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 1'355.59 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 1'018.49 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 815.66 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 680.16 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 583.30 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 510.58 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 453.99 pps/0.00 Gbps][total 4'096.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 408.68 pps/0.00 Gbps][total 4'096.00 pkts]
^CLeaving...
TX rate: [current 0.00 pps/0.00 Gbps][average 382.84 pps/0.00 Gbps][total 4'096.00 pkts]`

I am able to confirm that ZC has been loaded using the following

> `[root@ip-172-31-36-166 ec2-user]# cat /proc/net/pf_ring/dev/eth1/info
> Name:         eth1
> Index:        24
> Address:      0A:79:05:4C:6A:C7
> Polling Mode: NAPI/ZC
> Promisc:      Disabled
> Type:         Ethernet
> Family:       Intel ixgbe VF
> TX Queues:    1
> RX Queues:    1
> Num RX Slots: 4096
> Num TX Slots: 4096
> RX Slot Size: 2048
> TX Slot Size: 2048`

pf_ringcfg

> ` pf_ringcfg --list-interfaces
> Name: eth0                 Driver: ixgbevf    RSS:     1    [Running ZC]
> Name: eth1                 Driver: ixgbevf    RSS:     1    [Running ZC]
> Name: eth2                 Driver: ixgbevf    RSS:     1    [Running ZC]`

Interfaces config:

`[root@ip-172-31-36-166 ec2-user]# cat /etc/pf_ring/interfaces.conf
MANAGEMENT_INTERFACES="eth0"
CAPTURE_INTERFACES="eth1 eth2"`

I have set forcestart to bypass the management check and am starting and stopping the service using systemctl
The logs for systemctl status pf_ring are as follows
`

May 12 16:26:40 ip-172-31-36-166.us-east-2.compute.internal systemd[1]: Starting PF_RING service...
May 12 16:26:41 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[68328]: Starting PF_RING module:
May 12 16:26:41 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[68557]: netlink error: Operation not supported
May 12 16:26:41 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[68580]: netlink error: Operation not supported
May 12 16:26:41 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[68613]: netlink error: Operation not supported
May 12 16:26:44 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[68328]: [ OK ]
May 12 16:26:44 ip-172-31-36-166.us-east-2.compute.internal systemd[1]: Started PF_RING service.
May 12 17:26:26 ip-172-31-36-166.us-east-2.compute.internal systemd[1]: Stopping PF_RING service...

`

Using set -x found the following command creating the "Operation not supported" error. However this command appears to be something that has to be executed from host. So I think it is safe to ignore it.

> `May 12 18:03:51 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[76633]: + ethtool -L eth0 combined 2
> May 12 18:03:51 ip-172-31-36-166.us-east-2.compute.internal pf_ringctl[76911]: netlink error: Operation not supported`

I have tried several different OS(Redhat, Ubuntu) and OS versions, including manual compilation from git. All of them resulted in the same error.

[cardigliano]

@dbdante1 could you provide more details about the configuration and how reproduce this? First of all, what is the application? Do you need to transmit traffic, while receiving?

[dbdante1]

The application is to use masscan. To verify pf_ring setup I am using pfsend. The easiest way to reproduce this to setup PF_RING on c4.8xlarge in AWS and it costs less than a 1$ if we choose 2 vCPUS. The instance by default comes with sriov support and has ixgbevf driver. I am not sure what configuration you are looking for. Can you please let me know the commands you would like to run and I can share the output? Thanks for the response.