I'm benchmarking two machines with SFC9250 cards direct connected with 25G fiber on one port, as well as via a 10G switch on the other port. If the ctpio threshold is larger than the packet size, there is no problem on either port. On the 10G port, if the ctpio threshold is set lower than the payload length in eflatency, the ctpio fallback seems to work -- I see the expected CRC error on the receive side and it then receives the correct packet afterwards (I'm running with my #238 patches to validate that the packet received matches, although it occurs with the stock build as well).
On the 25G direct connect port, however, the fallback doesn't always work. It isn't deterministic; sometimes it makes it through several iterations before failing, other times it fails in the first few loops like this one. Usually after a EF_EVENT_TYPE_RX_DISCARD with a bad CRC caused by the CTPIO underun, there is a good EF_EVENT_TYPE_RX, but sometimes it never arrives and the ping or the pong can get hung waiting for the next packet.
cpuset-exec taskset -c 2 ./thudson/eflatency -s 128 -c 64 -r -v -v pong ens2f0
# ef_vi_version_str: b58f121a 2024-08-29 eflatency-echo ef_vi
# NIC(s) 22 -1
# udp payload len: 128:128:1
# iterations: 100000
# warmups: 10000
# frame len: 170
# mode: CTPIO
# vlan: 0
# validating: yes
# verbose: 2
event [ev:2]
RX discard bad CRC (2)
event [ev:0]
event [ev:1]
event [ev:2]
RX discard bad CRC (2)
event [ev:0]
event [ev:1]
event [ev:0]
event [ev:1]
event [ev:2]
RX discard bad CRC (2)
event [ev:0]
event [ev:1]
event [ev:0]
event [ev:1]
event [ev:2]
RX discard bad CRC (2)
Interestingly, it seems to also depend on the value of ctpio threshold and the size of the packet. Very lower values for the threshold work (8-16), as do ones close to the packet size (90-128). It also does not seem to affect very short (<64bytes) or larger packets like 1K.
This is with onload 8.1.3 and ethtool -i ens2f0 reports:
driver: sfc
version: 5.3.18.1012
firmware-version: 8.0.0.1015 rx1 tx1
expansion-rom-version:
bus-info: 0000:5e:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes
lspci | grep 5e
5e:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
5e:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
I'm benchmarking two machines with SFC9250 cards direct connected with 25G fiber on one port, as well as via a 10G switch on the other port. If the ctpio threshold is larger than the packet size, there is no problem on either port. On the 10G port, if the ctpio threshold is set lower than the payload length in
eflatency, the ctpio fallback seems to work -- I see the expected CRC error on the receive side and it then receives the correct packet afterwards (I'm running with my #238 patches to validate that the packet received matches, although it occurs with the stock build as well).On the 25G direct connect port, however, the fallback doesn't always work. It isn't deterministic; sometimes it makes it through several iterations before failing, other times it fails in the first few loops like this one. Usually after a
EF_EVENT_TYPE_RX_DISCARDwith a bad CRC caused by the CTPIO underun, there is a goodEF_EVENT_TYPE_RX, but sometimes it never arrives and thepingor thepongcan get hung waiting for the next packet.Interestingly, it seems to also depend on the value of ctpio threshold and the size of the packet. Very lower values for the threshold work (8-16), as do ones close to the packet size (90-128). It also does not seem to affect very short (<64bytes) or larger packets like 1K.
This is with onload 8.1.3 and
ethtool -i ens2f0reports:lspci | grep 5e