Intel Xeon-D x552/x557 10baseT fails to negiotiate a 10Gb link
The link with an old US-16-XG switch negotiates at 1Gbps (note: this is a US-16-XG, not a ES-16-XG)
When the Supermicro Xeon-D system is plugged into the 10Gbase-T port of a US-16-XG switch a 1Gbps link is established. Setting the switch port to negotiate at 10G doesn't overcome the limitation.
TinkerTry talks about an updated firmware file SDV23B_UEFI.zip. I got SDV23C_NUP.zip from Supermicro (without an NDA) with very simple instructions. This took the firmware from '0x800003e7' to '0x800008af, 1.1937.0'.
FIX: Apply firmware from Supermicro.
Links
- https://tinkertry.com/how-to-work-around-intermittent-intel-x557-network-outages-on-12-core-xeon-d
- https://community.ui.com/questions/es-16-xg-and-Xeon-D-10Gbase-T-problems/ef4079e4-0f7b-44bd-8af0-a4d9951bea50
- https://www.supermicro.com/en/products/motherboard/X10SDV-8C-TLN4F
- https://www.supermicro.com/en/products/system/1U/5018/SYS-5018D-FN4T.cfm
- https://community.ui.com/questions/US-16-XG-10G-port-15-and-16-stuck-at-1gigabit/c700d3b4-b634-415c-adee-a05872f80a11
- https://community.ui.com/questions/UniFi-Network-Product-Chipset-Matrix-and-Birth/feb69f12-321d-4102-a5dd-bbab81d08483
- https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm5341x
- https://www.supermicro.com/support/faqs/faq.cfm?faq=32181
- https://www.intel.com/content/www/us/en/download/15755/29137/intel-ethernet-connections-boot-utility-preboot-images-and-efi-drivers.html?product=88342
- https://portal.nutanix.com/page/documents/kbs/details?targetId=kA00e000000CxriCAC
Appendices
starting versions
These versions were obtained from the IPMI UI:
Firmware Revision: 03.68 Firmware Build Time: 03/20/2018 BIOS Version: 2.0a BIOS Build Time: 10/12/2018 Redfish Version : 1.0.1
X557 version (after update)
# ethtool -i eno3 driver: ixgbe version: 6.2.16-6-pve firmware-version: 0x800008af, 1.1937.0 expansion-rom-version: bus-info: 0000:03:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
manually set port speed
(UBNT) >enable (UBNT) #configure (UBNT) (Config)#interface 0/13 (UBNT) (Interface 0/13)#speed auto 10G full-duplex
ethtool info
# ethtool -i eno3 driver: ixgbe version: 6.2.16-6-pve firmware-version: 0x800003e7 expansion-rom-version: bus-info: 0000:03:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
CPU Info
processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 86 model name : Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz stepping : 3
PCI Info
# lspci -vvs 03:00.0
03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
DeviceName: Intel X557-AT2 Ethernet #1
Subsystem: Super Micro Computer Inc Ethernet Connection X552/X557-AT 10GBASE-T
Physical Slot: 0-1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at fbc00000 (64-bit, prefetchable) [size=2M]
Region 4: Memory at fbe04000 (64-bit, prefetchable) [size=16K]
Expansion ROM at 90100000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number 00-00-c9-ff-ff-00-00-00
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 15a8
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000090200000 (64-bit, non-prefetchable)
Region 3: Memory at 0000000090300000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1b0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [1c0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: ixgbe
Kernel modules: ixgbe
SDV23C_NUP files
The update file SDV23C_NUP.zip from Supermicro contains a firmware update for EFI and Linux. Sha256 hashes below:
73bdf725a68180efca1f4eedbed350c10eb1132401c59c8fc2a6e982e7c2d1a7 10G.txt 98531eb253849251ca6170354584c6d73dc5b2fec313e7bbb2d92336adf438d1 BDXDE23C.bin 4e6a02fc722de5761cd7ef12d9ec9a633e92400b119dc8bc3c6425cf11060f3b FWUpdate.nsh 0245392a9d78a9d75cf4f99eabe0a749168a50100ed24dadd66c5c46b6f6a85d FWUpdate.sh 1e977f46826cd75e1a7f660e5a8c4a1917054b53b03821de64da5a89659e8d1c Instruction.txt 6345b1fe7545ef4108dd7a2d6d72ac34794ae5f5d7813c68feb765fcc3589642 nvmupdate64e 2a3d730333ec085b393750b373d379a469ca8fe12256508a265776a219a16a04 nvmupdate64e.efi 5b99dc3b6884c74243f53627d0b5b3ef51c3b1f42f7c5628ef25d0bd59147e35 nvmupdate.cfg 3eabb3095818d9cdc05c0f3a31a8587b27bcaf4db111a88ad66e30295e3e85be PHY310816.bin e8a3d3d2c7584ae715f6576a5f0204d7be7f6bbba25d164e6dda8a7004275ccc Release Note.txt
Firmware update log
# ./FWUpdate.sh
Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.32.13.0
Copyright (C) 2013 - 2018 Intel Corporation.
Config file read.
Inventory
[00:003:00:00]: Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
Flash inventory started.
Shadow RAM inventory started.
Shadow RAM inventory finished.
Flash inventory finished.
OROM inventory started.
OROM inventory finished.
PHY NVM inventory started.
PHY NVM inventory finished.
[00:003:00:01]: Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
Device already inventoried.
Update
[00:003:00:00]: Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
Flash update started.
|======================[100%]======================|
NVM image verification started.
Shadow RAM image verification started.
|======================[100%]======================|
Shadow RAM image verification finished.
Flash image verification started.
|======================[100%]======================|
Flash image verification finished.
NVM image verification finished.
Flash update successful.
PHY NVM update started.
|======================[100%]======================|
PHY NVM image verification started.
|======================[100%]======================|
PHY NVM image verification finished.
PHY NVM update successful.
PHY reset started.
PHY reset successful.
Checking update availability for next tool run.
Power Cycle is required to complete the update process.
