Troubleshooting
Known problems and their fixes.
Dual-port ConnectX-7 throughput collapse
If one receiver delivers full bandwidth but two functions on the same ConnectX-7 fall to approximately 30 Gb/s each, check PCI_WR_ORDERING. The accompanying NIC counters usually show sustained rx_discards_phy, tx_pause_ctrl_phy, and rx_buffer_passed_thres_phy growth even though the link and PCIe bus are healthy.
This is especially indicative of PCIe write-order blocking when each function targets a separate GPU. Configure every function to force_relax(1) by following ConnectX-7 PCIe Write Ordering.
Invalid HDF5 file generated by the writer blocks
Sometimes, the libhdf5 and VFD-GDS installations can fall out of sync, leading to issues. If the HDF5 file generated by the UVH5 Writer or FBH5 Writer ends up invalid, check whether both installations are compatible by consulting their respective documentation. As a good practice, ensure that any previous installation is fully removed to avoid stale library objects lingering around. For reference, the following commits are known to work well together:
hdf5 -> d4525d758f49da5972f34b1c53d189dcd9bcc3d0
vfd-gds -> e33bae67a16238c2677f23abac76b9fe30b99869