The size of your distributed storage is not important; its the number of packets per second and bits per second that each node will need to support which matters when delivering your solution.
The specific NICs you select will make a difference. The "NIC speed" is the network attachment bit rate, it is not the throughput capability of the NIC hardware.
I saw a post on the misc@ mailing list this week (
link) from an OpenBSD that noted that the more popular 10Gb NICs may be those that use the ix(4) driver. Issuing
$ apropos 10gb produced a slightly longer list:
Code:
$ apropos 10gb
ix(4) - Intel 82598/82599/X540/X550 PCI Express 10Gb Ethernet device
ixgb(4) - Intel PRO/10GbE 10Gb Ethernet device
myx(4) - Myricom Myri-10G PCI Express 10Gb Ethernet device
nep(4) - Sun Neptune 10Gb Ethernet device
oce(4) - Emulex OneConnect 10Gb Ethernet device
tht, thtc(4) - Tehuti Networks 10Gb Ethernet device
xge(4) - Neterion Xframe/Xframe II 10Gb Ethernet device
The network stack has been historically single threaded through CPU 0. Work to improve this has been ongoing in recent years, but much of the stack is still single-threaded and that advice for faster/fewer CPUs is still valid.
Routers do not use much memory, so most of your 32 GB of RAM in each router is likely to sit idle.
For more specific hardware advice, you might review the misc@ archives.