Cheap Desktop 400GbE Switch MikroTik CRS804-4DDQ-HRM Review

Noting actually says those ports have to be for management.

Most people probably would use them for something else.

I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.

By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.

I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway.

Please be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.

The second sentence of the article:

> We have a number of these now and have been running our RDMA network backend for the 8x NVIDIA GB10 cluster for several months now, using one.

I have a friend who uses something similar for his 4 NVIDIA GDX Spark cluster.

DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.

I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.