Corundum: An Open-Source 100-Gbps NIC

[Ken Eguro]

Corundum: An Open-Source 100-Gbps NIC – Link for PDF
Alex Forencich (University of California, San Diego), Alex C. Snoeren (University of California, San Diego), George Porter (University of California, San Diego), and George Papen (University of California, San Diego)

[alex.forencich]

Thanks for dropping by and listening to my presentation on Corundum! Feel free to ask questions, I will get back to you as soon as I can.

If it sounds like an interesting project, be sure to check out the source code on GitHub. You can also join the Google group.

[kanshi]

Good work, congratulations. Few questions:
1. What’s the packet-rate performance (i.e. using small pkts such as 64B) that can be achieved from this work?
2. For both Tx and Rx, it is not clear how pkts are forwarded to ports. NIC will normally embody packet parser, classifier and lookup/exact-match/LPM/TCAM module for pkt lookup and forwarding, but those are missing from this design. Is this handled by the SW, i.e. determining which pkt goes to which queue, and directly mapping a given queue to a certain port?
3. The above question really is, that if this design is intended to be a ‘smart NIC’ or normal NIC. The former will offload several common network function stacks to FPGAs and free the CPU cores, and that’s the main point of these types of designs.
4. Is this NIC using customized SW driver? If so, two main issues. One, performance may be limited, but authors mentioned to use DPDK-based driver in the future work, so it will improve performance. Second, limited generalization capability.

[alex.forencich]

Thanks for the questions, kanshi.

First, without a DPDK driver, it’s hard to measure performance for small packet sizes without being limited by overhead of the Linux network stack (after all, we want to measure capabilities of the NIC hardware, not the network stack software), so we don’t currently have any reliable measurements for packet sizes other than MTU-size frames. Once we have a DPDK driver up and running, we’ll have a better idea of what the design is capable of in terms of small packet sizes. However, I will say that the current design is relatively simple and as such has some limitations – namely, it currently only supports fixed-size descriptor blocks with no support for inlining data in the descriptor rings. As a result, throughput for minimum-length frames is rather poor due to PCIe overheads. The theoretical max with the current configuration is around 58 Mpps for minimum-length frames, compared to the 142 Mpps theoretical max of 100G Ethernet.

In the TX direction, software classifies packets into queues, and then the transmit schedulers on each port determine which queues traffic will be sent from. In the receive direction, RSS flow hashing is used to select the receive queue. It would be possible to add additional logic to implement something more complex by pre-processing packets before handing them off to Corundum to pass to the host. At any rate, software-based classification in the transmit direction is required to dedicate queues to individual flows to avoid issues with head-of-line blocking.

Corundum is intended to support high-precision transmit scheduling, something that most NICs (even most smart NICs) can only do in a very limited way. More complex functionality such as match-action rules, if necessary, can be implemented outside of the main corundum modules. In this case, corundum could be used as a high-performance host interface for a smart NIC design.

Corundum uses a custom driver that registers the device as a standard Ethernet interface. All of the testing carried out so far has been with the standard Linux networking stack.

[Dustin Richmond]

Thank you for posting your question!

[Author]

Posts