BE Networks
Demo

Enabling RoCEv2 in Enterprise

By Luke Williams – Product Lead

RDMA over Converged Ethernet (RoCE) provides Remote Direct Memory Access (RDMA) over an Ethernet network, enabling memory transfer between two devices that bypasses the CPU on each device. This is a capability that is configurable in Enterprise SONiC. We at BE Networks have worked closely with our partners to bring this capability into Verity, to simplify fabric-wide configuration of SONiC on their hardware platforms.

Verity 6.3 (current GA version) supports configuring RoCE in SONiC. Verity uses Device Setting Templates to enable this feature on a per switch basis. This guide will detail the steps to create the template, apply it to the switch, and then verify the configuration on the switch.

Create a Device Setting Template
To create the template, log into Verity and navigate to the Templates Dashboard. In Provisioning, there is a box labeled Device Settings. Double click that box to zoom in and see existing templates.
Figure 01

Figure 1

Next, we need to create a new template. To do this, click the and give it a name. In our example we called it RoCEv2 Enabled.

Figure 2

Once the template is created, click the Pencil icon in the upper right corner to edit the template. Place a check in the Enable box and then put a check in the RoCEv2 box in Network Configuration and click the Check mark to save the template.
Figure 03

Figure 3

Apply the Template to Devices
Now we need to apply this template to the switches that will use these RoCE settings. This includes all leafs and spines that are in the path of the RoCE traffic. To apply this template, in the Topology view, select the switch, and zoom in to the Device Settings box, shown in the following screenshot:
Figure 04

Figure 4

Double click this box to zoom in, then click the Pencil Icon to apply the template. From the drop-down menu, select the “RoCEv2 Enabled” template.
Figure 05

Figure 5

Click the Checkbox to save the settings and apply the template to the switch. A dialog box will appear confirming you want to make the change and that the switch will reboot to apply these settings.
Figure 06

Figure 6

Click Yes to make the changes. The switch will reboot to apply these settings. Once the switch comes online and reports in, you will see the RoCEv2 setting in the Device Settings in the Topology view of the switch:

Figure 7

To verify that the RoCE settings are applied to the switch, we can look at the running configuration of the switch and see the following entries: First, near the top of the running-configuration, you see “roce enable”
Figure 08

Figure 8

Next, if you scroll down to the QoS settings, you will see the configured RoCE QoS settings:

Figure 09

Figure 9

And last, if you scroll down to the interface configurations, you will see the interfaces are all configured to use the QoS and PFC settings:
Figure 10

Figure 10

You can apply this Device Settings template to any switches that need to have RoCE enabled.
In a future blog post, we will cover how to manually configure these settings to fine tune RoCE traffic. This functionality is due in a near-term release of Verity.

RoCE Limitations

RoCE is currently limited to Trident 3×7, Trident 4×11, Tomahawk 3, Tomahawk 4 and Tomahawk 5 chips sets.

  • Enabling RoCE requires the switch to reboot after configuration is applied.
  • Breakout ports must be configured BEFORE enabling RoCE.
  • Tuning RoCE parameters is a future capability in Verity and is not supported in the current release.
  • RoCE is required to be enabled on all devices that will be in the path of RoCE traffic.
  • Servers need to be configured to use RoCE.
BE Networks logo
Product
Verity ®
Features
Hardware
Cost Savings
Whitepapers
Try Verity
Solutions
SONiC
Data Center
AI Networks
Private Cloud
Campus Edge
Infrastructure as Code
Network Migration
About
Leadership
News & Videos
Contact Us
Stay Connected
LinkedIn

LinkedIn

© BE Networks 2025 – Terms – Privacy
en_US
ja ko_KR fr_FR en_US
Contact Us
We really like talking about networks!