SONIC and Leaf and Spine

If that title makes you think of a pixelated hedgehog who has acquired two new sidekicks then you’re probably as old as me.

However, SONIC is an open source networking operating system originally developed by Microsoft and its command line will be familiar to anybody who has worked with Cisco. I used it to play with leaf and spine networking.

This fine gentlemen provides a YouTube tutorial on how to set-up a simple leaf and spine network in GNS3 using vxlan evpn which I wholeheartedly recommend. This post is essentially me following this video, adapting it slightly and just playing with it to try and gain a little more understanding.

My network looked like this – and probably had a few more hosts than I needed. I will include the spine and leaf switch configurations but they are all explained in the video above which also has links to the relevant configuration elements rather than the full configs.

I thought I would try and see what was going on when we ping from PC4 (11.1.1.3) on the left and which hangs off leaf1 to PC2 (11.1.1.2) on the right which hangs off leaf2; both these hosts are in VLAN 11.

And the arp for this is…

…which is the VXLAN interface on the leaf1 switch

The capture from PC4 to leaf1 is unremarkable..

However, the capture between the leaf1 switch and the spine switch shows the vxlan protocol nestled between UDP and Ethernet…

Expanding this out, we see that VLAN 11 has been mapped to a VNI of 100011, as described in the video…

…with the source ip address of 100.1.1.1 and the destination ip address of 100.2.2.2 being the leaf1 and leaf2 vlxan vtep addresses respectively.

Vtep address on leaf1…

…And on leaf2…

The capture from leaf2 to PC2 has no special encapsulation

But the source mac address is shown on leaf2 as originating from the leaf1 vxlan interface

So any host on vlans 10 and 11 (which are defined in the configuration) can connect to any other host on these vlans on either switch (which would probably be in different datacentres) which in the age of spinning up VMs wherever you want them is a boon. This is also the starting point of Azure Cloud networking where by default anything you spin up can connect to anything else. In connectivity terms, it’s great, in security terms obviously the old model of network segmentation goes out of the window.