Current Trends in DC Networking - VXLAN Overview

The data center has two main struggles when it comes to networking.

  1. Loop free layer two by sacrificing half the links.
  2. VM or workload mobility anywhere at any time.

Smaller shops might be fine with giving up a few links, but extending a L2 domain is asking for trouble.  Larger shops need every link they can get and demand the flexibility to move a VM anywhere. So we are being forced down a road of L2 everywhere. How do we do it?

Our current model is broken, or better said, it's out dated. We need a technology that is adaptable to the movement of the application. This will take  some time...

Right now though, we have VMs needing to move within pods or zones or even DCs. Yup and application teams still require L2 domains between all this mess. So how do we make everyone happy?

Well... VXLAN

Its not the protocol we need but its the protocol we deserve.

Im not going deep into VXLAN but just covering the basics. I dont want to spend too much time here as we have much funner things to talk about. But this is an important topic for any network engineer working in the DC.


RFC 7348 - Virtual eXtensible Local Area Network (VXLAN)
RFC 7432 - BGP MPLS-Based Ethernet VPN
Arista VXLAN: Scaling Data Center Capacity
VXLAN Without Controller for Network Virtualization with Arista physical VTEPs
VXLAN bridging with MLAG
CiscoLive BRKDCT-2404 - VXLAN Deployment Models - A practical perspective
VXLAN Overview: Cisco Nexus 9000 Series Switches

What is is?

VXLAN basically tunnels traffic across an infrastructure allowing you to stretch broadcast domains across routed infrastructure. VXLAN has been adopted by multiple vendors as the base model for DC design.

VXLAN is not perfect but its a great protocol to solve our current issues. 


VNI (VXLAN Network Identifier) - Unique 24 bit identifier usually mamed 1-to-1 with a VLAN
VTEP (Virtual Tunnel End Points) - Network device/interface used to encapsulate and de-encapsulate VXLAN services. 
VTEP Gateway - Bridges traffic between a VXLAN segment and non-VXLAN network
VXLAN Bridge - Bridging a VLAN/VNI between two VTEPs over IP network. 
VXLAN Segment - The VLAN/VNI which spans the VXLAN Bridge 


VXLAN encapsulation is straight forward if you understand the basics of UDP packet construction. The original L2 frame has to be tagged or marked in some way for the destination VTEP to know what VLAN it's destine for. A new header is placed on the frame to accomplish this. 

A 24 bit value called a VNI is placed into a VXLAN header to identify the VLAN for the destination. VTEPs use a table to determine VLAN to VNI mapping. The original frame and new VXLAN header are then encapsulated into a UDP packet and processed like any other packet on a network. Within the UDP packet the VTEP interface MAC, and IP addresses are used for source/destination address fields. 

 Traffic Flow

To better understand how VXLAN works lets walk through the common traffic flows between the two VMs in the topology below.

Initial ARP
  1. VM1 does not know the MAC address of the destination device VM2 and sends an ARP.
  2. VTEP1 receives the ARP and encapsulates it into a VXLAN packet with a VNI of 800 then
    • multicast the packet to the assigned multicast group or
    • forwards it to all VTEPs he knows. 
    • NOTE: ARP is considered BUM traffic. We will cover this in a minute and the various options
  3. VTEP2 receives the packet and sees VNI 800 in the VXLAN header. Then looks up the VNI to VLAN mapping which is associated with VLAN 101
  4. Just like normal switching VTEP2 then places VM1s MAC address in the VXLAN table associated with VTEP1 and VNI 800
  5. VTEP2 then floods the original ARP request to all nodes associated with VLAN 101
  6. VM2 responds to the ARP and VTEP2 sees its destine to VM1, encapsulates with VXLAN header inside UDP and forwards to VTEP1.
  7. VTEP1 receives the packet and sees VNI 800 in the VXLAN header along with VM1s MAC address in arp reply and forwards the ARP response to VM1
    • VTEP1 then places VM2s MAC address in the VXLAN table associated with VTEP2 and VNI 800
  8. VM1 receives the ARP response and now knows how to get to VM2
Standard Traffic flow

  1. VM1 send traffic with a destination MAC towards VTEP1
  2. VTEP1 sees VM2s MAC associated with VTEP2 and VNI 800 and encapsulates the traffic with a VXLAN header with VNI 800. The UDP source is VTEP1s IP and destination is VTEP2s IP. 
  3. VTEP2 receives the traffic and sees VNI 800 in the vxlan header and VM2s MAC and forwards the traffic to VM2. 
  4. Return traffic is handled the same way.

BUM Traffic

Broadcast, Unicast and Multicast (BUM) traffic can be handled in a VXLAN environment a number of ways. Which method is recommend and supported varies by vendor.

Multicast - Multicast is used to distribute BUM traffic to VTEPs who have joined the multicast group associated to the VNI. Obviously multicast must be setup in the network and it adds an extra layer of complexity, but it is by far the most efficient method to handle BUM traffic. Most all vendors support it as well.

Unicast - Not all vendors support this option and each seems to handle it a little differently. In a nut-shell any BUM traffic that lands on a VTEP has to be sent to all other VTEPs in the environment. This one makes my head hurt when comparing different vendors implementation but its available so there are use cases for it.

BGP EVPN - EVPN is an open standard (RFC 7432) and VXLAN utilizes the BGP EVPN address family to distribute VXLAN traffic, utilizing communities to distribute the VXAN header. But from my understanding BUM traffic is handle the same as unicast mode above. The VTEP (PE) router still hase to send the traffic to all VTEPs. EVPN is complex and if you are not already used to setting up MPLS VPNs then you will struggle to manage EVPN.


VXLAN is good but its still only a band-aid for what we need. We dont need more BGP address-families or tunneling protocols. We need a revamp of the whole stack, but that is a ways out. So VXLAN is the current king in DC networking trends and we will focus there.

Next up we get into the fun stuff, configuring VXLAN!!