Skip to main content

DCI with PBB-EVPN and Cisco ASR9000

Lately I have been spending some time with LAB-testing new Cisco ACI environment (more about ACI in further posts). As a multi-DC service provider DC-interconnect is of utmost importance. I have implemented some Nexus 7000/5000 environments using vPC DC-interconnect (dark fiber, CWDM) which have worked quite nicely. Especially if there are requirements of MACSEC (802.1AE) and such, N7K M-series linecards are quite nice with 10G line rate MACSEC.

However there has been some cases where direct L2-interconnect is not possible. Especially in this ACI-case if you want to stretch the fabric (one fabric) to other DC you need 8 x 40G (!) in between, which is quite a lot. You could do the ACI interconnect (two separate ACI fabrics) from Leaf to Leaf with N x 10G, but still the L2 over L3 brings some advantages.  Therefore I have been looking into different technologies for DCI. Lately I have lab-tested some PBB-EVPN (Provider Backbone Bridging – Ethernet VPN) DCI with Cisco ASR’s, and it works quite nicely I must say. In the lab tests I used 1 x 10G towards DC’s in both ends and 2 x 10G bundled between the ASR’s simulating the EVPN/MPLS in between. As there is no “real” MPLS in between, this simply can be thought of as two directly connected PE-routers. There are two extended VLAN’s in this example; 503 and 751.

Physical topology

 

EVPN physical topology

 

Logical topology

EVPN logical topology

 

Configurations for ASR9001 – 1. Configs for the second router are exactly the same apart from the IP’s.

Interface configurations. Here the TenGigE0/0/2/1 are the ports towards the switching infrastructure, where subinterfaces 503 and 751 are enabled as tagged l2transport interfaces. TenGigE0/0/2/2 and /3 are bundled as Bundle-Ether100 and these form the DCI. Loopback10 is used as iBGP update source.

 

OSPF and BGP configurations. OSPF and BGP configurations are very basic for this lab. One OSPF area with MPLS LDP sync (which will sync OSPF and LDP processes to prevent MPLS packet loss). BGP configuration does not need anything else for EVPN to work, only address-family l2vpn evpn for BGP to be able to propagate MAC-addresses.

 

Basic MPLS LDP configuration. Nothing special here.

 

PBB-EVPN configuration. Here are the required EVPN configurations. Starting from the beginning PBB backbone source MAC, this is not necessary as the router will select this automatically (configure this to distinguish endpoints from each other more easily). At least two bridge groups are required. First bridge group contains the PBB Edge bridge domains, these are the DC-facing BD’s (in my case one L2 transport interface (VLAN) per BD). Every Edge bridge domain must have their own I-SID (Instance Service Identifier) to distinguish them in both/all endpoints, the Edge bridge domain must also be attached to a Core bridge domain.

Second bridge group contains the PBB Core bridge domains which are the MPLS core / EVPN facing BD’s. This only requires EVI (Ethernet VPN Instance) ID in basic configuration.

 

Verify OSPF, MPLS and BGP. Verify that OSPF neighbors are up and routes are in OSPF database. MPLS LDP neighborship is up. BGP neighborship is up and also EVPN address family is up and present. Also you can verify the BGP EVPN table.

 

Verifying EVPN MAC-address table. You can run the following commands to verify on ASR that the MAC’s are being learnt on the correct interface and bridge-domain. Also you can see the remote MAC’s learnt, where the endpoint is the neighbor backbone source MAC-address (in this case manually assigned ASR1: 0001.0002.1111 and ASR2: 0001.0002.2222).

And on ASR2:

Also note that on ASR-platform PBB-EVPN requires Enhanced Ethernet linecards (Typhoon) (in this example inbuilt ports on ASR9001 were used, which are of Typhoon spec). Also software required is at least IOS-XR 4.3.2 or 5.1.1.

This post just serves as an example on how to configure L2 over L3 using PBB-EVPN and is by no means all-in-all guide on how to do it in production. You would want to tune for better OSPF and BGP convergence by adjusting timers, configuring BFD and such. There is some really nice EVPN documentation and material at Cisco Live! On-demand Library and also the ASR documentation has some great guides and specifications. Especially I would recommend reading Cisco Live! course material “EVPN & PBB-EVPN: the Next Generation of MPLS-based L2VPN”, which gives really nice insight on PBB-EVPN, how it works, use cases, problem scenarios and such.

I’m also trying to get some hands on with some Juniper MX -routers for EVPN tests with Juniper. We’ll see more about that in later posts.

7 thoughts to “DCI with PBB-EVPN and Cisco ASR9000”

  1. Hi Jesse,

    Thanks for the nice writeup on PBB-EVPN with working example. I tried ur example on identical setup( instead of Bundle-Ether, I have one 10G port on core side) on my lab, however my bgp neighbour session didn’t came up( ospf, ldp working though). below is “sh bgp neighbor” output:

    ===========================================
    RP/0/RSP0/CPU0:C9006_0001#show bgp neighbor
    Tue Oct 27 14:25:05.448 UTC

    BGP neighbor is 7.7.7.2
    Remote AS 65100, local AS 65100, internal link
    Remote router ID 0.0.0.0
    BGP state = Idle
    Last read 00:00:00, Last read before reset 00:00:11
    Hold time is 180, keepalive interval is 60 seconds
    Configured hold time: 180, keepalive: 60, min acceptable hold time: 3
    Last write 00:00:11, attempted 22, written 22
    Second last write 00:00:16, attempted 19, written 19
    Last write before reset 00:00:16, attempted 19, written 19
    Second last write before reset 00:00:18, attempted 53, written 53
    Last write pulse rcvd Oct 27 14:24:53.995 last full not set pulse count 3694
    Last write pulse rcvd before reset 00:00:11
    Socket not armed for io, not armed for read, not armed for write
    Last write thread event before reset 00:00:11, second last 00:00:16
    Last KA expiry before reset 00:00:00, second last 00:00:00
    Last KA error before reset 00:00:00, KA not sent 00:00:00
    Last KA start before reset 00:00:00, second last 00:00:00
    Precedence: internet
    Multi-protocol capability received
    Received 2617 messages, 38 notifications, 0 in queue
    Sent 3292 messages, 524 notifications, 0 in queue
    Minimum time between advertisement runs is 0 secs

    For Address Family: L2VPN EVPN
    BGP neighbor version 0
    Update group: 0.2 Filter-group: 0.0 No Refresh request being processed
    Route refresh request: received 1, sent 1
    0 accepted prefixes, 0 are bestpaths
    Cumulative no. of prefixes denied: 0.
    Prefix advertised 0, suppressed 0, withdrawn 0
    Maximum prefixes allowed 2097152
    Threshold for warning message 75%, restart interval 0 min
    AIGP is enabled
    An EoR was not received during read-only mode
    Last ack version 0, Last synced ack version 0
    Outstanding version objects: current 0, max 1
    Additional-paths operation: None

    Connections established 571; dropped 571
    Local host: 7.7.7.1, Local port: 34575, IF Handle: 0x00000000
    Foreign host: 7.7.7.2, Foreign port: 179
    Last reset 00:00:11, due to BGP Notification sent: illegal network
    Time since last notification sent to neighbor: 00:00:11
    Error Code: illegal network
    Notification data sent:
    03
    Time since last notification received from neighbor: 00:01:35
    Error Code: illegal network
    Notification data received:
    03
    ===========================
    RP/0/RSP0/CPU0:C9006_0001#sh tcp br
    Tue Oct 27 14:30:18.930 UTC
    PCB VRF-ID Recv-Q Send-Q Local Address Foreign Address State
    0x10188610 0x60000000 0 0 :::179 :::0 LISTEN
    0x10147cdc 0x00000000 0 0 :::179 :::0 LISTEN
    0x10159658 0x60000000 0 0 7.7.7.1:646 7.7.7.2:56071 ESTAB
    0x101377bc 0x60000000 0 32 47.134.11.102:23 10.128.233.207:61522 ESTAB
    0x10131ad8 0x60000000 0 0 0.0.0.0:23 0.0.0.0:0 LISTEN
    0x1012d5d8 0x00000000 0 0 0.0.0.0:23 0.0.0.0:0 LISTEN
    0x101325b4 0x60000000 0 0 0.0.0.0:646 0.0.0.0:0 LISTEN
    0x1012e1b4 0x00000000 0 0 0.0.0.0:646 0.0.0.0:0 LISTEN
    0x10147944 0x60000000 0 0 0.0.0.0:179 0.0.0.0:0 LISTEN
    0x101369b4 0x00000000 0 0 0.0.0.0:179 0.0.0.0:0 LISTEN
    0x1015edb4 0x00000000 0 0 0.0.0.0:0 0.0.0.0:0 CLOSED

    ======================================================

    Thank you in advance for ur time.

    Regards,
    Prakash

    1. Do you have similar device terminating the L2VPN EVPN BGP session (ASR9006)? If it’s not an XR device you might have to add “prefix-length-size 2” under the L2VPN EVPN BGP neighbor in the other device.

      – Jesse

      1. Yes, I have two similar ASR9006s (with Typhoon cards and IOS-XR 5.1.1). I have replaced DC switches from ur topology with 10G Ixia testports only. However, I am getting “Invalid prefix received in update from 7.7.7.2” error message on debug like it was interop between IOS and IOS-XR case ( as you mention earlier) . Is there any thing missing to tuneup BGP ?
        Below is BGp debug output:

        RP/0/RSP0/CPU0:C9006_0001#debug bgp update
        Sun Nov 1 07:55:55.826 UTC
        RP/0/RSP0/CPU0:C9006_0001#show debug
        Sun Nov 1 07:56:01.957 UTC

        #### debug flags set from tty ‘vty0’ ####
        ip-bgp default update flag is ON with value ‘##########’

        RP/0/RSP0/CPU0:C9006_0001#RP/0/RSP0/CPU0:Nov 1 07:56:03.038 : bgp[1049]: [default-upd]: Allocating filter-group 0.3in TBL:default (25/70)
        RP/0/RSP0/CPU0:Nov 1 07:56:03.038 : bgp[1049]: [default-upd] (l2evpn): Added neighbor 7.7.7.2 to update filter-group 0.3 in L2VPN EVPN sub-group 0.1
        RP/0/RSP0/CPU0:Nov 1 07:56:03.038 : bgp[1049]: [default-upd] (l2evpn): Added neighbor 7.7.7.2 to update sub-group 0.1 in L2VPN EVPN update-group 0.2
        RP/0/RSP0/CPU0:Nov 1 07:56:03.038 : bgp[1049]: [default-upd] (l2evpn): Started updgrp timer for updgrp 0.2:: delay=0.010, delaytype=0
        RP/0/RSP0/CPU0:Nov 1 07:56:03.038 : bgp[1049]: [default-upd] (l2evpn): Adding neighbor 7.7.7.2 to new filter-group 0.3 in L2VPN EVPN sub-group 0.1 rtset size 0 in updgrp 0.2
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: UPDATE from 7.7.7.2 contains nh 7.7.7.2/32, gw_afi 0, flags 0x0, nlri_afi 14
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): ##!!##!!## Starting updgen walk for updgrp 0.2:: targetver=55780: tblver=55780, labelver=55780, minfwdver=55780, ackdfwdver=1, standbyver=55780
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): Computing updates for update sub-group 0.1 (Regular)
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: NH-Validate-Create: addr=7.7.7.2/32, len=4, nlriafi=14, nbr=7.7.7.2, gwafi=0, gwlen=4, gwaddrlen=32::: nhout=0x107b2a5c, validity=1, attrwdrflags=0x00000000
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] VRF ES:GLOBAL: table-attr walk for table TBL:ES:GLOBAL (25/70), resume version 0, subgrp version 0, target version 55780
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] VRF core_bdomain1_EVPN: table-attr walk for table TBL:core_bdomain1_EVPN (25/70), resume version 0, subgrp version 0, target version 55780
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr] (l2evpn): Received UPDATE from 7.7.7.2 with attributes:
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): ===UPDATE===: tbl=TBL:default (25/70), afi=14: ug=0.2, (Regular), pelem (Regular), sg=0.1, ugfl=0x00104183: bgpctxfl=0x01, tblctxfl=0x00001022, ltblctxfl=0x02000021, sendlab=0: net=v4Addr:7.7.7.1:1000:[2][0][48][0001.0002.2222][0]/104, nver=55780: PELEM=107c2f3c (lpathid=1, ver=55780, fl=0x00001001): PATH=0 (NULL,0.0.0.0,0, 0x0)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr] (l2evpn): nexthop 7.7.7.2/32, origin i, localpref 100, extended community RT:65100:1000
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr] (l2evpn): Received prefix v4Addr:7.7.7.2:1000:[2][0][48][0001.0002.2222][0]/104 (path ID: none) with MPLS label 16004 from neighbor 7.7.7.2
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): No unreachable (no path is available) sent to sub-group 0.1 (Regular) with v4Addr:7.7.7.1:1000:[2][0][48][0001.0002.2222][0]/104 – already withdrawn
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: –bgp4_rcv_attributes–: END: nbr=7.7.7.2:: msg=0x1003fe58/96, updlen=77, attrbl=0x1003fe6f/73, ipv4reachlen=0, msginpath=0x3f53be0, asloopcheck=1, attrwdrfl=0x00000000:: samecluster=0, myascount=0:: rcvdata=0x1003feb8/0, errptr=0x1003fead/11
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: Received UPDATE from 7.7.7.2 (length incl. header = 96)
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: ffff ffff ffff ffff ffff ffff ffff ffff
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): ===UPDATE===: tbl=TBL:default (25/70), afi=14: ug=0.2, (Regular), pelem (Regular), sg=0.1, ugfl=0x00104183: bgpctxfl=0x01, tblctxfl=0x00001022, ltblctxfl=0x02000021, sendlab=0: net=v4Addr:7.7.7.1:1000:[2][0][48][0001.0002.1111][4][0.0.0.0]/136, nver=2: PELEM=107c301c (lpathid=1, ver=2, fl=0x00000001): PATH=107a2fec (0.0.0.0/32,0.0.0.0,0, 0x400002000504000b)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: 0060 0200 0000 4990 0e00 2c00 1946 0407
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: 0707 0200 0221 0001 0707 0702 03e8 0000
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: 0000 0000 0000 0000 0000 0000 3000 0100
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: 0222 2200 03e8 4140 0101 0040 0200 4005
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): : tbl=TBL:default (25/70), afi=14: ug=0.2, sg=0.1, ugfl=0x00104183: net=v4Addr:7.7.7.1:1000:[2][0][48][0001.0002.1111][4][0.0.0.0]/136, PELEM=107c301c(lpathid=1, fl=0x00000001), PATH=107a2fec(0.0.0.0/32,0.0.0.0,0, 0x400002000504000b), reflected=0, bmsgfl=0x00000004, wdr=0::: netlab=1/16001, pathlab=1048577, updlab=16001(0x00003e81)
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: 0400 0000 64c0 1008 0002 fe4c 0000 03e8
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): Permit UPDATE to filter-group 0.3 (Regular, pelem Regular) for 0001.0002.1111 (changedfl=0x0/0x0), path
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: UPDATE from 7.7.7.2 contains nh 7.7.7.2/32, gw_afi 0, flags 0x0, nlri_afi 14
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: NH-Validate-Create: addr=7.7.7.2/32, len=4, nlriafi=14, nbr=7.7.7.2, gwafi=0, gwlen=4, gwaddrlen=32::: nhout=0x107b2a5c, validity=1, attrwdrflags=0x00000000
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): Sending UPDATE message(0x10047bac) to sub-group 0.1 (Regular, pelem Regular) for 0001.0002.1111 (changedfl=0x0/0x0) – creating new message with bmsgflags=0x00000004, attributes: nexthop 0.0.0.0, originator 0.0.0.0
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): origin i, path , localpref 100, extended community RT:65100:1000
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: –bgp4_rcv_attributes–: END: nbr=7.7.7.2:: msg=0x1003fcc0/111, updlen=92, attrbl=0x1003fcd7/88, ipv4reachlen=0, msginpath=0x3f53be0, asloopcheck=1, attrwdrfl=0x00000000:: samecluster=0, myascount=0:: rcvdata=0x1003fd2f/0, errptr=0x1003fd23/12
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): Created msg elem 0x1015e1e8 (pointing to message 0x10047bac), for filtergroup 0.3
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: !!!!!!!!!!!!!:: ERROR_HANDLING<> :: errfl=0x00000008, attrfl=0x00, attrcode=14, attrlen=0:: errhenabled=0, action=2:: MSG=10999154 (fl=0x00000008, finalaction=2, elemcount=1):: msglindex=1, msgcount=13927
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr] (l2evpn): Invalid prefix received in update from 7.7.7.2
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): ===UPDATE===: tbl=TBL:default (25/70), afi=14: ug=0.2, (Regular), pelem (Regular), sg=0.1, ugfl=0x00104183: bgpctxfl=0x01, tblctxfl=0x00001022, ltblctxfl=0x02000021, sendlab=0: net=v4Addr:7.7.7.1:1000:[3][2001][16][::ffff:7.7.7.1]/176, nver=4: PELEM=107c2fac (lpathid=1, ver=4, fl=0x00000001): PATH=107a2f4c (0.0.0.0/32,0.0.0.0,0, 0x400000000504000b)::: allowbe=0, isbe=0, allowspurwdr=0, pelem-send=1, pelem-wdr=0
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: Receive message dump for 7.7.7.2:
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-rtr]: !!-!-#-!-#-!!:: Received Malformed UPDATE from nbr 7.7.7.2:: msglen=111, errflags=0x00000008, finalaction=Reset:: elemcount=1, lastelem={“Error 0x00000008, Field “MP_REACH-NLRI”, Attribute 14 (Flags 0x00, Length 0), Data [03]”}:: ERRCTX={{msglindex=1, malformcount=13927, memfailcount=0}}
        RP/0/RSP0/CPU0:Nov 1 07:56:08.039 : bgp[1049]: [default-upd] (l2evpn): Permit UPDATE to filter-group 0.3 (Regular, pelem Regular) for v4Addr:7.7.7.1:1000:[3][2001][16][::ffff:7.7.7.1]/176 (changedfl=0x0/0x0), path

    1. Here is my OSPF and BGP config:

      *****************************ASR9K_0001******************************************

      !
      router ospf 2
      router-id 7.7.7.1
      passive enable
      mpls ldp sync
      address-family ipv4 unicast
      area 0
      interface Loopback10
      passive enable
      !
      interface TenGigE0/1/0/0
      passive disable
      !
      !
      !
      router bgp 100
      bgp router-id 7.7.7.1
      address-family l2vpn evpn
      !
      neighbor 7.7.7.2
      remote-as 100
      update-source Loopback10
      address-family l2vpn evpn
      !
      !
      *****************************ASR9K_0002******************************************
      router ospf 2
      router-id 7.7.7.2
      passive enable
      mpls ldp sync
      address-family ipv4 unicast
      area 0
      interface Loopback10
      passive enable
      !
      interface TenGigE0/1/0/0
      passive disable
      !
      !

      router bgp 100
      bgp router-id 7.7.7.2
      address-family l2vpn evpn
      !
      neighbor 7.7.7.1
      remote-as 100
      update-source Loopback10
      address-family l2vpn evpn
      !
      !
      !

  2. *****************************ASR9K_0001******************************************

    RP/0/RSP0/CPU0:C9006_0001#show bgp update in error detail
    Mon Nov 2 06:52:40.740 UTC

    VRF “default”
    Malformed Update messages: 1744
    Neighbors that received malformed Update messages: 1
    Last malformed update received: Nov 2 06:52:27.032 (00:00:13 ago)

    update-source Loopback10
    address-family l2vpn evpn
    !
    !

    *****************************ASR9K_0002******************************************

    RP/0/RSP0/CPU0:ios#show bgp update in error detail
    Tue Nov 3 06:44:01.824 UTC

    VRF “default”
    Malformed Update messages: 1572
    Neighbors that received malformed Update messages: 1
    Last malformed update received: Nov 3 06:42:29.428 (00:01:32 ago)

    RP/0/RSP0/CPU0:ios#show bgp update in error detail
    Tue Nov 3 06:44:10.200 UTC

    VRF “default”
    Malformed Update messages: 1573
    Neighbors that received malformed Update messages: 1
    Last malformed update received: Nov 3 06:44:08.647 (00:00:01 ago)

  3. Suomalaiset casino select a particular web site in accordance with the grapple stereo: perform not look like a game men and women, as washed out by other players’ recommendations. Nor will we believe it is significant the frequency of which the casino methods us featuring its advertising communication.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.