How to setup a MAC address routing peering, in a nutshell. This is the a simple network and protocol setup using FRRouting daemon. The netlab is found here in previous blog entry and is build using freeRtr SR ISIS EVPN iBGP-IX-peering-fabric.
The network topology and idea is based on the modernizing-IXP-design article, written by Phil Bedard on xrdocs.io.
This is a blind try out what is possible using a free router implementation, at this point I have no clue what can be configured and how and to what degree.
I need to put here some information before. This configuration uses IS-IS and SR MPLS. Everything is based on IPv4 only. The segment routing in this network topology will get interesting as soon as the TI-LFA and path protection will be computed based on the network topology and the available information.
The fast convergence setup part here is only important for the IS-IS, underlay. BFD is not used for BGP.
This configuration does not explain any dynamic routing protocol basics. You need to be familiar with all dynamic routing protocols, setup and operation.
The appliance used in example is available in Building 64bit alpine linux GNS3 FRRouting appliance.
Network topology
The FRRouting appliances have 9 ports allocated eth0 - eth8. 4 ethernet ports per GNS3 appliance are sufficient to build shown example, but then it is fixed and no routers could be added at all.
- P1, P2
- PE11, PE12, PE13, PE14
Network topology with IP addressing:
lo lo lo
192.0.2.11 192.0.2.1 192.0.2.13
+-------+ +-------+ +-------+
+0 7+------------+1 3+------------+7 0+
| PE11 | | P1 | | PE13 |
| 8+-+ +-------+2 4+-+ +-------+8 |
+-------+ | | +-------+ | | +-------+
| | | |
+--(-----+ +--(-----+
| | | |
+-------+ | | +-------+ | | +-------+
| 7+----+ +-+1 2+----+ +-+7 |
| PE12 | | P2 | | PE14 |
+0 8+------------+2 3+------------+8 0+
+-------+ +-------+ +-------+
lo lo lo
192.0.2.12 192.0.2.2 192.0.2.14
The interface link IP addressing between the routers in the topology are point-to-point /30.
The appliance physical port number is f.e.: +7 for eth7
Configuration overview
Building a L2 MAC address BGP EVPN routing IX-fabric using 6 FRR routers. The routers will forward MAC address only, there is no IP interface involved or SVI. Only Layer2 ethernet MAC addresses. The solution works like the switch or bridge at your home.
ALL topology routers use following protocols:
- IPv4
- IS-IS
- SR MPLS
- MPLS-TE
PE router only:
- BGP
- EVPN
- VXLAN
- ethernet bridge
If there are any useless configuration steps that could be removed from the setup to keep it as simple as possible, write me an email. It will be removed. Learning BGP EVPN configuration and writing this blog entry about it.
The FRRouting version used for the setup:
FRRouting 9.1 (P1) on Linux(6.6.16-0-virt).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
Configuration
Not all used and configured networking protocols for the router setup can be done using the vtysh command line.
The PE router configuration having, linux VXLAN and linux bridge interface configuration needs to be done using iproute2, using the linux shell. After the configuration is finished in bash or your shell, the according interfaces are availble in vtysh command line. For showing statistics of a given interface. This is a complex and lengthy setup routine.
IS-IS process
Specific configuration settings are set to following:
- The FRR IS-IS
lsp-timer gen-interval 1setting
This is for fast-convergence on startup of the router. Using default FRR IS-IS timers the router adjacency forms immediately, on given interface, but the router waits 30 seconds to converge. It takes the router 30 seconds to be forward traffic. 30 seconds
- No IS-IS keychain or encryption configured.
This is nice to have here, left it out to not make the the FRR configuration even more complex on top with optional feature.
- The IOS-XR interface command
adjacency-sid absolute 15002 protected
This command is not available at interface level using FRR version 9.1.
Not sure here how this could be configured using FRR. An idea would probably using a PCE (Path Computation Element. A PCE is a external router that has knowledge of the whole topology and can pre-calculate the paths and configure the SR network aftewards usind the PCEP protocol. This could be a additional FRRouting appliance
- The IOS-XR command
prefix-sid absolute 16041
Configuring absolute SR label value works using FRR, but then follwing syslog messages appear:
Mar 1 14:44:46 P2 daemon.warn isisd[2369]: [YHD7P-BK2Z9][EC 67108867] tilfa_compute_label_stack: SID index 16012 falls outside remote SRGB range
Mar 1 14:44:46 P2 daemon.warn isisd[2369]: [J8AXP-8H71T] tilfa_repair_list_apply: IP TE 192.0.2.12/32 adjacency PE11: failed to compute label stack
Mar 1 14:44:46 P2 daemon.warn isisd[2369]: [PNNS0-WQHRP] ISIS-LFA: failed to compute repair path(s) of IP TE 192.0.2.12/32 w.r.t PE2.0's failure (link protection)
The SR topology works fully, or specifically the ISIS-LFA calculation works, when the absolute setting is changed to a index command instead:
segment-route prefix 192.0.2.1/32 index 1
The router prefix will have the label id 16001 using that index setting in the shown configuration.
IOS-XR template
This is the IOS-XR configuration shown in blog entry modernizing-IXP-design.
SRLB must not overlap with the SRGB. The SRGB and SRLB are configured in IOS-XR with the following configuration:
segment-routing
segment-routing
global-block 16000 16999
local-block 17000 17999
And this is the IOS-XR IS-IS IPv4 configuration:
router isis example
set-overload-bit on-startup wait-for-bgp
is-type level-2-only
net 49.0002.1921.6801.4003.00
distribute link-state
log adjacency changes
log pdu drops
lsp-refresh-interval 65000
max-lsp-lifetime 65535
lsp-password keychain ISIS-KEY
address-family ipv4 unicast
maximum-paths 16
metric-style wide
mpls traffic-eng level-2-only
mpls traffic-eng router-id Loopback0
maximum-paths 32
segment-routing mpls
!
interface Loopback0
passive
address-family ipv4 unicast
metric 10
prefix-sid absolute 16041
!
!
!
interface GigabitEthernet0/0/0/1
circuit-type level-2-only
point-to-point
address-family ipv4 unicast
fast-reroute per- fast-reroute per-prefix ti-lfa
metric 10
adjacency-sid absolute 15002 protected
FRR
The IS-IS underlay network assures the reachability of the loopback lo interfaces. This is achieved by using and configuring following protocols on all routers:
lointerface IP from the loopback network -192.0.2.x/24- IS-IS (NET:
49.0001.xxxx.xxxx.xxxx.00) - BFD on physical routed IS-IS interface
ethX
- BFD on physical routed IS-IS interface
- SR MPLS
segment-routing mpls on
- SR MPLS
- MPLS-TE
m̀pls-te router-id 192.0.2.x
- MPLS-TE
- TI-LFA - a post-convergence backup path is pre-calculated from the given network topology and settings using SPF calculation
P1
IOS-XR configuration applied to FRRouting daemon, most configuration entries are available, most of the configuration can be
!P1
config
!
hostname P1
!
interface eth1
mpls enable
isis metric 10
isis fast-reroute ti-lfa
ip address 10.0.0.1/30
ip router isis 1
isis network point-to-point
exit
!
interface eth2
mpls enable
isis metric 10
isis fast-reroute ti-lfa
ip address 10.0.0.5/30
ip router isis 1
isis network point-to-point
exit
!
interface eth3
mpls enable
isis metric 10
isis fast-reroute ti-lfa
ip address 10.0.0.9/30
ip router isis 1
isis network point-to-point
exit
!
interface eth4
mpls enable
isis metric 10
isis fast-reroute ti-lfa
ip address 10.0.0.13/30
ip router isis 1
isis network point-to-point
exit
!
interface lo
isis metric 10
ip address 192.0.2.1/32
ip router isis 1
isis passive
exit
!
router isis 1
is-type level-2-only
net 49.0001.1111.1111.1110.00
set-overload-bit on-startup 20
lsp-timers gen-interval 1 refresh-interval 65235 max-lifetime 65535
spf-interval 2
log-adjacency-changes
log-pdu-drops
max-lsp-lifetime 65535
lsp-refresh-interval 65235
metric-style wide
!
mpls-te on
mpls-te router-address 192.0.2.1
!
segment-routing on
segment-routing global-block 16000 16999 local-block 17000 17999
segment-routing prefix 192.0.2.1/32 index 1
end
wr
PE11
This is PE1 configuration:
!PE11
config
!
hostname PE11
!
interface eth7
mpls enable
isis metric 10
isis fast-reroute ti-lfa
ip address 10.0.0.2/30
ip router isis 1
isis network point-to-point
exit
!
interface eth8
mpls enable
isis metric 10
isis fast-reroute ti-lfa
ip address 10.0.0.18/30
ip router isis 1
isis network point-to-point
exit
!
interface lo
isis metric 10
ip address 192.0.2.11/32
ip router isis 1
isis passive
exit
!
router isis 1
is-type level-2-only
net 49.0001.1111.1111.1111.00
log-adjacency-changes
log-pdu-drops
max-lsp-lifetime 65535
lsp-refresh-interval 65235
metric-style wide
!
mpls-te on
mpls-te router-address 192.0.2.11
!
segment-routing on
segment-routing global-block 16000 16999 local-block 17000 17999
segment-routing prefix 192.0.2.11/32 index 11
end
wr
IS-IS configuration for all routers in the network topology:
BFD
Separted the BFD configuration. The BFD configuration has certain default settings. Display used defaults using the
show bfd peers
This command shows a full of detail output displaying the currently used default settings. That is the output of P1 router using FRR BFD default configuration settings:
P1# show bfd peers
BFD Peers:
peer 10.0.0.6 local-address 10.0.0.5 vrf default interface eth2
ID: 1061577166
Remote ID: 3130140265
Active mode
Status: up
Uptime: 2 hour(s), 2 minute(s), 56 second(s)
Diagnostics: ok
Remote diagnostics: ok
Peer Type: dynamic
RTT min/avg/max: 0/0/0 usec
Local timers:
Detect-multiplier: 3
Receive interval: 300ms
Transmission interval: 300ms
Echo receive interval: 50ms
Echo transmission interval: disabled
Remote timers:
Detect-multiplier: 3
Receive interval: 300ms
Transmission interval: 300ms
Echo receive interval: 50ms
The interesting setting are the Local timers and Remote timers default settings. Since the settings are unified in this solution, displaying only the Localtimers the Remote timers have the same setting
| Description | Parameter |
|---|---|
| Detect Multiplier | 3 |
| Receive interval | 300ms |
| Trasmission interval | 300ms |
| Echo receive interval | 50ms |
| Echo trasmission interfall | disabled |
To make the solution shown more comparable in terms of measuring common convergence statistics.
The freeRtr SR ISIS EVPN iBGP IX-fabric article uses following settings, these are the freeRtr example defaults:
interface ethernet1.1
ipv4 bfd 100 100 3
Table overview:
| Description | Parameter |
|---|---|
| Detect Multiplier | 3 |
| Receive interval | 100ms |
| Trasmission interval | 100ms |
BFD profile
This is the best matching FRR profile named P2P, setting the BFD timers to the freeRtr example defaults used. This BFD profile is named P2P, it will be applied only on directly connected point-to-point links only. This BFD profile will be used for IS-IS sessions only.
Create the BFD profile on every router:
configure
!
bfd
profile P2P
transmit-interval 100
receive-interval 100
echo transmit-interval 100
echo receive-interval 100
end
wr
P router
Now the profile has been created bind the BFD profile to each point-to-point IS-IS interface
!P routers
config
int eth1
isis bfd
isis bfd profile P2P
int eth2
isis bfd
isis bfd profile P2P
int eth3
isis bfd
isis bfd profile P2P
int eth4
isis bfd
isis bfd profile P2P
end
wr
PE router
Now the profile has been created bind the BFD profile to each point-to-point IS-IS interface
!PE routers
config
int eth7
isis bfd
isis bfd profile P2P
int eth8
isis bfd
isis bfd profile P2P
end
wr
BFD profile verification
After applied settings the P2P profile BFD timers applied have to match the outupt below:
P1# show bfd peers
BFD Peers:
peer 10.0.0.2 local-address 10.0.0.1 vrf default interface eth1
ID: 1563819205
Remote ID: 527232675
Active mode
Status: up
Uptime: 20 second(s)
Diagnostics: ok
Remote diagnostics: ok
Peer Type: dynamic
RTT min/avg/max: 0/0/0 usec
Local timers:
Detect-multiplier: 3
Receive interval: 100ms
Transmission interval: 100ms
Echo receive interval: 100ms
Echo transmission interval: disabled
Remote timers:
Detect-multiplier: 3
Receive interval: 100ms
Transmission interval: 100ms
Echo receive interval: 100ms
Now the BFD timers are matching the freeRtr example defaults.
Verification
Use following verification steps to inspect the configuration settings and show the network protocol state.
show ip route
This is to verify if the lo interfaces 192.0.2.x/32. Notify the b backup paths in the routing table. This entry will only show if ti-lfa is configured on the interface level. This also only works in MPLS networks, so this is stricly tied to the SR MPLS implementation. Using IPv4 only with IS-IS, not sure it would calculate backup paths. Did not bother investigating this or trying out in a netlab, but I am pretty sure there are no backup paths on plain IPv4 and IS-IS.
P
Here the P1 routers IP routing table, in a converged state. Notice prefixes and the b backup routes with attached labels.
P1#show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
I 10.0.0.0/30 [115/20] via 10.0.0.2, eth1 inactive, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.6, eth2, label 16002, weight 1
b via 10.0.0.10, eth3, label 16002, weight 1
b via 10.0.0.14, eth4, label 16002, weight 1
C>* 10.0.0.0/30 is directly connected, eth1, 00:17:29
I 10.0.0.4/30 [115/20] via 10.0.0.6, eth2 inactive, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, label 16002, weight 1
b via 10.0.0.10, eth3, label 16002, weight 1
b via 10.0.0.14, eth4, label 16002, weight 1
C>* 10.0.0.4/30 is directly connected, eth2, 00:17:29
I 10.0.0.8/30 [115/20] via 10.0.0.10, eth3 inactive, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, label 16002, weight 1
b via 10.0.0.6, eth2, label 16002, weight 1
b via 10.0.0.14, eth4, label 16002, weight 1
C>* 10.0.0.8/30 is directly connected, eth3, 00:17:29
I 10.0.0.12/30 [115/20] via 10.0.0.14, eth4 inactive, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, label 16002, weight 1
b via 10.0.0.6, eth2, label 16002, weight 1
b via 10.0.0.10, eth3, label 16002, weight 1
C>* 10.0.0.12/30 is directly connected, eth4, 00:17:29
I>* 10.0.0.16/30 [115/20] via 10.0.0.2, eth1, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.6, eth2, weight 1
b via 10.0.0.10, eth3, weight 1
b via 10.0.0.14, eth4, weight 1
I>* 10.0.0.20/30 [115/20] via 10.0.0.6, eth2, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, weight 1
b via 10.0.0.10, eth3, weight 1
b via 10.0.0.14, eth4, weight 1
I>* 10.0.0.24/30 [115/20] via 10.0.0.10, eth3, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, weight 1
b via 10.0.0.6, eth2, weight 1
b via 10.0.0.14, eth4, weight 1
I>* 10.0.0.28/30 [115/20] via 10.0.0.14, eth4, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, weight 1
b via 10.0.0.6, eth2, weight 1
b via 10.0.0.10, eth3, weight 1
C>* 192.0.2.1/32 is directly connected, lo, 00:17:29
I>* 192.0.2.2/32 [115/30] via 10.0.0.2, eth1, label 16002, weight 1, 00:17:27
* via 10.0.0.6, eth2, label 16002, weight 1, 00:17:27
* via 10.0.0.10, eth3, label 16002, weight 1, 00:17:27
* via 10.0.0.14, eth4, label 16002, weight 1, 00:17:27
I>* 192.0.2.11/32 [115/20] via 10.0.0.2, eth1, label implicit-null, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.6, eth2, label 16002/16011, weight 1
b via 10.0.0.10, eth3, label 16002/16011, weight 1
b via 10.0.0.14, eth4, label 16002/16011, weight 1
I>* 192.0.2.12/32 [115/20] via 10.0.0.6, eth2, label implicit-null, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, label 16002/16012, weight 1
b via 10.0.0.10, eth3, label 16002/16012, weight 1
b via 10.0.0.14, eth4, label 16002/16012, weight 1
I>* 192.0.2.13/32 [115/20] via 10.0.0.10, eth3, label implicit-null, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, label 16002/16013, weight 1
b via 10.0.0.6, eth2, label 16002/16013, weight 1
b via 10.0.0.14, eth4, label 16002/16013, weight 1
I>* 192.0.2.14/32 [115/20] via 10.0.0.14, eth4, label implicit-null, weight 1, backup 0,1,2, 00:17:24
b via 10.0.0.2, eth1, label 16002/16014, weight 1
b via 10.0.0.6, eth2, label 16002/16014, weight 1
b via 10.0.0.10, eth3, label 16002/16014, weight
The IP routing table is pretty hudge. The routing table now not only has the current forwarding state, but also a pre-calculated backup paths how to reach a certain IP prefix, interesting are only the /32 loopback prefixes only.
The 192.0.2.2/32 index 2 is reachable from P1 router only using the label 16002
P1# show ip route 192.0.2.2
Routing entry for 192.0.2.2/32
Known via "isis", distance 115, metric 30, best
Last update 05:32:36 ago
* 10.0.0.2, via eth1, label 16002, weight 1
* 10.0.0.6, via eth2, label 16002, weight 1
* 10.0.0.10, via eth3, label 16002, weight 1
* 10.0.0.14, via eth4, label 16002, weight 1
And this is how linux iproute2 displays the ip same routing table for the 192.0.2.2/32, showing the attached label to the prefix entry:
user@host ~ ip route show 192.0.2.2/32
192.0.2.2 nhid 215 proto isis metric 20
nexthop encap mpls 16002 via 10.0.0.6 dev eth2 weight 1
nexthop encap mpls 16002 via 10.0.0.2 dev eth1 weight 1
nexthop encap mpls 16002 via 10.0.0.14 dev eth4 weight 1
nexthop encap mpls 16002 via 10.0.0.10 dev eth3 weight 1
This is one the reason why route is deprecated in linux. The linux net-tools are not able to deal with the linux-6.x IP routing table implementation at this scale. net-tools will not show IP MPLS packets and its labels at all, not even the isis routing protocol. Try it out.
PE device
Show ip route on the PE device
PE11# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
I 10.0.0.0/30 [115/20] via 10.0.0.1, eth7 inactive, weight 1, backup 0, 00:21:04
b via 10.0.0.17, eth8, label 16012, weight 1
C>* 10.0.0.0/30 is directly connected, eth7, 00:27:18
I>* 10.0.0.4/30 [115/20] via 10.0.0.1, eth7, weight 1, backup 0, 00:21:04
b via 10.0.0.17, eth8, weight 1
I>* 10.0.0.8/30 [115/20] via 10.0.0.1, eth7, weight 1, backup 0, 00:21:04
b via 10.0.0.17, eth8, weight 1
I>* 10.0.0.12/30 [115/20] via 10.0.0.1, eth7, weight 1, backup 0, 00:21:04
b via 10.0.0.17, eth8, weight 1
I 10.0.0.16/30 [115/20] via 10.0.0.17, eth8 inactive, weight 1, backup 0, 00:21:04
b via 10.0.0.1, eth7, label 16012, weight 1
C>* 10.0.0.16/30 is directly connected, eth8, 00:27:18
I>* 10.0.0.20/30 [115/20] via 10.0.0.17, eth8, weight 1, backup 0, 00:21:04
b via 10.0.0.1, eth7, weight 1
I>* 10.0.0.24/30 [115/20] via 10.0.0.17, eth8, weight 1, backup 0, 00:21:04
b via 10.0.0.1, eth7, weight 1
I>* 10.0.0.28/30 [115/20] via 10.0.0.17, eth8, weight 1, backup 0, 00:21:04
b via 10.0.0.1, eth7, weight 1
I>* 192.0.2.1/32 [115/20] via 10.0.0.1, eth7, label implicit-null, weight 1, backup 0, 00:21:04
b via 10.0.0.17, eth8, label 16012/16001, weight 1
I>* 192.0.2.2/32 [115/20] via 10.0.0.17, eth8, label implicit-null, weight 1, backup 0, 00:21:04
b via 10.0.0.1, eth7, label 16012/16002, weight 1
C>* 192.0.2.11/32 is directly connected, lo, 00:27:18
I>* 192.0.2.12/32 [115/30] via 10.0.0.1, eth7, label 16012, weight 1, 00:21:04
* via 10.0.0.17, eth8, label 16012, weight 1, 00:21:04
I>* 192.0.2.13/32 [115/30] via 10.0.0.1, eth7, label 16013, weight 1, 00:21:04
* via 10.0.0.17, eth8, label 16013, weight 1, 00:21:04
I>* 192.0.2.14/32 [115/30] via 10.0.0.1, eth7, label 16014, weight 1, 00:21:04
* via 10.0.0.17, eth8, label 16014, weight 1, 00:21:04
show isis database
The show isis database output from the PE11 router:
PE11# show isis database
Area 1:
IS-IS Level-2 link-state database:
LSP ID PduLen SeqNumber Chksum Holdtime ATT/P/OL
P1.00-00 263 0x00000010 0xa672 61220 0/0/0
PE11.00-00 * 190 0x00000003 0x0bd2 61313 0/0/0
PE12.00-00 190 0x00000007 0x3f83 61200 0/0/0
PE13.00-00 176 0x00000007 0x56f4 62810 0/0/0
PE14.00-00 190 0x00000007 0xb7dc 61874 0/0/0
P2.00-00 263 0x00000011 0x9992 61485 0/0/0
6 LSPs
For detailed information use following command.
show isis databse detail
This command will list the kitchen sink output. All available and configured used options. Very extensive and lengthy output. Will not show it here.
Example command output from the P1 IS-IS router database.
SR Segment Routing
To have TI-LFA fully working the isis mpls-te setting must be set. To see the difference in computing backup paths the output below shows
Verification
Using this command shows the previously configured global SRGB and SRLB label block. The output shows each node's configuration.
P1# show isis segment-routing node
Area 1:
IS-IS L1 SR-Nodes:
IS-IS L2 SR-Nodes:
System ID SRGB SRLB Algorithm MSD
--------------------------------------------------------------
1111.1111.1110 16000 - 16999 17000 - 17999 SPF 0
1111.1111.1111 16000 - 16999 17000 - 17999 SPF 0
1111.1111.1112 16000 - 16999 17000 - 17999 SPF 0
1111.1111.1113 16000 - 16999 17000 - 17999 SPF 0
1111.1111.1114 16000 - 16999 17000 - 17999 SPF 0
2222.2222.2220 16000 - 16999 17000 - 17999 SPF 0
show isis mpls-te
show isis mpls-te router
This command shows if the router-id is set, here in example it is the lo IPv4 address
P1# show isis mpls-te router
Area 1:
MPLS-TE IPv4 Router-Address: 192.0.2.1
show isis fast-reroute summary
This is fast reroute showing classified topology, with half working SR. Here MPLS-TE is not working.
P1# show isis fast-reroute summary
Area 1:
IS-IS L2 IPv4 Fast ReRoute summary:
Protection \ Priority Critical High Medium Low Total
--------------------------------------------------------------------------
Classic LFA 0 0 0 0 0
Remote LFA 0 0 0 0 0
Topology Independent LFA 0 0 0 4 4
ECMP 0 0 1 0 1
Unprotected 0 0 4 4 8
Protection coverage 0.00% 0.00% 20.00% 50.00% 38.46%
Once MPLS-TE is configured accordingly the output for TI-LFA is different:
P1# show isis fast-reroute summary
Area 1:
IS-IS L2 IPv4 Fast ReRoute summary:
Protection \ Priority Critical High Medium Low Total
---------------------------------------------------------------------------
Classic LFA 0 0 0 0 0
Remote LFA 0 0 0 0 0
Topology Independent LFA 0 0 4 8 12
ECMP 0 0 1 0 1
Unprotected 0 0 0 0 0
Protection coverage 0.00% 0.00% 100.00% 100.00% 100.00%
This is from the point of view of the PE device w/o MPLS-TE
PE1# show isis fast-reroute summary
Area 1:
IS-IS L2 IPv4 Fast ReRoute summary:
Protection \ Priority Critical High Medium Low Total
--------------------------------------------------------------------------
Classic LFA 0 0 0 0 0
Remote LFA 0 0 0 0 0
Topology Independent LFA 0 0 0 6 6
ECMP 0 0 3 0 3
Unprotected 0 0 2 2 4
Protection coverage 0.00% 0.00% 60.00% 75.00% 69.23%
This is the calculation after the MPLS-TE option is configured correctly:
PE1# show isis fast-reroute summary
Area 1:
IS-IS L2 IPv4 Fast ReRoute summary:
Protection \ Priority Critical High Medium Low Total
---------------------------------------------------------------------------
Classic LFA 0 0 0 0 0
Remote LFA 0 0 0 0 0
Topology Independent LFA 0 0 2 8 10
ECMP 0 0 3 0 3
Unprotected 0 0 0 0 0
Protection coverage 0.00% 0.00% 100.00% 100.00% 100.00%
linux
linux setup steps only for PE devices in the topology. The iproute2 commands shown are mandatory prior to setup.
- create
bridge100interface - create
vxlan100interface - assign
eth0port on PE router tobridge100
One information on that $loopback variable used in the script. Each router has a /32 assigned IPv4 address on the linux lo interface. This is the router ID AND the primary in-band management address of that particular router.
VXLAN
FRRouting shows this extensive example for various needs
Cherry picked few command out of big example guide to make L2 VXLAN only, one bridge.
From the CLI create following script.
- add
vxlan100 - assings VXLAN to bridge
br100
Creating startup script:
cat > /etc/local.d/00_VXLAN.start
#!/bin/bash
LOOPBACK=$(ip a s lo | grep 32 | awk '{print $4}')
VNIS="100"
for VNI in $VNIS; do
ip link add vxlan${VNI} type vxlan id ${VNI} dstport 4789 local $LOOPBACK nolearning
brctl addbr br${VNI};
brctl addif br${VNI} vxlan${VNI};
brctl stp br${VNI} off;
ip link set up dev br${VNI};
ip link set up dev vxlan${VNI};
done
Script runs after the FRR daemons are started on bootup sequence of the appliance. The lo interface IP address 192.0.2.x/32 is set up at this point.
bridge
Assigning the physical port eth0 of the PE router to a bridge. This needs to be done in a separate script. If scripts run to quickly on startup linux fails creating according interfaces, or more specific, probably tries to execute command as quickly as possible and at given time some settings are still not in place. Reason to run a separate script for assigning the interface to the bridge:
cat > /etc/local.d/01_eth0-br100.start
#!/bin/sh
VNIS="100"
for VNI in $VNIS; do
ip link set eth0 master br${VNI};
done
There is no way yet to add a port to a bridge in vtysh. FRR is a routing daemon in first place and ethernet or L2 related commands are not FRR's primary function area.
Adjust both scripts to be executable:
chmod +x /etc/local.d/00_VXLAN.start
chmod +x /etc/local.d/01_eth0-br100.start
Verify the files setting:
ls /etc/local.d/
Add the /etc/local.d/ scripts to startup routine:
rc-update add local default
At this state the eth0 port is always assigned to bridge100, on a reboot.
All PE routers are now prepared.
BGP
The overlay network routes MAC addresses using BGP EVPN address family. The iBGP sessions rely on lo IP being reachable all the time.
- bridge (
br100) - VXLAN (
vxlan100) - BGP (
AS 65000) - EVPN (
address-family evpn)
- EVPN (
The example has one bridge domain br100, and one VXLAN interface vxlan100.
Instead of using a BGP route reflector, the topology uses iBGP mesh instead, removing the need of a functional router for the topology. It is a fixed small network topology. Removing configuration complexity here.
This is handling the overlay network. And works on top of the configured IS-IS network topology. The underlay network guarantees reachability and fast convergence for the lo IP addresses.
According to the latest documentation only 2 commands are needed to enable BGP EVPN, from the documentation example:
configure
!
router bgp 65001
!
address-family l2vpn evn
adversite-all-vni
end
!
wr
Everything else is defining a template called IBGP with 2 defaults and binding to its peers
configure
!
router bgp 65000
!
neighbor IBGP peer-group
neigibor IBGP remote-as 65000
neighbor IBGP update-source lo
!
address-family ipv4 unicast
neighbor 192.0.2.12 peer-group IBGP
neighbor 192.0.2.13 peer-group IBGP
neighbor 192.0.2.14 peer-group IBGP
!
neighbor 192.0.2.12 activate
neighbor 192.0.2.13 activate
neighbor 192.0.2.14 activate
end
!
wr
Tested setting up BFD for iBGP, but that just slowed down the convergence time, and disrupted the iBGP session. Without using BFD on BGP the BGP sessions are stable, and the routing converges without disrupting additionally the iBGP sessions.
The overlay network adds the BGP EVPN - MAC address routing capability. This is the result configration. Even if you configure address-family ipv4 unicast and its BGP neighbor entries, the resulting and shown FRRouting configuration will not explicitly list the settings. Some things are implied in FRR and not shown if matching some defaults:
Example configuraiton for PE1 router:
configure
!
router bgp 65000
bgp router-id 192.0.2.11
neighbor IBGP peer-group
neighbor IBGP remote-as 65000
neighbor IBGP update-source lo
neighbor 192.0.2.12 peer-group IBGP
neighbor 192.0.2.13 peer-group IBGP
neighbor 192.0.2.14 peer-group IBGP
!
address-family l2vpn evpn
neighbor IBGP activate
advertise-all-vni
exit-address-family
exit
end
wr
BGP configuration for all PE routers:
Verify
show ip bgp summary
PE1# show ip bgp summary
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.11, local AS number 65000 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 5, using 66 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.0.2.12 4 65000 66 68 0 0 0 01:02:08 0 0 FRRouting/9.1
192.0.2.13 4 65000 66 68 0 0 0 01:02:10 0 0 FRRouting/9.1
192.0.2.14 4 65000 68 68 0 0 0 01:02:08 0 0 FRRouting/9.1
Total number of neighbors 3
One can use any other command here to like f.e.:
show bgp l2vpn evpn summary
show bgp summary
show bgp peer-group
This shows the internal Peer-group and the configured BGP address-families to BGP neighbors L2VPN EVPN:
PE1# show bgp peer-group
BGP peer-group IBGP, remote AS 65000
Peer-group type is internal
Configured address-families: IPv4 Unicast; L2VPN EVPN;
Peer-group members:
192.0.2.12 Established
192.0.2.13 Established
192.0.2.14 Established
show bgp l2vpn evpn vni
This command shows the (VNI) -VXLAN Network Identifier (or VXLAN Segment ID), notice the L2 VNI entry in the command line output:
PE1# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Enabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 1
Number of L3 VNIs: 0
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF
* 100 L2 192.0.2.11:2 65000:100 65000:100 default
show bgp l2vpn evpn route detail
The BGP kitchen-sink output command, showing all information using one command:
PE4# show bgp l2vpn evpn route detail
Route Distinguisher: 192.0.2.11:2
BGP routing table entry for 192.0.2.11:2:[3]:[0]:[32]:[192.0.2.11]
Paths: (1 available, best #1)
Not advertised to any peer
Route [3]:[0]:[32]:[192.0.2.11]
Local
192.0.2.11 (metric 30) from 192.0.2.11 (192.0.2.11)
Origin IGP, localpref 100, valid, internal, best (First path received)
Extended Community: RT:65000:100 ET:8
Last update: Sun Mar 3 21:50:25 2024
PMSI Tunnel Type: Ingress Replication, label: 100
Route Distinguisher: 192.0.2.12:2
BGP routing table entry for 192.0.2.12:2:[3]:[0]:[32]:[192.0.2.12]
Paths: (1 available, best #1)
Not advertised to any peer
Route [3]:[0]:[32]:[192.0.2.12]
Local
192.0.2.12 (metric 30) from 192.0.2.12 (192.0.2.12)
Origin IGP, localpref 100, valid, internal, best (First path received)
Extended Community: RT:65000:100 ET:8
Last update: Sun Mar 3 21:50:25 2024
PMSI Tunnel Type: Ingress Replication, label: 100
Route Distinguisher: 192.0.2.13:2
BGP routing table entry for 192.0.2.13:2:[3]:[0]:[32]:[192.0.2.13]
Paths: (1 available, best #1)
Not advertised to any peer
Route [3]:[0]:[32]:[192.0.2.13]
Local
192.0.2.13 (metric 30) from 192.0.2.13 (192.0.2.13)
Origin IGP, localpref 100, valid, internal, best (First path received)
Extended Community: RT:65000:100 ET:8
Last update: Sun Mar 3 21:50:23 2024
PMSI Tunnel Type: Ingress Replication, label: 100
Route Distinguisher: 192.0.2.14:2
BGP routing table entry for 192.0.2.14:2:[3]:[0]:[32]:[192.0.2.14]
Paths: (1 available, best #1)
Advertised to non peer-group peers:
192.0.2.11 192.0.2.12 192.0.2.13
Route [3]:[0]:[32]:[192.0.2.14] VNI 100
Local
192.0.2.14 from 0.0.0.0 (192.0.2.14)
Origin IGP, weight 32768, valid, sourced, local, best (First path receive)
Extended Community: ET:8 RT:65000:100
Last update: Sun Mar 3 21:50:17 2024
PMSI Tunnel Type: Ingress Replication, label: 100
Displayed 4 prefixes (4 paths)
Tried out configure bgp multipath but this configuration command is available only with bgp address-family ipv4 unicast
show interface vxlan100
Although the VXLAN interface creation is done by iproute2 linux tool. The interface vxlan100 can be displayed using vtysh:
PE1# show interface vxlan100
Interface vxlan100 is up, line protocol is up
Link ups: 1 last: 2024/02/22 21:35:05.18
Link downs: 2 last: 2024/02/22 21:35:05.16
vrf: default
index 11 metric 0 mtu 1500 speed 4294967295 txqlen 1000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: a2:40:c4:40:9e:93
inet6 fe80::a040:c4ff:fe40:9e93/64
Interface Type Vxlan
Interface Slave Type Bridge
VTEP IP: 192.0.2.11 VxLAN Id 100 Access VLAN Id 1
Master interface: br100
protodown: off
show interface br100
This is the output for the configured br100 bridge interface as seen shown by vtysh.
PE1# sh int br100
Interface br100 is up, line protocol is up
Link ups: 2 last: 2024/02/22 21:35:05.19
Link downs: 3 last: 2024/02/22 21:35:05.17
vrf: default
index 12 metric 0 mtu 1500 speed 1000 txqlen 1000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 0c:e1:95:86:00:00
inet6 fe80::a040:c4ff:fe40:9e93/64
Interface Type Bridge
Interface Slave Type None
Bridge VLAN-aware: no
protodown: off
It is interesting FRR shows the bridge as a interface. Not show br100, it is show interface br100 command.
show evpn mac vni all
This is simple command to display all MAC addresses that are currently routed across the network. Command output as shown on the PE1 router:
PE1# show evpn mac vni all
VNI 100 #MACs (local and remote) 4
Flags: N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
aa:bb:cc:00:01:00 local eth0 0/0
aa:bb:cc:00:02:00 remote 192.0.2.12 0/0
aa:bb:cc:00:04:00 remote 192.0.2.14 0/0
aa:bb:cc:00:03:00 remote 192.0.2.13 0/0
show evpn vni detail
All (VNI) - VXLAN Segment ID details. Notice the Number of MACs (local and remote) known for this VNI: 4 entry in the command line output:
PE1# show evpn vni detail
VNI: 100
Type: L2
Vlan: 1
Bridge: br100
Tenant VRF: default
VxLAN interface: vxlan100
VxLAN ifIndex: 11
SVI interface: br100
SVI ifIndex: 12
Local VTEP IP: 192.0.2.11
Mcast group: 0.0.0.0
Remote VTEPs for this VNI:
192.0.2.12 flood: HER
192.0.2.13 flood: HER
192.0.2.14 flood: HER
Number of MACs (local and remote) known for this VNI: 2
Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 0
Advertise-gw-macip: No
Advertise-svi-macip: No
show isis route prefix-sid
This output shows basically the is-is database. This output matches the show ip route output
P1# show isis route prefix-sid
Area 1:
IS-IS paths to level-2 routers that speak IP
Vertex Type Metric Next-Hop Interface Parent
P1
10.0.0.0/30 IP internal 0 P1(4)
10.0.0.4/30 IP internal 0 P1(4)
10.0.0.8/30 IP internal 0 P1(4)
10.0.0.12/30 IP internal 0 P1(4)
192.0.2.1/32 IP internal 0 P1(4)
PE1 TE-IS 10 PE1 eth1 P1(4)
PE2 TE-IS 10 PE2 eth2 P1(4)
PE3 TE-IS 10 PE3 eth3 P1(4)
PE4 TE-IS 10 PE4 eth4 P1(4)
P2 TE-IS 20 PE1 eth1 PE1(4)
PE2 eth2 PE2(4)
PE3 eth3 PE3(4)
PE4 eth4 PE4(4)
10.0.0.0/30 IP TE 20 PE1 eth1 PE1(4)
10.0.0.16/30 IP TE 20 PE1 eth1 PE1(4)
192.0.2.11/32 IP TE 20 PE1 eth1 PE1(4)
10.0.0.4/30 IP TE 20 PE2 eth2 PE2(4)
10.0.0.20/30 IP TE 20 PE2 eth2 PE2(4)
192.0.2.12/32 IP TE 20 PE2 eth2 PE2(4)
10.0.0.8/30 IP TE 20 PE3 eth3 PE3(4)
10.0.0.24/30 IP TE 20 PE3 eth3 PE3(4)
192.0.2.13/32 IP TE 20 PE3 eth3 PE3(4)
10.0.0.12/30 IP TE 20 PE4 eth4 PE4(4)
10.0.0.28/30 IP TE 20 PE4 eth4 PE4(4)
192.0.2.14/32 IP TE 20 PE4 eth4 PE4(4)
192.0.2.2/32 IP TE 30 PE1 eth1 P2(4)
PE2 eth2
PE3 eth3
PE4 eth4
IS-IS L2 IPv4 routing table:
Prefix Metric Interface Nexthop SID Label Op. Algo
----------------------------------------------------------------------
10.0.0.0/30 20 eth1 10.0.0.2 - - 0
10.0.0.4/30 20 eth2 10.0.0.6 - - 0
10.0.0.8/30 20 eth3 10.0.0.10 - - 0
10.0.0.12/30 20 eth4 10.0.0.14 - - 0
10.0.0.16/30 20 eth1 10.0.0.2 - - 0
10.0.0.20/30 20 eth2 10.0.0.6 - - 0
10.0.0.24/30 20 eth3 10.0.0.10 - - 0
10.0.0.28/30 20 eth4 10.0.0.14 - - 0
192.0.2.2/32 30 eth1 10.0.0.2 16002 no-op. 0
eth2 10.0.0.6 16002 no-op. 0
eth3 10.0.0.10 16002 no-op. 0
eth4 10.0.0.14 16002 no-op. 0
192.0.2.11/32 20 eth1 10.0.0.2 16011 Pop(16011) 0
192.0.2.12/32 20 eth2 10.0.0.6 16012 Pop(16012) 0
192.0.2.13/32 20 eth3 10.0.0.10 16013 Pop(16013) 0
192.0.2.14/32 20 eth4 10.0.0.14 16014 Pop(16014) 0
At this point it is proved that FRRouting implementation is good enough to be tested on real hardware. In the netlab it is converging fast, almost without losing IP packets.
Convergence test
This test is identical approach like described in the
This is a random ICMP test for IPv4. The CE routers, here CE1 - CE4. are connected to the PE routers. For the CE routers perspective the network topology looks like this:
+-----+ +-----+ +-----+ +-----+
| | | | | | | |
| CE1 | | CE2 | | CE3 | | CE4 |
| | | | | | | |
+--0--+ +--0--+ +--0--+ +--0--+
| | | |
0-----------0-----------0-----------0
192.168.1.0/29
The CE routers are connected to eth0 PE interface. That interface is in bridge-group 100.
For the test, on a fully converged network topology. The test ideally would run like that:
1) The P1 router is powered off (power outage simulation) 2) Result the IX peering network converges
ICMP echo for the interface IP on the subnet assigned to each CE on the eth0 interface 192.168.1.0/29 IP subnet. All CE routers are running OSPFv2 and the interface setting is to ip ospf point-to-multipoint for IPv4 and IPv6, identical setting. However the interface IP address is directly connected.
Results while one of the P routers, here P1 is reloaded, CE1 pinging the eth0 interface of CE4:
CE1#ping 192.168.1.4 repeat 10000
Success rate is 99 percent (9998/10000), round-trip min/avg/max = 1/1/7 ms
The P1 router starting up and converging again is like watching - SIA Stucked In Active. The startup routine is stucked on starting zebra daemon each time. This is how it is seen from the P1 router while CE routers are doing ICMP or ICMPv6 tests:
[...]
* Seeding random number generator ... * Seeding 256 bits and crediting
* Saving 256 bits of creditable seed for next boot
[ ok ]
* Starting busybox syslog ... [ ok ]
* Starting busybox acpid ... [ ok ]
* Starting busybox crond ... [ ok ]
Starting watchfrr with command: ' /usr/lib/frr/watchfrr -d -F traditional '
[<----long timeout here---->]
Started watchfrr
* Starting sshd ... [ ok ]
P1 login: zebra (automatic login)
Welcome to FRRouting Alpine!
Hello, this is FRRouting (version 9.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
P1# show ip route
zebra is not running
The last entry is important zebra is not started. The watchfrr command runs into a timeout routine , possibly.
The startup routine is stucked for watchfrr while there is traffic passing the fabric interfaces. Possibly it time outs and looks like the most important daemon zebra is not started.
Found older bug issue#10826.
The startup routine works if there is no IP traffic passing the interfaces from the CE side, zebra starts and everything depending on it starts too.
Also this solution does not work reliable every time in GNS3. Sometimes yes sometimes, FRR routing routers need to be reloaded a 2-nd time then it works, for all routers.
Tried without BFD and the result is the same, watchfrr times out and zebra is not started, if there is IP traffic from the CE router across the network.
The issue is reproducible almost each time. I need to lookup how it could be narrowed down on the github issue bug list, there are useful hints in the bug reports how to troubleshoot issues in FRR.
See also
Try setup following FRR GNS3 netlabs:
- Building 64bit alpine linux GNS3 FRRouting appliance
- FRRouting basic IS-IS router configuration
- FRRouting openfabric tech demo
- FRRouting SR Segment Routing tech demo
- FRRouting IS-IS Segment Routing tech demo