I have discovered a interesting default behaviour on a Nexus 7000 Router while troubleshooting. The Cisco Nexus 7000 switches (in this case a router), seem to drop ICMP echo replies.
Pinging from a directy connected IOS/IOS-XE device:
S1#ping 192.0.2.1 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 192.0.2.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (998/1000), round-trip min/avg/max = 1/3/24 ms
You always want to have Success rate is 100 percent if devices are directly connected.
If pinging a target IOS/IOS-XE device over a routed network the result looks like this:
S1#ping 198.51.100.250 repeat 1000 size 1500 df-bit
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to 198.51.100.250, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/4/20 ms
Success rate is 100 percent with the df-bit and the maximum packet size. That is called a reliable link.
- No packetloss if pinging over a routed network
- Packetloss on directly connected devices
On the NX-OS router pinging the directly connected S1 switch. S1 has the IP 192.0.2.10.
R1# ping S1 count 1000
...
64 bytes from 192.0.2.10: icmp_seq=995 ttl=254 time=1.396 ms
64 bytes from 192.0.2.10: icmp_seq=996 ttl=254 time=2.188 ms
64 bytes from 192.0.2.10: icmp_seq=997 ttl=254 time=1.339 ms
64 bytes from 192.0.2.10: icmp_seq=998 ttl=254 time=1.352 ms
64 bytes from 192.0.2.10: icmp_seq=999 ttl=254 time=1.333 ms
--- S1 ping statistics ---
1000 packets transmitted, 996 packets received, 0.40% packet loss
round-trip min/avg/max = 1.014/1.786/17.317 ms
This link is not reliable enough. It looks like something is blocking the traffic.
And it is true. The documentation reveals a default security feature called CoPP (Control Plane Policing).
R1# sh copp status
Last Config Operation: None
Last Config Operation Timestamp: None
Last Config Operation Status: None
Policy-map attached to the control-plane: copp-system-p-policy-strict
It says the control plane has a policy-map attached to it. Strict or policy-stict. Looking deeper reveals:
R1# sh policy-map type ?
control-plane Control-Plane
network-qos Type network-qos
qos Type qos
queuing Type queuing
...4 different types of policy-maps are configured. The one we are interested in is the control-plane policy map:
R1# sh policy-map type control-plane
...
class copp-system-p-class-monitoring
set cos 1
police cir 130 kbps bc 1000 ms
conform transmit violate drop
...
And here a detailed output:
R1# sh policy-map type control-plane expand
...
class copp-system-p-class-monitoring
match access-group name copp-system-p-acl-icmp
match access-group name copp-system-p-acl-icmp6
match access-group name copp-system-p-acl-mpls-oam
match access-group name copp-system-p-acl-traceroute
match access-group name copp-system-p-acl-http-response
match access-group name copp-system-p-acl-smtp-response
match access-group name copp-system-p-acl-http6-response
match access-group name copp-system-p-acl-smtp6-response
set cos 1
police cir 130 kbps bc 1000 ms
conform transmit violate drop
...
There is a class named copp-system-p-class-monitoring This particular class applies to following ACL's f.e:
- copp-system-p-acl-icmp
- copp-system-p-acl-icmp6
And some other well known protocols listed.
The policy map assignes the matched traffic (ICMP,HTTP,SMTP...) as class 1. This policy map has a Commited information rate (CIR) of 130 kbps. If the ICMP Traffic exceeds this threshhold the traffic will be dropped.
Hence the Quest is over.
We found a configuration snipplett for rate-limits incoming/outgoing traffic.
Limits are applied to modules 3-6:
R1# sh policy-map interface Control-plane | b monitoring
class-map copp-system-p-class-monitoring (match-any)
match access-group name copp-system-p-acl-icmp
match access-group name copp-system-p-acl-icmp6
match access-group name copp-system-p-acl-mpls-oam
match access-group name copp-system-p-acl-traceroute
match access-group name copp-system-p-acl-http-response
match access-group name copp-system-p-acl-smtp-response
match access-group name copp-system-p-acl-http6-response
match access-group name copp-system-p-acl-smtp6-response
set cos 1
police cir 130 kbps bc 1000 ms
conform action: transmit
violate action: drop
module 3:
conformed 194465448 bytes,
5-min offered rate 16 bytes/sec
peak rate 80 bytes/sec at Sun Dec 11 07:59:43 2016
violated 0 bytes,
5-min violate rate 0 bytes/sec
peak rate 0 bytes/sec
module 4:
conformed 2019193 bytes,
5-min offered rate 0 bytes/sec
peak rate 550 bytes/sec at Thu Dec 29 14:29:46 2016
violated 510 bytes,
5-min violate rate 0 bytes/sec
peak rate 1 bytes/sec at Thu Dec 29 14:29:46 2016
module 5:
conformed 0 bytes,
5-min offered rate 0 bytes/sec
peak rate 0 bytes/sec
violated 0 bytes,
5-min violate rate 0 bytes/sec
peak rate 0 bytes/sec
module 6:
conformed 275859216 bytes,
5-min offered rate 13 bytes/sec
peak rate 505 bytes/sec at Sun Oct 02 09:14:32 2016
violated 64 bytes,
5-min violate rate 0 bytes/sec
peak rate 0 bytes/sec
In the example above the drop policy has been hit on Thu Dec 29 14:29:46 2016 to module 4.
NX-OS cisco devices have default settings preconfigured to protect the control plane.
Further documentation about NX-OS CoPP feature is found here:
oneliner to see the troublemakers:
R1#show policy-map interface control-plane | i "class|conform|violated"