Troubleshooting MPLS

So today I got a complaint from one of the customers, what is wrong with the MPLS link? The services are very slow….So here are just a few commands you can use to find out what is happening.

Perhaps there is a log Error

G00-7800-01#sh log
Syslog logging: enabled (11 messages dropped, 1 messages rate-limited,
0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.
Console logging: level debugging, 93 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level warnings, 9 messages logged, xml disabled,
filtering disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled

No active filter modules.

Trap logging: level informational, 98 message lines logged
Logging to 10.10.10.42 (udp port 514, audit disabled,
authentication disabled, encryption disabled, link up),
98 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging to 10.78.0.13 (udp port 514, audit disabled,
authentication disabled, encryption disabled, link up),
98 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging to 10.0.129.172 (udp port 514, audit disabled,
authentication disabled, encryption disabled, link up),
98 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled

Log Buffer (51200 bytes):

*Sep 5 17:31:36.871: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Sep 5 17:31:36.871: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
*Sep 5 17:31:36.871: %LINK-3-UPDOWN: Interface Special-Services-Engine1/0, changed state to up
Sep 11 11:53:42.249: %BGP-3-NOTIFICATION: received from neighbor 10.129.78.10 4/0 (hold time expired) 0 bytes
Sep 15 21:18:08.768: %BGP-3-NOTIFICATION: sent to neighbor 10.78.239.252 4/0 (hold time expired) 0 bytes
Sep 16 10:10:52.235: %BGP-3-NOTIFICATION: sent to neighbor 10.78.239.252 4/0 (hold time expired) 0 bytes
Sep 21 15:10:35.881: %BGP-3-NOTIFICATION: sent to neighbor 10.78.239.252 4/0 (hold time expired) 0 bytes
Sep 21 15:30:42.604: %BGP-3-NOTIFICATION: sent to neighbor 10.78.239.252 4/0 (hold time expired) 0 bytes
Sep 25 08:26:36.959: %BGP-3-NOTIFICATION: sent to neighbor 10.129.78.10 4/0 (hold time expired) 0 bytes

In my case, there seems to be nothing wrong. The Link is up. The last time the link was down was on the 25th. Next, who are the top talkers

G00-7800-01#sh ip flow top-talkers

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Tu200 10.2.0.251 Fa0/1.127* 10.78.2.130 06 1F4E 051C 53K
Tu200 10.2.0.251 Fa0/1.127 10.78.2.130 06 1F4E 051C 53K
Fa0/1.127 10.78.2.130 Tu200* 10.2.0.251 06 051C 1F4E 30K
Fa0/1.127 10.78.2.130 Tu200 10.2.0.251 06 051C 1F4E 30K
Tu200 10.2.0.251 Fa0/1.127* 10.78.0.204 06 1F4E 054F 29K
Tu200 10.2.0.251 Fa0/1.127 10.78.0.204 06 1F4E 054F 29K
Tu200 10.63.0.251 Fa0/1.127 10.78.0.197 06 1F4E 05AA 17K
Tu200 10.63.0.251 Fa0/1.127* 10.78.0.197 06 1F4E 05AA 17K
Tu200 10.1.5.12 Fa0/1.127 10.78.0.13 06 0016 6E0E 17K
Tu200 10.1.5.12 Fa0/1.127* 10.78.0.13 06 0016 6E0E 17K
10 of 10 top talkers shown. 604 flows processed.

Check the CPU Processes

G00-7800-01# sh processes cpu
CPU utilization for five seconds: 29%/27%; one minute: 29%; five minutes: 25%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 205088 370412 553 0.00% 0.00% 0.00% 0 Chunk Manager
2 55024 441836 124 0.08% 0.05% 0.06% 0 Load Meter
4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN
5 3318216 275219 12056 0.16% 0.12% 0.12% 0 Check heaps
6 1528 1811 843 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 64 36821 1 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
9 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
10 4496 2208916 2 0.00% 0.00% 0.00% 0 IPC Periodic Tim
11 3388 2208915 1 0.08% 0.00% 0.00% 0 IPC Deferred Por
12 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
13 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure
14 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
15 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
16 2436 73640 33 0.00% 0.00% 0.00% 0 Environmental mo
17 64124 159281 402 0.00% 0.00% 0.00% 0 ARP Input
18 7784 2303964 3 0.00% 0.00% 0.00% 0 ARP Background
19 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
20 0 3 0 0.00% 0.00% 0.00% 0 AAA high-capacit
21 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
22 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
23 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers
24 4 2 2000 0.00% 0.00% 0.00% 0 Entity MIB API
25 12 56 214 0.00% 0.00% 0.00% 0 EEM ED Syslog
26 4136 660916 6 0.00% 0.00% 0.00% 0 HC Counter Timer

Another helpfull command

G00-7800-01# show ip cache flow
IP packet size distribution (1079M total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .467 .030 .027 .102 .031 .067 .007 .005 .004 .008 .003 .002 .001 .003

512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.002 .001 .052 .014 .164 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
673 active, 3423 inactive, 61352192 added
896261273 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 50376 bytes
667 active, 1381 inactive, 60910634 added, 60910634 added to flow
0 alloc failures, 793 force free
2 chunks, 626 chunks added
last clearing of statistics never
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-Telnet 1146 0.0 19 88 0.0 6.1 9.1
TCP-FTP 458 0.0 9 48 0.0 1.5 4.5
TCP-FTPD 218 0.0 2917 594 0.2 17.5 1.8
TCP-WWW 891290 0.4 35 668 14.2 1.9 5.7
TCP-SMTP 39193 0.0 272 778 4.8 11.3 3.0
TCP-X 40 0.0 10 146 0.0 0.4 5.1
TCP-BGP 54530 0.0 1 49 0.0 2.7 15.6
TCP-Frag 1302 0.0 1 113 0.0 0.0 15.5
TCP-other 43644130 19.7 20 375 396.9 2.3 10.7
UDP-DNS 4657500 2.1 1 64 2.1 0.1 15.5
UDP-NTP 167134 0.0 1 78 0.0 0.0 15.4
UDP-TFTP 96 0.0 1 63 0.0 1.3 15.4
UDP-Frag 4262 0.0 1 92 0.0 1.8 15.4
UDP-other 8872913 4.0 16 92 67.2 1.4 15.5
ICMP 2755772 1.2 2 127 2.5 1.2 15.4
GRE 1552 0.0 212 781 0.1 18.2 15.5
IP-other 10 0.0 41555 621 0.1 18.5 15.5
Total: 61091546 27.6 17 347 488.6 2.0 11.9

Next, Check the MPLS Link. I have no drops – so the link seems fine.

G00-7800-01#sh policy-map int fa0/0 out
FastEthernet0/0

Service-policy output: OutVPN10MParent

Class-map: class-default (match-any)
119049004 packets, 44505653563 bytes
5 minute offered rate 1738000 bps, drop rate 0 bps
Match: any
Traffic Shaping
Target/Average Byte Sustain Excess Interval Increment
Rate Limit bits/int bits/int (ms) (bytes)
10000000/10000000 62500 250000 250000 25 31250

Adapt Queue Packets Bytes Packets Bytes Shaping
Active Depth Delayed Delayed Active
- 0 119049012 1555980705 1123276 1301408861 no

Service-policy : OutVPN10M

Class-map: VoIP (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: ip precedence 5
Queueing
Strict Priority
Output Queue: Conversation 264
Bandwidth 640 (kbps) Burst 16000 (Bytes)
(pkts matched/bytes matched) 0/0
(total drops/bytes drops) 0/0

Class-map: Video-Control (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: ip precedence 3
Queueing
Output Queue: Conversation 265
Bandwidth 1840 (kbps)Max Threshold 64 (packets)
(pkts matched/bytes matched) 0/0
(depth/total drops/no-buffer drops) 0/0/0

Class-map: Management-Marked (match-all)
20944081 packets, 5287845346 bytes
5 minute offered rate 37000 bps, drop rate 0 bps
Match: ip precedence 2
Queueing
Output Queue: Conversation 266
Bandwidth 200 (kbps)Max Threshold 64 (packets)
(pkts matched/bytes matched) 19018/4881708
(depth/total drops/no-buffer drops) 0/0/0

Class-map: Business-Marked (match-all)
97396867 packets, 38718492227 bytes
5 minute offered rate 1694000 bps, drop rate 0 bps
Match: ip precedence 1
Queueing
Output Queue: Conversation 267
Bandwidth 7300 (kbps)Max Threshold 64 (packets)
(pkts matched/bytes matched) 1001342/1149468806
(depth/total drops/no-buffer drops) 0/0/0

Class-map: class-default (match-any)
708064 packets, 499315018 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: any

G00-7800-01#sh policy-map int fa0/0 inp
FastEthernet0/0

Service-policy input: InVPNTTK

Class-map: SPRemark2 (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: ip precedence 2
QoS Set
precedence 3
Packets marked 0

Class-map: class-default (match-any)
239556069 packets, 129058341808 bytes
5 minute offered rate 3149000 bps, drop rate 0 bps
Match: any

G00-7800-01#sh policy-map int fa0/1.127 inp
FastEthernet0/1.127

Service-policy input: FromLAN

Class-map: VVoIP (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: access-group name Video_VoIP
police:
cir 40500000 bps, bc 1265625 bytes, be 1500 bytes
conformed 0 packets, 0 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
drop
violated 0 packets, 0 bytes; actions:
drop
conformed 0 bps, exceed 0 bps, violate 0 bps

Class-map: Management (match-any)
41220006 packets, 7802063674 bytes
5 minute offered rate 26000 bps, drop rate 0 bps
Match: protocol ssh
39522603 packets, 7207967840 bytes
5 minute rate 26000 bps
Match: protocol telnet
5349 packets, 850936 bytes
5 minute rate 0 bps
Match: protocol syslog
625727 packets, 115980273 bytes
5 minute rate 0 bps
Match: protocol ntp
95313 packets, 9071722 bytes
5 minute rate 0 bps
Match: protocol snmp
962468 packets, 467474491 bytes
5 minute rate 0 bps
Match: access-group name TACACS-server
8545 packets, 719054 bytes
5 minute rate 0 bps
QoS Set
precedence 2
Packets marked 41220006

Class-map: Business (match-all)
217813705 packets, 71424022222 bytes
5 minute offered rate 1303000 bps, drop rate 0 bps
Match: access-group name LocalTraffic
QoS Set
precedence 1
Packets marked 217813709

Class-map: class-default (match-any)
45867169 packets, 7989029279 bytes
5 minute offered rate 129000 bps, drop rate 0 bps
Match: any
QoS Set
precedence 0
Packets marked 45867171

Since everything looks fine, I  decided to follow up on the top talkers to find out what applications are running.