Cisco ASA: High CPU utilization

Came across this issue where application performance was poor and pings were hitting 500ms. At first I blamed it on FirePOWER module and of course disabling it fixed the issue. However, when I looked at CPU utilization on ASA (with FirePOWER off the policy-map) it was still sitting between 70-80%. I’ve checked online and there are many articles describing different causes for high CPU on ASA but I do not think this one was covered.

Start with sh cpu usage.

asa# sh cpu usage

CPU utilization for 5 seconds = 75%; 1 minute: 73%; 5 minutes: 73%

This gives you a clear indication of a problem. Unfortunately, there is no cpu history command to go back in time.

Next check for non-zero CPU processes. As you can see my datapath was high – 30% and CP processing was at 16%.

asa# show processes cpu-usage sorted non-zero
Hardware: ASA5555
Cisco Adaptive Security Appliance Software Version 9.8(2)28
ASLR enabled, text region 7fe46627f000-7fe46a56b67c
PC Thread 5Sec 1Min 5Min Process
– – 31.6% 31.8% 31.7% DATAPATH-1-2277
– – 31.5% 31.7% 31.6% DATAPATH-0-2276
0x00007fe467079049 0x00007fe4050bd860 15.6% 15.7% 15.5% CP Processing
0x00007fe4686869bc 0x00007fe4050ce100 0.3% 0.3% 0.3% Logger
0x00007fe468947cb5 0x00007fe4050ae920 0.2% 0.1% 0.1% vpnfol_thread_timer
0x00007fe46752cc62 0x00007fe4050b88a0 0.1% 0.1% 0.1% ARP Thread

So I had 2 issues going-on. CP processing issue was related to duplicate configuration command in policy-map. In my example I had pptp inspection configured in 3 different policies and applied to 3 interfaces including global. So if you see this process high one place to check is you policy-map config.

Fixing that dropped my CPU utilization for CP process to 0.2

asa# sh processes cpu-usage non-zero sorted
Hardware: ASA5555
Cisco Adaptive Security Appliance Software Version 9.8(2)28
ASLR enabled, text region 7fe46627f000-7fe46a56b67c
PC Thread 5Sec 1Min 5Min Process
– – 27.9% 29.4% 30.0% DATAPATH-1-2277
– – 27.7% 29.3% 29.9% DATAPATH-0-2276
0x00007fe4686869bc 0x00007fe4050ce100 0.4% 0.4% 0.4% Logger
0x00007fe467079049 0x00007fe4050bd860 0.2% 1.6% 9.3% CP Processing
0x00007fe468947cb5 0x00007fe4050ae920 0.1% 0.1% 0.1% vpnfol_thread_timer
0x00007fe46752cc62 0x00007fe4050b88a0 0.1% 0.1% 0.1% ARP Thread
0x00007fe46746c15a 0x00007fe4050bf900 0.1% 0.0% 0.0% IKEv2 Daemon

But my total utilization was still at 60%

asa# sh cpu
CPU utilization for 5 seconds = 59%; 1 minute: 60%; 5 minutes: 69%

So next step is troubleshooting Datapath issues. First thing to check is you connection stats with show conn all command.

There will be a lot of data but the goal is to find connections with the most bytes. Pick a few and look for ingress/egress interfaces. If they are the same it may indicate a routing loop which is driving your CPU. To confirm validate destination network or do packet capture.

asa# capture 1 interface inside match udp host src_ip host dst_ip

asa# show cap 1 detail

On the packet capture TTL will be decreasing if it is a routing loop.

Once confirmed modify command to reflect the interface in question. If for example connection ingress/egress is inside interface then modify the command as show conn all | inc inside.*inside. Here is an example output.  1 is destination interface, 2 source interface, 3 huge number of bytes.

Copy output and paste it in Excel, do Text to Columns and sort by most bytes. I got a big list and started with highest talkers first. Then I’ve correlated destinations with valid subnets and found out that most of them did not exist. Traffic was following default route taking it to ASA, ASA in turn was sending it back to inside, because of RC 1812 routing, creating a routing loop.

At this point, you need to decide how to fix it. Ideally, you need to go to the source and remove the invalid configuration. You will have destinations and ports which should be enough to pinpoint misconfigured application. In my case it was Syslog and SNMP traffic. If it is not possible then for invalid routes you can do Null0 for specific host or subnet similar to below.

route Null0 10.x.x.x 255.255.255.255 1

After that, you need to clear connection to invalid IP and check again to make sure it did not reappear.

clear conn address10.x.x.x

As soon as the connection was cleared my CPU dropped to 32%.