Splunk and Cisco ISE: Parsing fragmented data

Cisco ISE has huge reporting section but only a few of the reports are useful to me. So the need came up for a flexible searching and reporting tool. Splunk seemed to be a great candidate especially since it has all the plugins available and Cisco Security Suite App adds  great value to the product. Manuals and How-To guides are also available and it does not take much effort to get data going. My setup was simple: Splunk single instance configured as Remote Logging Target over TCP on ISE and Maximum Length set to 8192. At first, data looked ok, but when I started to dig deeper into pre-build queries I noticed not all the data fields were populated. For example, I’ve seen messages similar to the one below. Step data was obviously part of the fragmented message and needed to be fixed.

Broken ISE log message on Splunk

Cisco ISE: 2.0
Splunk: 6.2.3

Instead of modifying message size on Splunk we’ve followed Splunk Best Practice and built a dedicated Syslog server with Splunk Forwarder. I did not have much luck with Syslog server running on Windows OS so I’d recommend Linux OS and  rsyslog for Syslog service as it is easier to setup.

The following recommendations may help anyone without deep knowledge of Linux or Splunk to setup rsyslog and Splunk Forwarder on Linux OS.

First, configure remote logging on ISE under Administration > System > Logging > Remote Logging Targets > Add

Configure Remote Logging Target IP as rsyslog server. Set Maximum Length to 8192.

Add newly created logging target under Logging Categories and Save.

Now lets move onto rsyslog server configuration.

Once you have your IP address and DNS configured temporary disable any firewalls

systemctl stop firewalld
systemctl mask firewalld
service iptables status

service iptables stop
service ip6tbles stop
chkconfig iptables off
chkconfig ip6tables off

Turn off selinux to avoid issues with log file creation

vi /etc/selinux/config

Reboot the box.

Install NTP service to ensure accurate time stamp recording

yum install ntpd
yum install ntpdate

Install rsyslog

yum search rsyslog
yum install rsyslog
service rsyslog start
chkconfig rsyslog on

Confirm port 514 is open for listening with netstat command. Install it if necessary.

yum install net-tools
netstat -pan | grep 514

Modify rsyslog.conf configuration file

vi /etc/rsyslog.conf

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

#write ISE logs to file, make sure “contains” clause matches part of the Cisco ISE PSN hostname
if ($fromhost contains ‘ISE’) then /logs/ISE.log
#encrease MAX message size to 16k
$MaxMessageSize 16k

Restart rsyslog
service rsyslog restart

Download and install Splunk forwarder

rpm -Uvh splunkforwarder-6.3.3-f44afce176d0-linux-2.6-x86_64.rpm
cd /opt/splunkforwarder/bin
./splunk start
./splunk enable boot-start

Edit inputs.conf to read local log file

vi  /opt/splunkforwarder/etc/system/local/inputs.conf

disabled = false
sourcetype = cisco:ise:syslog
crcSalt = <SOURCE>

Forward log data to the indexer. Login credentials are default admin/changeme

./splunk add forward-server <indexer IP>:9997

Restart Splunk Forwarder service

cd /opt/splunkforwarder/bin/
./splunk restart

Verify Receiving data configuration is setup on Indexer for port TCP/9997 under Setting > Forwarding and Receiving > Receive Data > Configure Receiving

Splunk Indexer Receiving data configurationRun search on Splunk and make sure you are receiving messages from new Universal Forwarder.

eventtype=cisco-ise MESSAGE_CLASS=”Passed-Authentication” SelectedAuthorizationProfiles!=NULL

Full messages will end with LicenseTypes=””, the host will be your Linux rsyslog server hostname and source will be the file you are logging ISE messages to.

ISE log proper



Leave a reply:

Your email address will not be published.

Site Footer

Sliding Sidebar