Splunk: eStreamer fails after FMC upgrade

For the last few days I was working on and off troubleshooting Splunk eStreamer client issue where application (App) was running but no IPS or File/Malware Events were populating panels.  Log files under  /opt/splunk/etc/apps/eStreamer/log were generated but for some reason App was not pulling them in. Eventually it would fail with Client Status – Error.

Components:
Cisco FMC: 6.1.0.3 
Splunk: 6.5.1

If I reinstall eStreamer it would run for a few minutes and stop updating again. Long story short downgrading Cisco Firepower Management Center (FMC) to version 6.1.2 fixed it but here’s some commands I found useful as I was narrowing down my issue.

On  FMC:

Check eStreamer port, it should be listen or established.

root@firepower:/Volume/home/root# netstat -an | grep 8302
tcp        0      0 172.16.1.1:8302      0.0.0.0:*               LISTEN      
tcp        0      0 172.16.1.1:8302      10.10.10.10:38736     ESTABLISHED
tcp        0      0 172.16.1.1:8302      10.10.10.10:38734     ESTABLISHED

If you just reinstalled eStreamer on Splunk connection may be hanged on FMC so good idea is to restart service.

root@firepower:/Volume/home/root# manage_estreamer.pl

Enter choice: 1
<SNIP>
sfestreamer (normal) – Running 21721
<SNIP>

Enter choice: 4

Reloading EStreamer… at /usr/local/sf/lib/perl/5.10.1/SF/EStreamer.pm line 594, <STDIN> line 3.
Restarted

Enter choice: 1

<SNIP>
sfestreamer (normal) – Running 21627
<SNIP>

Enter choice: 0

Check for running processes. You should see several of them.

root@firepower:~$ pstree -a | grep -i sfestreamer
  |   |-sfestreamer –nodaemon –sftunnel
  |   |   |-{sfestreamer}
<SNIP>
  |   |   `-{sfestreamer}
  |   |-sfestreamer –nodaemon
  |   |   |-sfestreamer –nodaemon
  |   |   |   |-{sfestreamer}
<SNIP>
  |   |   |   `-{sfestreamer}
  |   |   `-sfestreamer –nodaemon
  |   |       |-{sfestreamer}
<SNIP>
  |   |       `-{sfestreamer}
  |               |-grep -i sfestreamer
root@firepower:~$

Check for any errors in the log.

root@firepower:~$ grep -i estreamer /var/log/messages | grep -i error
May  3 04:13:44 firepower SF-IMS[6043]: [6080] Event Streamer:sfestreamer [ERROR] Unable to receive message: Closed
May  3 04:16:09 firepower SF-IMS[6043]: [17723] Event Streamer:sfestreamer [WARN] MessageSocket_Receive returned 20: General read error

On Splunk check eStreamer client has all the dependencies installed. When executed, script should not generate any errors.

[root@sim-01 bin]# ./estreamer_client.pl
Usage:  estreamer_client.pl [options]
Options:
        [-c]onfig=<config filename>
        [-l]ogfile=<log filename>
        [-t]est
        [-d]aemon

If you are checking eStreamer settings and forgot certificate password you can always retrieve it from CLI.

Under eStreamer local folder check estreamer.conf file.

[root@sim-01 local]# more estreamer.conf
[estreamer]
changed = 0
client_disabled = 0
log_extra_data = 1
log_flows = 1
pkcs12_file = /opt/splunk/etc/apps/eStreamer/bin/cert.pkcs12
pkcs12_password = password >> in clear text
server = 10.10.10.10

When all checked out fine I went with eStreamer App reinstall on Splunk. It fixed the issue for a few minutes and started failing again so I knew nothing on Splunk could of caused this. All of the findings above led me to believe that upgrade to 6.1.3 could of been the reason for eStreamer failures.  Once I downgraded to 6.1.2 eStreamer operation returned to normal.

Leave a reply:

Your email address will not be published.

Site Footer

Sliding Sidebar