For the last few days I was working on and off troubleshooting Splunk eStreamer client issue where application (App) was running but no IPS or File/Malware Events were populating panels. Log files under /opt/splunk/etc/apps/eStreamer/log were generated but for some reason App was not pulling them in. Eventually it would fail with Client Status – Error.
Components:
Cisco FMC: 6.1.0.3
Splunk: 6.5.1
If I reinstall eStreamer it would run for a few minutes and stop updating again. Long story short downgrading Cisco Firepower Management Center (FMC) to version 6.1.2 fixed it but here’s some commands I found useful as I was narrowing down my issue.
On FMC:
Check eStreamer port, it should be listen or established.
root@firepower:/Volume/home/root# netstat -an | grep 8302
tcp 0 0 172.16.1.1:8302 0.0.0.0:* LISTEN
tcp 0 0 172.16.1.1:8302 10.10.10.10:38736 ESTABLISHED
tcp 0 0 172.16.1.1:8302 10.10.10.10:38734 ESTABLISHED
If you just reinstalled eStreamer on Splunk connection may be hanged on FMC so good idea is to restart service.
root@firepower:/Volume/home/root# manage_estreamer.pl
Enter choice: 1
<SNIP>
sfestreamer (normal) – Running 21721
<SNIP>
Enter choice: 4
Reloading EStreamer… at /usr/local/sf/lib/perl/5.10.1/SF/EStreamer.pm line 594, <STDIN> line 3.
Restarted
Enter choice: 1
<SNIP>
sfestreamer (normal) – Running 21627
<SNIP>
Enter choice: 0
Check for running processes. You should see several of them.
root@firepower:~$ pstree -a | grep -i sfestreamer
| |-sfestreamer –nodaemon –sftunnel
| | |-{sfestreamer}
<SNIP>
| | `-{sfestreamer}
| |-sfestreamer –nodaemon
| | |-sfestreamer –nodaemon
| | | |-{sfestreamer}
<SNIP>
| | | `-{sfestreamer}
| | `-sfestreamer –nodaemon
| | |-{sfestreamer}
<SNIP>
| | `-{sfestreamer}
| |-grep -i sfestreamer
root@firepower:~$
Check for any errors in the log.
root@firepower:~$ grep -i estreamer /var/log/messages | grep -i error
May 3 04:13:44 firepower SF-IMS[6043]: [6080] Event Streamer:sfestreamer [ERROR] Unable to receive message: Closed
May 3 04:16:09 firepower SF-IMS[6043]: [17723] Event Streamer:sfestreamer [WARN] MessageSocket_Receive returned 20: General read error
On Splunk check eStreamer client has all the dependencies installed. When executed, script should not generate any errors.
[root@sim-01 bin]# ./estreamer_client.pl
Usage: estreamer_client.pl [options]
Options:
[-c]onfig=<config filename>
[-l]ogfile=<log filename>
[-t]est
[-d]aemon
If you are checking eStreamer settings and forgot certificate password you can always retrieve it from CLI.
Under eStreamer local folder check estreamer.conf file.
[root@sim-01 local]# more estreamer.conf
[estreamer]
changed = 0
client_disabled = 0
log_extra_data = 1
log_flows = 1
pkcs12_file = /opt/splunk/etc/apps/eStreamer/bin/cert.pkcs12
pkcs12_password = password >> in clear text
server = 10.10.10.10
When all checked out fine I went with eStreamer App reinstall on Splunk. It fixed the issue for a few minutes and started failing again so I knew nothing on Splunk could of caused this. All of the findings above led me to believe that upgrade to 6.1.3 could of been the reason for eStreamer failures. Once I downgraded to 6.1.2 eStreamer operation returned to normal.