Splunk and Certs

Splunk and Certs

I inherited a Splunk installation and haven't spent as much time with the search functions as our development teams has, but I've seen it do some really cool stuff. My role is pretty much to make sure that it stays up and to help folks when they think something is broken.

This installation probably has a lot more moving parts than it really needs and I only spend a little time on it, so it's not unusual to get a surprise from it every now and then.

Finding out that data that wasn't getting indexed was Splunk's way of telling me that I had some expired certs. In *our* infrastructure we use a lot of intermediate forwarders, and I'm thinking that the reason for that was that our indexers stored data on NFS. That's not something that Splunk support endorses, and something that we changed (we're now running on block storage) but we're still left with 7 different environments and a total 12 heavy forwarders (several in HA configurations) that had expiring certs.

Each forwarder has cert called forwarder.pem and ours expired expired on 3/2018
STEP 1.a
lgs06: Expiration check
find /splunk/etc/deployment-apps/ -name forwarder.pem | xargs -I crt sh -c 'ls crt; openssl x509 -in crt -text -noout | grep "Not After"'
STEP 1.b
lgs06: Expiration fix
/splunk/bin/splunk createssl server-cert -d /splunk/etc/auth/ -n forwarder -p
password is Jostens123
Copy the file to each of the apps that require it. You found them in STEP 1.a
cp /splunk/etc/auth/forwarder.pem <app_location/certs>
STEP 1.c
lgs06:/deployment-apps/*/certs/cacert.pem forwarder.pem
Contents of each app's cacert.pem must match the last paragraph of the NEW forwarders.pem
/splunk/bin/splunk reload deploy-server
splunk admin name: admin
splunk admin pass: <redacted>


STEP 2.a
Heavy Forwarders
Expiration check.
openssl x509 -in owbsleplgf01.pem -text -noout | grep "Not After"
STEP 2.b
Heavy Forwarders
Expiration fix
/opt/splunkforwarder/bin/splunk createssl server-cert -d /opt/splunkforwarder/etc/auth/ -n <forwarderhostname> -p
cp /opt/splunkforwarder/etc/auth/<forwarderhostname> /opt/splunkforwarder/etc/certs
Contents of /opt/splunkforwarder/etc/certs/cacert.pem must match the last paragraph of the NEW forwarder.pem from STEP 1.b
STEP 2.c
Heavy Forwarders
Replace the hashed  password in /opt/splunkforwarder/etc/system/local/inputs.conf
Before: password = <encryptedpassword>
After: password =<redacted>
service splunk restart
Troubleshooting:
Deployment server permissions issues? chown -R splunk:splunk /splunk/etc/deployment-apps
Deployments not getting to the hosts? Make sure /opt/splunkforwarder/etc/system/local/deploymentclient.conf isn't disabled

Done!