Our storage team wanted to get some input from the powermt command periodically inserted into Splunk so that they would be able to run alerts against that data.
[root@entltdbb02:apps]$ powermt display
VNX logical device count=142
XtremIO logical device count=13
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
4 iSCSI Initiator over TCP/I optimal 142 0 - 0 2218
6 iSCSI Initiator over TCP/I optimal 142 0 - 6 0
8 iSCSI Initiator over TCP/I optimal 142 0 - 7 0
10 iSCSI Initiator over TCP/I optimal 142 0 - 2 1367
19 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
20 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
21 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
22 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
23 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
24 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
25 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
26 iSCSI Initiator over TCP/I optimal 13 0 - 0 0
There were 6 systems that required this alert to be run against them, and here's what I did to make that happen. This is not the most elegant solution, but it was quick and effective enough.
1. Create an app on the deployment server that contains basic app settings, a script, instructions on when to run the script and how to manage the output.
2. Create a class of servers to let Splunk know which servers to include in the app.
3. Enable the Splunk user to run the command on the target hosts via sudo.
Step 1 details:
1.a) Create the app structure;
mkdir /splunk/etc/deployment-apps/powermt/
mkdir /splunk/etc/deployment-apps/powermt/bin/
mkdir /splunk/etc/deployment-apps/powermt/local/
mkdir /splunk/etc/deployment-apps/powermt/metadata/
1b.) Write out basic app settings
cat /splunk/etc/deployment-apps/powermt/local/app.conf
[default]
[ui]
is_visible = false
[install]
state = enabled
1c.) Create a script to run the command in the bin directory. cat /splunk/etc/deployment-apps/powermt/bin/powermt.sh
#!/bin/bash
sudo powermt display
1d.) Write local/inputs.conf to describe what to run and when to run it. Note that the script location references its ultimate destination on the host.
cat /splunk/etc/deployment-apps/local/inputs.conf
##### Powermount scripted Inputs ######
[script:///opt/splunkforwarder/etc/apps/powermt/bin/powermt.sh]
## Run every 15 minutes
disabled = false
interval = 900
source = powermt
sourcetype = script:///opt/splunkforwarder/etc/apps/powermt/bin/powermt.sh]
1e.) Write local/props.conf to record the time for the script event cat /splunk/etc/deployment-apps/powermt/local/props.conf
[powermt]
TZ = America/Chicago
DATETIME_CONFIG = CURRENT
*Using a TZ is critical. I mean it. Ask me how I know.
Step 2 details:
2.a) Create a server class to distribute the app to the correct servers. Because this is a new server class, Splunk will need to be restarted.
#Checking on the powermt connection
[serverClass:powermt]
restartSplunkd = true
whitelist.0 = entlpdbc01*
whitelist.1 = entlpdbc02*
whitelist.2 = entlpdb07*
whitelist.3 = entltdbb01*
whitelist.4 = entltdbb02*
whitelist.5 = entltdb07*
[serverClass:powermt:app:powermt]
Step 3 details:
3.a) Because this script requires root access and splunk runs the script, I needed to add a sudo entry with EMC settings, etc. for each server. Here's what my entry looks like.
#========EMC COMMAND ACCESS===========
# User alias specification
User_Alias CMGU=splunk
# Cmnd alias specification
Cmnd_Alias CMGEMC=/tmp/nl_dwd/inq,/sbin/powermt
# User privilege specification
root ALL=(ALL) ALL
CMGU ALL=NOPASSWD:CMGEMC
#=========================================
After the clients phone home and pick up the new app, the data shows up in Splunk;
Again, this was quick and dirty and can certainly be cleaned up or made part of more broad "input script" support environment.
No comments:
Post a Comment