Create a new Splunk Index

Create a new Index

I wanted to collect some data that no one else would be interested in. It had no need to be retained and was purely a one-off for test information. In short, a good case for a temporary index.

In this example, I'm going to show a simple scripted input and a simple logfile monitor. 

We use a deployment server and our indexers are not replicated, so this is fairly simple.

1. Create the new index in the indexes.conf that gets sent to your indexers;

tail /splunk/etc/deployment-apps/all_indexers/indexes.conf

[throwaway]
homePath   = volume:primary/throwaway
coldPath   = volume:primary/throwaway/colddb
thawedPath = $SPLUNK_DB/throwaway/thaweddb
tstatsHomePath = volume:primary/throwaway/datamodel_summary
summaryHomePath = volume:primary/throwaway/summary
maxMemMB = 20
maxHotBuckets = 10
maxConcurrentOptimizes = 6
maxTotalDataSizeMB = 1024
maxWarmDBCount = 30

Note, our data is managed on EMC storage so data ages and migrates on that according to other requirements.

These might be decent requirements for short retention;

#1GB bucket
maxDataSize = 1024
#1 day
maxHotIdleSecs = 86400
#30 buckets = 30days
maxWarmDBCount = 30
#90 days
frozenTimePeriodInSecs = 7776000 (90 days in sec, cold to frozen)
Not set. Once 90 days has passed the data is deleted.
#coldToFrozenDir = 

2. Create a new app. Mine is in /splunk/etc/deployment-apps/throwaway_app;

.
|-- bin
|   `-- script.sh|-- local
|   |-- app.conf
|   |-- inputs.conf
|   `-- props.conf
`-- metadata
    |-- default.meta
    `-- local.meta

cat bin/script.sh

#!/bin/bash
top -n 1 | grep splun[k] | awk '{print $3" Virt:"$6" Res:"$7}'
ps -ef | grep splunk

 cat local/inputs.conf

[script:///opt/splunkforwarder/etc/apps/throwaway_app/bin/script.sh]
disabled = false
index = throwaway
# Run every 15 minutes
interval = 900
source = throwaway.script
sourcetype = script:///opt/splunkforwarder/etc/apps/throwaway_app/bin/script.sh

[monitor:///opt/logfile.log]
disabled=false
index = throwaway
sourcetype = throwaway.logfile

cat local/inputs.conf

[top]
TZ = America/ChicagoDATETIME_CONFIG = CURRENT

Note: You'll need to specify where the scripts will be (I need to set some system variables.)

Note: This is where you direct the data to the index you want it to go.

Note: For the love of everything holy, put the timezone in there so so you'll be able to find any non-timestamped data later.

Next, update the appropriate serverclass.conf so the app can be pulled down by the targeted hosts.

#New app for a short term index
[serverClass:throwaway_app]
restartSplunkd = true
whitelist.0 = sljdsb02*
[serverClass:throwaway_app:app:throwaway_app]

At this point you can make everything available to the indexers and the forwarding hosts;

./splunk  reload deploy-server