Blog

Intro

Recently, we worked with a client that was using a multi-tiered storage configuration for their Splunk deployment.  One tier was used for hot/warm data and the other tier for cold storage.  We wanted to test the cold storage tier specifically.  We used the Splunk event generator to produce data and tweaked some index settings to generate buckets (if you haven’t used the event generator before, here's a previous blog post for reference). 

Index Configuration

In the configuration below we'll use an index named "test."  Below is the index configuration for the "test" index. 

[test]
homePath = $SPLUNK_DB/test/db
coldPath = $SPLUNK_DB/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb
maxTotalDataSizeMb = 300000
maxDataSize = 1
maxWarmDBCount= 1

Let’s breakdown these settings in more detail:

• The home, cold, and thawed paths are required index settings in Splunk.  These file paths have to be writeable and specify where Splunk will write buckets. 

• The "maxTotalDataSizeMb" setting, which is in MB, is set to 300,000.  The default is 500,000, however this setting should be adjusted based on space available to prevent reaching a full disk issue on an indexer.

• The "maxDataSize" setting, which is in MB, is set to 1 so that the hot bucket can roll over to the warm bucket quickly to generate several warm buckets.  If your event generator is creating enough events this can be set higher so that buckets are not created too quickly.

• The "maxWarmDBCount" setting determines the maximum number of buckets kept in a warm state before rolling to a cold state.  The default is 300.  Typically this would be specified according to your data retention policy and storage needs.  In this example configuring the setting to 1, will get buckets to roll quickly from warm to cold.  As with the other settings this would not be appropriate for a production scenario. 

Conclusion

A couple of things to keep in mind:

• If you are not generating enough data, consider adding more inputs to the event generator. 

• You can also adjust the index settings to generate larger-sized buckets depending on the test scenarios.

• Make sure to monitor your storage and the rate at which you generate buckets.  It might be helpful to slow the event generation down and change the pace in which buckets are created.

By using the event generator and adjusting the index settings we were able to test the multi-tiered storage system of our client’s Splunk deployment.  We adjusted the index settings accordingly depending on the test case.  Thanks for reading!