Splunk Alerts in Slack!
Here at Function1 we use Slack in order to stay in constant contact with our co-workers. If you haven't heard of Slack before, Slack is a team chat and communication tool. We use it to talk about our projects, company announcements, sports, random water cooler talk, technical questions, etc. Slack has integration built-in with a lot of services. We rely heavily on the GitHub, Asana, and Twitter integrations, but they have many others.
Since we do a lot of Splunk development for our own projects and our clients, we thought, "Wouldn't it be great to get Splunk alerts in Slack?" We had written custom alert scripts for clients before and Slack has a pretty great method of programatically creating content (Webhooks). In a couple hours, we had a script that did just that:
This alert provides the number of results in your search and a link so you can quickly view the actual search and result set in Splunk.
Check out our script and feel free to customize it in any way you like. You can snag a copy here.
You are some what limited with the type of output you can produce from a Splunk alert. You can create alerts that include the number of events returned in a data set, the search string used to produce the results, and the name of the saved report that was executed. The Splunk documentation has a full list of arguments that are available for you to utilize.
If you look at the code below, you can see where we are using the arguments in generating the JSON that we send to the Webhook URL.
There are a few other ways to further customize the Slack message. You can update the color that appears next to the alert by updating line 18 with either 'good' for green, 'warning' for amber, and 'danger' for red. Updating the title is as simple as updating line 27 of the script.
We are working on a full Slack app that has enhanced alerting functionality, modular inputs to get data out of Slack, and reports/dashboards. If you would like a preview version or have specific requirements for it, please feel free to reach out to us at firstname.lastname@example.org.
Also, if you don't get the privelige of using Slack at work (or even if you do), and you want to communicate with other Splunk users, Splunk has set up a User Groups Slack instance that has a lot of good discussions on it. To join, fill out this form. Join and say Hi!