Using HTTP Event Collector and Splunk Logging Driver to Gain Visibility into Your Docker Applications

image

Splunk is always looking for ways to make it easy for its users to collect machine generated logs to gain real-time visibility into their infrastructure and IT OperationsThe traditional way of collecting logs is through the Splunk universal forwarder, which is installed at the data source to route events to the Splunk.  Splunk introduced a new way to collect logs using the HTTP Event Collector (HEC), a token-based JSON API that allows you to send application events to your Splunk deployment using HTTP or Secure HTTP (HTTPS). HEC was mainly created for developers in mind as a way to send application logs to Splunk without using a universal forwarder.

 

In this post I will explain how to set up HEC and the Splunk Logging Driver, which is a new driver available in Docker that works alongside HEC to route events to Splunk.  The scenario involves collecting logs from services running in Docker containers.

 

 

Setting Up HTTP Event Collector in Splunk 

 

  • The first thing is to enable HEC in Splunk as it is not enabled by defaultTo do this open Splunk's Web UI and navigate to Settings - > Data Inputs -> HTTP Event Collector and click Global Settings at the top right corner. The screen below should pop up. 

               Image

  • For All Tokensselect Enabled 

  • Default Index drop-down allows you to select an index for all HEC tokens  

  • To set the default forwarding output group for all HEC tokens, choose an output group from the Default Output Group drop-down. 

  • If you wish to use deployment server to manage HEC tokens configurations, click Use Deployment Server. 

  • For secure HEC communications, click the Enable SSL. This allows you to communicate via HTTPS instead of HTTP.  

  • You can specify the port number that HEC listens on, it is 8088 by default

  • Click Save when you are done. 

  • HEC endpoint is now enabled to listen for data on select port! 

  • Since HEC uses token-based authentication, you are required to setup at least one token. To generate a token navigate to Settings - >  Data Inputs -> HTTP Event Collector and click New Token on the top right of the page. Below wizard to create a new input/tokxtcaen will appear 

                  Image

  • Enter a descriptive Name for the new token, Source name override, and Description

  • Click Next when you are done

  • On the next page, you can specify the source type and index for the token/input.

             Image

 

  • HEC will automatically set the source type if you don't specify one.
  • Assign your token to an index. This is where your data will be stored for the input. 
  • Click Review 

          Image

 

  • You now have a new token! Take note of the token value that was generated as you will need this for your Docker configurations. 

Setting up Splunk Logging Driver in Docker  

  • First, in order to use the Splunk Logging Driver feature, ensure you have Docker version 1.10 or higher 
  • Next, there are few config items you will have to set (example below) 

      Example:  

[Service] 

ExecStart= 

ExecStart=/usr/bin/dockerd --log-driver=splunk --log-opt  

splunk-token=HTTPEVENTCOLLECTORTOKEN --log-opt  

splunk-url=https://your_splunk_instance:8088 --log-opt  

 

  • splunk-token insert the token you generated in Splunk here after the "=" sign

  • splunk-url - specify the Path to your Splunk instance  

  • With these few configurations you are now able to send events from Docker to your Splunk instance  

  • Here is a link that provides a list of all the available options in Splunk Logging Driver https://docs.docker.com/engine/admin/logging/splunk/ 

 

Best Practices for Creating Tokens and Assigning Indexes 

 

I recommended that you generate one token for each application/service in Docker that is routing data to SplunkSetting it up this way gives you a lot of control in terms of managing your data in Splunk. If you ever need to send data to Splunk from a new application or service in Docker, you can simply create a new token and then configure the sourcetype and index on the HEC.  Creating tokens this way also allows you to quickly identify and trouble shoot issues with your inputs if they ever come up.  

 

In terms of organizing data into indexes, separate tokens can be configured to send data to the same index if it makes sense to.  So if you had several web application services running, each with their own HEC token, all the HEC inputs could all be configured to send their data to a "web_app" index.  Configuring the data inputs in HEC allows for ease of configuration at the data source and flexibility in organizing data in Splunk.  Read here for Splunk's best practices for creating custom indexes. 

 

Thanks for reading and happy Splunking! 

Subscribe to Our Newsletter


Stay In Touch