Blog

Intro

So you did it. You early adopter you! You love having the latest and greatest Splunk Enterprise has to offer and upgraded to Splunk Enterprise 6.2. The new UI is snazzy, the new regex field extractor wizard is magical, the Search Head Clustering feature is what we've all been waiting for, and how about that savvy new App bar display? And that is not all Splunk Enterprise 6.2 has to offer. Here’s the situation: you are on your Splunk 6.2 instance and you navigate over to the settings drop-down and there it is in green staring at you, the Distributed Management Console (DMC).

The Distributed Management Console

The Distributed Management Console or DMC is a new feature introduced in Splunk Enterprise 6.2 that allows you to monitor your Splunk Deployment by providing real-time monitoring and performance reporting from a central search head via eight different dashboards. So think S.O.S, Splunk Deployment Monitor, the Splunk App for Unix and Linux, and bonus features all wrapped up into one app without the need for additional add-ons and without the additional resources and license usage of running three additional apps. With the DMC, you are able to view your Splunk license usage information, indexing performance, search performance and usage, CPU, disk space and memory usage, as well as reports surrounding the KV store collections.  For those Splunk users who are running single instances, no need to fret or fear, do not let the “Distributed” in the DMC fool you, the DMC has a “Standalone mode” which is perfect for users running a single Splunk Enterprise instance.

You can choose your mode by navigating over to the Setup option in the DMC and selecting either Standalone or Distributed mode. Those who are running Splunk Enterprise within a distributed environment will have a few more steps to walk through before fully harnessing the power of the DMC. Before configuring the DMC in your distributed deployment, be sure that you are following Splunk’s best practice to forward internal logs from all search heads to the indexer layer. This is a very important prerequisite as certain views within the DMC are dependent on $SPLUNK_HOME/var/log/splunk and $SPLUNK_HOME/var/log/introspection.  For more information about search head log forwarding see : http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/Forwardsearchheaddata

Distributed Deployments

Each Splunk Enterprise 6.2 component, except for the universal forwarders comes configured with the DMC. To monitor your deployment from a single Splunk instance, you must configure a search head to be the DMC’s host. The DMC’s host will be the Splunk Enterprise instance where users can view all performance reports and metrics surrounding their distributed deployment.  Now, depending on your environment, you have several different options for a host. The best option in a distributed Splunk environment is standing up a dedicated search head to be the DMC’s host. Another option for a distributed deployment is configuring a dedicated license master as the DMC’s host. Since the DMC’s host has to be a search head, in an indexer clustering environment you could configure the cluster master as the DMC’s host. In deployments where search head clustering is enabled, you could certainly use the deployer instead of a search head cluster member.

Once you’ve selected your host, you will want to configure all instances, except for clustered indexers (excluding the master) and universal forwarders, of your Splunk deployment as search peers to the DMC’s host. You can do this though Splunk Web on the DMC’s host by navigating over to Settings -->Distributed search --> Search peer --> Add New and adding the required information for each instance.

Note about heavy forwarders

If you had a chance to review the docs you would’ve noted that in the DMC’s architecture, the DMC's host is not interacting with the universal forwarders. In addition, there is not anything concerning heavy forwarders. That is because this release of the DMC is not catered to the forwarding level. That being said, since heavy forwarders are full Splunk Enterprise instances, you certainly can add it to the DMC as a search peer but you will need to be mindful of the Server roles. Once you’ve added your Splunk Enterprise instances as search peers to the DMC, the DMC will automatically assign a role(s) to peer based on the component’s responsibility in the Distributed Deployment. The six roles recognized by the DMC can be found in the image below.

Since the DMC doesn’t cater to the forwarding level, there is not a role listed for heavy forwarders. Also, since a Heavy Forwarder does some data parsing, the DMC will assign the indexer role to the Heavy Forwarder. Since the role category is required, the best course of action here is to create a custom group to tag the Heavy Forwarder(s).You can create groups within the DMC by navigating over to the setup option.

Monitoring your Deployment

After all instances have been successfully added as search peers, you will want to go back to the setup option in the DMC and turn on the Distributed Mode. Once the Distributed mode has been enabled, you will want to review the server roles to ensure that the appropriate role is given the right component. Other than that, you now have the power to monitor your entire Splunk Enterprise deployment to ensure that you are obtaining the maxium value out of your Splunk Enterprise deployment.

For more information about the DMC: http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/ConfiguretheMonitoringConsole

Happy Splunking!