Getting more intelligence on how much data splunk is eating.

As you know, there is a License pane in Splunk Manager (admin interface) that lets you know your "peak daily volume", and that figure is compared against your license volume. (free, or enterprise)

In the Splunk search app, (as of version 4.0.5) there is an "Index Activity" status dashboard in the search app (http://yoursplunkserver:8000/en-US/app/search/index_status). It does give you more information such as:

• Top five sourcetypes (by total KB indexed) in the last 24 hours
• Indexing rate over time by sourcetype (including internal indexes)
• Indexing amount per server

But most people want more detail, such as:

• For each host thats sending data, how much is it sending to Splunk?
• Which sourcetypes are consuming the lions share of data on a daily basis?
• Which sources are generating the most data today?
• Which indexes are receiving the bulk of the data?

Splunk eats up its own metrics logs — as you would expect it to (after all.. its a log). Those logs are stored in and index called "_internal". We can use Splunk search to ask Splunk a bit more about what its really doing when its indexing.

If you look at the search below, notice a few things.

I have it set for the last month (monthsago=1), so it will look over the last month and summarize the amount of data indexed. You can change the timerange—in search language, or just in the GUI time picker.
I am searching for “group=per_host_thruput” from that specific log. Additionally, since splunk is measuring in kilobytes, i am using the "eval" search command to multiply it so you can see it in megabytes, and thus creating a new field called "totalmb" -- which if ran, with the span=1d, will give you total megabytes per day. Get it?

index=_internal metrics kb series!=_* "group=per_host_thruput" monthsago=1 | eval indexed_mb = kb / 1024 | timechart fixedrange=t span=1d sum(indexed_mb) by series | rename sum(indexed_mb) as totalmb

After the summary table loads in your browser.. Click “Show Report”

But.. What other dimensions can I get from the data being indexed? I can modify the search and change the “group=’ and get summaries for each different vector.

group=per_host_thruput -
group=per_index_thruput
group=per_source_thruput
group=per_sourcetype_thruput

Example:

Turn it in to a graph by clicking "Show Report"

Got questions, fire away. Remember, these are pretty darn good estimates of what you are doing on a daily basis, but the License Manager rules all -- and does not use the log data to calculate your daily volume -- actually its the converse -- License Manager helps spit that log data out, and stores its understanding of "peak" in its own special database.