The dojo of Splunk. Learn, share, teach, mentor.

AutoHeader and field extraction

I've spent some period of time trying to get Splunk to index our proxy logs and while I've made progress I'm still having some problems.

My issues all revolve around field extraction, I was unable to use the "extract fields" option within the GUI because it wasn't correctly detecting the data format so the regex's were all wrong (and I don't know enough to write my own). I then progressed to the AutoHeader setting and after modifiying my log source a little (to better delimit fields) I now have the fields being detected successfully. The problem now is that I also have the header line being listed as an indexed event. I assume this isn't the correct behaviour? This header is the first, commented, line of every log file, followed by thousands of lines of individual events. The log file appears to be being correctly detected automatically, assigned the correct sourcetype, and mapping the fields correctly... it's just still logging the field names as an event.

Any suggestions? I've found explanations of how to setup AutoHeader, but no discussion of the headers being shown in events.

Cheers,
James

▶ Reply to This

Replies to This Discussion

Permalink Reply by James Fitzell on February 4, 2010 at 1:58pm: I'm also having terrible trouble getting Splunk to automatically import my logs.

The log files are being sent by FTP to a local directory on the Splunk server. Splunk shows me the correct number of files in the Manager - Data Inputs - Files and Directories summary, however it doesn't seem to do anything with them. The Inputs Activity status doesn't show anything, the files aren't "ignored" but they're not processed either. If I restart Splunk it seems to correctly import the log files but since I'm sending these logs every hour or more I don't really want to be restarting Splunk every time.

I've attempted the crcSalt=source setting in the thought that it was incorrectly seeing my files as similar however that had no impact.; ▶ Reply to This

Permalink Reply by James Fitzell on February 9, 2010 at 7:05pm: I've resolved the failing to import issue. It was the crcSalt=! setting, I just wasn't using it properly (I thought source was meant to be replaced by the source, rather than literally typing in !).

Still no luck with the headers, they're all still being indexed. thankfully I can filter them out of search results but it's rather annoying with the first string in every search simply removes all the header results.; ▶ Reply to This