Timestamping is the bane of my existance

Replies to This Discussion

Permalink Reply by Michael Wilde on March 5, 2010 at 7:21pm

Nicholas.. can we establish a pattern on what precedes the timestamp... like will it always be some number of digits, followed by a TPW, followed by your date pattern?

If so.. this should be easy..

▶ Reply to This

Permalink Reply by nicholas Lehman on March 6, 2010 at 4:24am

the information preceeding the timestamp will always be different. The beginning of the event log contains record length, sequence number, journal code and entry type. Even the information immediately before and after the timestamp will be different depending on the event type. However, the AS/400 logs have fixed field length, and the timestamp will always start at an offset of 19 characters. I attempted to enter it in as a regex and start 19 characters out, but it still comes back with no real rhyme or reason to the supposed timestamps that splunk picks.

▶ Reply to This

Permalink Reply by Michael Wilde on March 6, 2010 at 9:24am

This will work.

First, create a props.conf in an appropriate directory (system level or app level), I'll make mine in $SPLUNK_HOME/etc/system/local/props.conf

Add this to it:

#my sourcetype will be called "as400"
[as400]
#I want splunk to bypass the first 18 word (digit or letter) characters when looking for a timestamp
TIME_PREFIX = \w{18}
# i want splunk to turn off its intelligence on linemerging for multiline events (this is optional)
AUTO_LINEMERGE = FALSE
# i want to force splunk to not create multiline events (this is optional)
SHOULD_LINEMERGE = FALSE
# I shall apply the following strptime format.
TIME_FORMAT = %m%d%y%H%M%S

Next, when you index the file(s) lets say with Splunk's "monitor a directory" feature, hardcode the sourcetype as "as400"

My inputs.conf looks like this:

[monitor:///Users/michaelwilde/Logs/as400.log]
disabled = false
sourcetype = as400

And the result looks like this:

If your data is already indexed, and timestamping is incorrect, create the props.conf file, edit your inputs.conf (which is probably in $SPLUNK_HOME/etc/apps/search/local/inputs.conf -or- $SPLUNK_HOME/etc/apps/launcher/search/local/inputs.conf) Put the as400 sourcetype in there.

Then stop splunk, clean your index $SPLUNK_HOME/bin/splunk clean eventdata -index main. Then start Splunk again and it should index it properly. If not, check to see if its taking your new as400 sourcetype. If not, just blow away your inputs, start again. You'll get it.

▶ Reply to This

Permalink Reply by nicholas Lehman on March 8, 2010 at 7:41am

Awesome, it worked! One question though, for the monitor option in the inputs.conf, can I just leave the end of the path open ended to look in that directory for any new files? I see you put a file at the end, and am wondering if I'm a bit confused on the capabilities of the monitor directory funciton.

▶ Reply to This

Permalink Reply by Michael Wilde on March 8, 2010 at 7:50am

You are correct. Monitor a directory will work just fine. I user a single file to answer your challenge. Glad it worked for you!

▶ Reply to This

Permalink Reply by nicholas Lehman on March 8, 2010 at 8:02am

haha, I was just about to come back and let you know I answered my own question. Thanks for all the help Michael!

▶ Reply to This

Permalink Reply by Michael Wilde on March 8, 2010 at 9:47am

Question for you:

How'd you get the logs from the AS/400. Arent they in EBCDIC, if so did you convert them? Whats your strategy for AS/400. Please share. (might even be blog post worthy!)

▶ Reply to This

Permalink Reply by nicholas Lehman on March 8, 2010 at 10:02am

Well, this is a down and dirty compliance stop gap. The log journals get spat out to text file every night, and then a cron job runs a script to pull them off the as/400 via sftp along with an MD5 sum file. The script then performs an md5sum on the file that was pulled down and if the sums match, it throws the log file into the folder that Splunk is monitoring. I got a little creative with the script though, and if the as/400 can't be reached or the md5sums don't match it sends an alert e-mail to my phone. As for making the log info intelligible, I took a look at the IBM guide for reading logs and created custom fields for each section of a log event. It was very time consuming, and the regex I used (I'm positive) could have been better for it, but it works. The script I wrote was quite simple and if you want to have a look at it, I'll post it (minor editing done of course).

▶ Reply to This

Permalink Reply by Michael Wilde on March 8, 2010 at 10:52am

If your script is "post-able".. that'd be great... the "AS/400 question" comes up often and users would like to benefit from your experience.

▶ Reply to This

Permalink Reply by nicholas Lehman on March 8, 2010 at 12:16pm

Well, if this is a common occurrence, I may want to look into developing an app for AS/400. In the meantime, I can post the script I made for my particular application. Simple bash (cause perl confuses me).

#!/bin/bash
cd ~/
echo accessing AS400
sftp as400user@as400box SFTPDUN
get auditdata
get md5.txt
quit
SFTPDUN

if [ -e ~/auditdata ]; then

filename=as400_$(date +%Y%m%d).log
newsum=`md5sum auditdata |tr ' ' '\n' |head -n 1`
oldsum=`cat md5.txt | tr ' ' '\n' |head -n 1`

if [ "$newsum" = "$oldsum" ]; then
chown as4log:splunk auditdata
cp auditdata /opt/splunk/as400_logs/$filename
cp auditdata ~/as400_logs/$filename
cp md5.txt ~/as400_logs/md5$(date +%Y%m%d)
rm auditdata
rm md5.txt
else
SUBJECT="Bad AS/400 Log File"
EMAIL="me@me.com"
BODY="A check of the MD5 sum for AS/400 log file $filename, has returned a bad checksum"
echo $BODY | /bin/mail -s "$SUBJECT" "$EMAIL"
cp auditdata ~/as400_logs/$filename
cp md5.txt ~/as400_logs/md5$(date +%Y%m%d)
rm auditdata
rm md5.txt
fi
else
SUBJECT="Unable to access AS/400 for logs"
EMAIL="me@me.com"
BODY="ALERT! The scheduled log retrieval for $(date +%m-%d-%Y) could not be retrieved. Please verify that access is available."
echo $BODY | /bin/mail -s "$SUBJECT" "$EMAIL"
fi

▶ Reply to This