Application: Negative Searching Demo Bundle
Categories:
Description
This bundle, created jointly by Maverick and Stephen Sorkin, demonstrates a way to perform negative searches by indexing known patterns and catching anomalous patterns into a separate index.
author = Eric "Maverick" Garner and Stephen Sorkin
version = 1.0
minsplunkversion = 3.*
contactemail = maverick@splunk.com
description = This bundle demonstrates how to perform a negative search, or what is also known as "anomaly event detection".
Basically, it works like this. If you have a set of known log events and you can write a regular expression to identify each event separately, then you should be able to index those events accordingly.
Now, let's say a new event (one that have never been seen before and is, therefore, considered unknown) appears in a log file, you would want to have a way to catch that new unknown event and be alerted on it, take action, etc. Then, afterwards, it would be considered a known event, and you could simply add a new regular expression to identify it with the rest of the known events going forward.
For example, imagine that the following events are known and expected:
08/13/2007 10:00:01 NORMAL_1 test normal log event data here
08/13/2007 10:00:03 NORMAL_PATTERN_2 test normal log event data here
you could write regexes that identify NORMAL_1 and NORMAL_PATTERN_2 as known events. Simple enough.
Now, let's say the following event suddenly appears in the log file
08/13/2007 10:01:13 NORMAL_89 test unknown or "anomalous" pattern - WHJDNDBN KSKJ_junk_UF1D
since it would not match on any of the existing known regexes, this event would be considered an anomaly event and would be indexed appropriately.
Therefore, you could search that index for such anomalies, or essentially perform a "negative search".