Index time v/s Search time-Processing

Index time v/s Search time-Processing

Splunk-Enterprise-terms search time & index time differentiate between methods of processing which happens during indexing & when search operations are performed.

Index time

It’s the period when Splunk gets new data to the time the information is written to Splunk index. In between this period, the information is parsed into segments & events, timestamps & default fields are extracted, & transforms are used.

Search time

This takes place when one searches via data, Splunk forms the fields as it compiles search results & doesn’t keep them in an index. When you’re running a search, numerous operations are made by Splunk to bring numerous knowledge objects & apply them to events returned via the search. The knowledge object includes calculated fields, extracted fields, etc.

Understanding the terms gets to be more significant while administering Splunk Enterprise. For example, assume that you’re planning to utilize a custom meta-data like a host & source type, then one needs to define the meta-data before he/she starts indexing. With this, the indexing process could tag-relate events with them. The moment indexing is done, you won’t be able to change meta-data assignments.

If one wishes to apply a custom meta-data to already indexed information, one can select either to re-index data, to apply custom meta-data to existing data, and new data. Also one will have an alternative, of managing the issue at search-time by tagging events with alternative values.

Index time & search-time Extraction

When Splunk has indexing data, it analyzes data-stream into a series-of-events, as a section of processing. Furthermore, it adds several fields to event information. The field is comprised of the default fields which it auto adds & any custom fields that you specified.

The process of adding fields to events is identified as field extraction. The 2 kinds of field extraction include

Search-time field-extraction

It takes place when one searches via indexed data. Splunk forms the fields when gathering search results & doesn’t store them at the index.

Index-time field extraction

In these, fields are kept in the index & got to be part of event data. There are 2 kinds of indexed fields. They include:

  • Custom fields

Those that you have specified

  • Default fields

Those which Splunk auto adds to every event

Remember that, while working using fields, consider more machine data either aren’t structured or has a structure that changes constantly. With this kind of information, use search-time field-extraction for maximum reliability & flexibility. Search-time field extractions are easily changed even after you’ve defined them. The general rule as commended by Splunk is better in performing more knowledge-building activities, like field extraction, at search-time.

Index-time custom field extractions could cost performance at all index times & search time. Every time a new field is added to several fields-extracted when indexing. The indexing process slows and a search operation at index gets to be slower, due to more additional fields. To prevent these performance problems consider relying on search-time field extractions if possible.

Kinds of field-extraction

Splunk provides 3 field extraction kinds which include inline, automatic key-value, and transform.   

List of processes carried out at Index time & Search time via Splunk.

Index-time Processes

The procedures are accomplished between the point when data is spent from the source & the point when it’s written to disk on the indexer.

The following are processes that occur through Index time:

  • Meta-data/default field-extraction like host, source type, source & timestamp
  • Dynamic or Static-host assignment for particular contributions
  • Auto host-assignment overrides
  • Source kind customization
  • Custom-index time field removal
  • Organized information field-extraction
  • Event time-stamping
  • Event line-breaking
  • Event-segmentation

Search-time Procedures

This occurs while a search process is moving on, as an event is collected by search.

The processes which occur at search-time:

  • Event segmentation that also happens at index-time
  • Event kind matching
  • Search-time field-extraction can be automatic & custom field-extractions like multi-value fields & calculated fields.
  • Field aliasing
  • Adding fields from lookups
  • Source kind renaming
  • Tagging

Other Important Links:

Leave a Comment