Skip to content OpenLabs

Splunk Visual Handbook phase 2 · Where a byte stream becomes events

Parsing happens once, and you choose where

The parsing phase turns a byte stream into events, and it runs on exactly one machine: the first one in your path capable of running it. Which machine that is depends on your topology, not on where you put the file — so the same correct configuration is effective or inert depending on a decision taken in somebody else's ticket.

Primary sources

Cited above by name. Where a source has no link it is because none is stable enough to print — the citation is what you search for.

  1. Splunk Inc. (current). Configuration parameters and the data pipeline. Splunk Enterprise admin manual.

    The page this whole article is about. The table maps settings to phases, and under it is the rule and the worked example: with universal forwarders feeding a heavy forwarder feeding an indexer, input happens on the forwarders and parsing happens on the heavy forwarder.

  2. Splunk Inc. (current). How Splunk Enterprise handles your data. Splunk Enterprise documentation.

    What the parsing phase actually does, in one list: break the stream into lines, identify and set timestamps, annotate each event with the source-wide metadata, and transform the data according to regex rules.

  3. Splunk Inc. (current). Configure event line breaking. Splunk Enterprise documentation.

    The semantics of LINE_BREAKER in three sentences — where the previous event ends, where the next one starts, and what is discarded — plus the reason to prefer it over line merging, which is speed at ingestion volume.

  4. Splunk Inc. (current). Types of forwarders. Splunk Enterprise documentation.

    A heavy forwarder parses data before forwarding it, and can route on what it finds. This is the capability that, once present in a path, moves the phase.

  5. Splunk Inc. (current). Create custom fields at index time. Splunk Enterprise documentation.

    Read it for its warnings rather than its instructions: each indexed field enlarges the index and costs performance, and you cannot change fields on data that is already indexed.

  6. Splunk Inc. (current). Configure custom fields at search time. Splunk knowledge management manual.

    The other half of the same decision, and the sentence that should settle most arguments about it: you should generally extract fields at search time.

  7. Splunk Inc. (current). Splexicon: indexed field. Splunk documentation glossary.

    What is in the index by default, and what adding to it costs. Useful for seeing that the default fields are themselves index-time decisions — you are never choosing whether to have them, only whether to have more.